It seems that SparkContext textFile expects only files to be present in the given directory location - it does not either
- (a) recurse or
- (b) even support directories (tries to read directories as files)
Any suggestion how to structure a recursion - potentially simpler than creating the recursive file list / descent logic manually?
Here is the use case: files under
/data/tables/my_table
I want to be able to read via an hdfs call all the files at all directory levels under that parent directory.
UPDATE
The sc.textFile() invokes the Hadoop FileInputFormat via the (subclass) TextInputFormat. Inside the logic does exist to do the recursive directory reading - i.e. first detecting if an entry were a directory and if so then descending:
<!-- language: java -->
for (FileStatus globStat: matches) {
218 if (globStat.isDir()) {
219 for(FileStatus stat: fs.listStatus(globStat.getPath(),
220 inputFilter)) {
221 result.add(stat);
222 }
223 } else {
224 result.add(globStat);
225 }
226 }
However when invoking sc.textFile there are errors on directory entries: "not a file". This behavior is confusing - given the proper support appears to be in place for handling directories.
See Question&Answers more detail:os