One quick solution might be passing

sc.setLogLevel("INFO")

in Spark-shell, which gives you very verbose logging. It does include input information like this:

2017-12-21 09:59:00,434 [Executor task launch worker for task 6] INFO  NewHadoopRDD - Input split: file:/Users/ianmilligan1/Dropbox/git/aut/example.arc.gz:0+2012526
2017-12-21 09:59:00,435 [Executor task launch worker for task 7] INFO  NewHadoopRDD - Input split: file:/Users/ianmilligan1/Dropbox/git/aut/example2.arc.gz:0+2012526

No file close information, but I have used it to debug things before (such as finding bad W/ARCs).

Just re pinging on this @dportabella, is this something you want above and beyond the Spark logging options?

Just re pinging on this @dportabella, is this something you want above and beyond the Spark logging options?

No. Your solution sc.setLogLevel("INFO") is fine for me, thx!

But I would need file close information also.
Thought it is not an urgent issue.

@dportabella, my recent commit addresses your request. If you set the log level to "INFO", you will see messages like this:

2019-01-29 12:54:11 INFO  ArchiveRecordInputFormat:141 - Opening archive file file:/home/jrwiebe/aut/target/test-classes/arc/example.arc.gz
2019-01-29 12:54:11 INFO  ArchiveRecordInputFormat:240 - Closed archive file file:/home/jrwiebe/aut/target/test-classes/arc/example.arc.gz

Is this satisfactory?

Great, thanks!