In my original implementation I wrote a DataFrameLoader, but it seems to have rapidly fallen out of use... We should decide on the idiom we want for loading DataFrames.

Current implementation:

val df = RecordLoader.loadArchives("example.arc.gz", sc).extractImageDetailsDF()
// alternatively, extractValidPagesDF, extractHyperlinksDF, etc.

The downside of this is that the user has access to raw RDDs, which is what loadArchives returns... this is asking for trouble in mixing RDDs and DFs in unpredictable ways?

Another option would be to introduce a DF interface that does not give access to RDDs. Something like:

val df = DataFrameLoader.loadArchives("example.arc.gz", sc).images

The other nice feature is that we can have much shorter DF names like pages, links, images, image_links, etc. - don't need the DF part to disambiguate because DataFrameLoader makes this clear. One more nice features is the ability to selectively reduce scope down the road and hide RDDs from the user, as we move completely over to DFs.

I'm leaning towards this design, but would be happy to hear opinions from others...