Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert RecordLoader.loadArchives to a Spark Data Source #371

Open
ruebot opened this issue Nov 5, 2019 · 1 comment
Open

Convert RecordLoader.loadArchives to a Spark Data Source #371

ruebot opened this issue Nov 5, 2019 · 1 comment

Comments

@ruebot
Copy link
Member

@ruebot ruebot commented Nov 5, 2019

Since we're pivoting to full DataFrame support (#223, #190), we should convert/migrate RecordLoader.loadArchives, and any other related functions to a Spark Data Source. That way we could do things like:

spark.read.format("webArchive")
  .option("mode", "FAILFAST")
  .option("inferSchema", "true")
  .option("/path/to/files")
  .schema(someSchema)
  .load()

Then, we could, (since it's an open issue #147) write WARCs that way too? 🤷‍♂

spark.write.format("webArchive")
  .option("mode", "OVERWRITE")
  .option("/path/to/files")
  .save()

These are the Spark core data sources:

  • CSV
  • JSON
  • Parquet
  • ORC
  • JDBC/ODBC
  • Plain-text
  • Avro

Community implemented data sources:

  • Cassandra
  • HBase
  • MongoDB
  • AWS Redshift
  • XML
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.