@jrwiebe @lintool @ianmilligan1 finally wrote it up. Let me know if I missed anything, or y'all want me to provide more info.

@ruebot, you say:

Upon investigation, this does not seem to be a problem with a large ARC/WARC or large collection.

Can you clarify this? Are you saying that jobs run with the same parameters against other large collections – in terms of total archive size – do not result in heap space errors? Because in my testing I didn't find this to be the case. It seemed like the error was rather a function of the total size of all the WARCs being operated on simultaneously. I think I was seeing jobs bail when they didn't have about 4 times the amount of memory as [# tasks] * [average WARC size].

I found this garbage collection log analyzer useful for trying to understand how memory was being used, though I'm still puzzled that so much memory is being used. See for example how much memory is used by a job that simply chains RecordLoader.loadArchives(...).count(). Some useful background on Java garbage collection here.

To generate garbage collection logs you want to do something like this (the spark.driver.extraJavaOptions conf directive is the key):

/home/jrwiebe/spark-2.4.1-bin-hadoop2.7/bin/spark-shell --conf spark.driver.extraJavaOptions="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:3260-16-16g-35bigwarcs.log -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+UnlockExperimentalVMOptions -XX:G1LogLevel=finest -XX:+UnlockDiagnosticVMOptions -XX:+G1PrintRegionLivenessInfo" --master local[16] --driver-memory 16g --conf spark.network.timeout=10000000 --conf spark.executor.heartbeatInterval=600s --conf spark.driver.maxResultSize=4g --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.shuffle.compress=true --conf spark.rdd.compress=true --packages "io.archivesunleashed:aut:0.17.0" -i 3260.scala

For the record, I'm pretty certain KryoSerializer doesn't offer any benefits for this problem; I just kept using it because I copied @ruebot's command line. I ran tests with and without some of the other tuning options as well, and was seeing similar results.

Sorry, yeah, that's kinda vague. I meant overall, we've analyzed 170T of collections ranging from under a 1G of WARCs up to 12T with the same basic Spark settings.

I added the KryoSerializer during troubleshooting because it looked like it got further along in some cases, and reading up on it, it made sense at the time.

Thanks for putting in the GC info. I was hoping to tease it out by creating the issue 😄

...and a single WARC as large as 80G.

Reading the garbage collection post reminded me that Spark allocates (not sure if that is the right word), ~55-60% of "storage" memory for executors. A smarter version of me would have bookmarked that Stackoverflow or Spark mailing list thread. But, I can provide a screenshot of a the Spark UI of a currently running job.

That's a job running with 30 threads, and 105G of RAM allocated: --master local[30] --driver-memory 105g

I'm going to start adding collection info on failed collections here:

As a sanity check, I went to the data directory and did this:

$ gunzip -c *.gz | grep -a "Content-Length" | cut -d ' ' -f 2 | sort -n | less
2878090679
2878090400
2878090400
2247649420
2247649420
2247649180
2247649180

2878090679 = 2,878,090,679 ~ 2.9 GB. A bit silly to have records that big... dunno if this is the source of your errors though.

So an individual record that’s almost 3GB? I wonder if we could implement a default that skips records greater than say a GB? Or lower?

I can imagine users who might want to work with big files - video etc - but for the most part, and with our generic processing pipeline, we could do without.

I'm not certain that record size is the issue. I've collected some data from a number of collections that fail, and succeed. Large record size doesn't appear to be the sole factor in failure or success. We have some that are well above what @lintool listed having been a part of successful jobs. I'll take some time over the coming days, or weeks 😓, to try and isolate some of the WARCs that are actually causing the problems, then we can proceed digging in further from there if that makes sense.

PQ: Fails on extractVideoDetailsDF()

1. 263,051,370
2. 263,051,370
3. 263,051,370
4. 263,051,370
5. 146,567,997

InternationalesQc: Success

1. 585,726,559
2. 148,428,760
3. 148,428,512
4. 135,553,646
5. 135,553,230

10689: Fails on AUK

1. 2,878,090,679
2. 2,878,090,400
3. 2,878,090,400
4. 2,247,649,420
5. 2,247,649,420

7235: Fails on AUK

1. 65,608,807,060
2. 65,608,807,060
3. 65,608,807,060
4. 65,608,807,060
5. 65,608,807,060

AgricultureQc: Success

1. 526,026,425
2. 526,026,050
3. 341,618,524
4. 341,618,149
5. 238,542,670

EnvironnementQc: Success

1. 7,061,547,393
2. 7,061,547,393
3. 3,741,979,582
4. 3,733,722,254
5. 3,733,722,254

12190: Fails on AUK

1. 3,273,574,468
2. 3,273,574,204
3. 2,580,347,094
4. 2,580,346,830
5. 2,185,762,748

TourismeQc-2006-2017: Success

1. 50,182,078
2. 26,795,182
3. 24,213,112
4. 23,125,967
5. 22,184,611

Hit a very similar error a couple times over the last few days on two different collections with Java 11 and Spark 3.0.0. I had initially thought it was a freak new error because of the new setup, but I believe this is the same error that we've all been running into for a few years now, and is definitely a blocker for 1.0.0.

20/06/23 10:00:58 ERROR Utils: Aborting task
java.lang.NegativeArraySizeException: -1976675752
        at java.base/java.lang.StringCoding.decodeUTF8_0(StringCoding.java:755)
        at java.base/java.lang.StringCoding.decodeUTF8(StringCoding.java:712)
        at java.base/java.lang.StringCoding.decode(StringCoding.java:318)
        at java.base/java.lang.String.<init>(String.java:592)
        at java.base/java.lang.String.<init>(String.java:614)
        at io.archivesunleashed.ArchiveRecordImpl.<init>(ArchiveRecord.scala:119)
        at io.archivesunleashed.package$RecordLoader$.$anonfun$loadArchives$2(package.scala:79)
        at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:512)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:272)
        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1411)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:281)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$15(FileFormatWriter.scala:205)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:127)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

The common thread following the stack trace in all the errors in this issue are io.archivesunleashed.ArchiveRecordImpl.<init>(ArchiveRecord.scala.

@lintool had previously patched the issue with lintool/warcbase@16db934, which I believe was meant to just skip over broken records, but didn't address the underlying issue that @anjackson brought up here; that we're hitting the 2G array limit size in Java.

I think the best path forward then is to decide if we want to truncate payloads like Andy suggested (I'm not 100% sure we should be do that), or update the *RecordUtils to have a streaming interface like webarchive-discovery does (hopefully that's the right spot I'm pointing to 😅).

One important point I left out that should reinforce the conclusion of the above comment, is that I'm fairly certain that the issue isn't WARC/ARC file specific.

One example to share is from 3940 (Alberta Oil Sands). It's about a 25G collection. If I run it on all the files (25 files), it will fail with the error listed in the comment immediately above. If I pull out the WARC that causes the analysis to fail, the job with the 24 remaining WARCs completes successfully. That would lead one on to believe it's just a bad WARC, so toss it out! But, it is not. If I run the same job on just that WARC alone, it completes successfully. Which, I believe reinforces the above note about hitting the 2G array limit in Java since we're juggling all the records in memory at a given time.

@lintool's comment above hints at the problem as well; all those records above 2G.

There's a cruel irony in the fact that mere hours before @ruebot's comment I deleted all my old memory profiles from tests I ran trying to understand this issue a year ago. Not that I think they would be particularly helpful – I never managed to figure out what were the limits of memory allocation, file size, workers, etc., that would produce the error.