Skip to content

/aut

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Class issues when using aut-0.17.0-fatjar.jar #313

Closed
ruebot opened this Issue Mar 21, 2019 · 4 comments

Comments

Assignees

@ruebot ruebot

Labels
bug
Projects
None yet
Milestone
No milestone
2 participants
@ruebot @jrwiebe
@ruebot
Copy link
Member

ruebot commented Mar 21, 2019 •
edited

Describe the bug

java.lang.NoClassDefFoundError and java.lang.NoClassDefFoundError errors are produced when using aut-0.17.0-fatjar.jar. Looks like a potential class conflict in the fatjar, but not when the using --packages.

To Reproduce

  1. spark-shell --jars /home/nruest/Downloads/aut-0.17.0-fatjar.jar
  2. Load libraries, and RecordLoader.loadArchives:
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0
      /_/
         
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_181)
Type in expressions to have them evaluated.
Type :help for more information.

scala> :paste
// Entering paste mode (ctrl-D to finish)

import io.archivesunleashed._
import io.archivesunleashed.matchbox._
val r = RecordLoader.loadArchives("/home/nruest/tmp/auk/75/3552/warcs/ARCHIVEIT-3552-DAILY-IEDLRH-20130212071008-00001-wbgrp-crawl052.us.archive.org-6680.warc.gz", sc)

// Exiting paste mode, now interpreting.

java.lang.NoClassDefFoundError: net/sourceforge/cobertura/coveragedata/TouchCollector
  at io.archivesunleashed.package$RecordLoader$.__cobertura_init(package.scala)
  at io.archivesunleashed.package$RecordLoader$.<clinit>(package.scala)
  ... 49 elided
Caused by: java.lang.ClassNotFoundException: net.sourceforge.cobertura.coveragedata.TouchCollector
  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  ... 51 more
  1. Do it again in the same session:
scala> :paste
// Entering paste mode (ctrl-D to finish)


import io.archivesunleashed._
import io.archivesunleashed.matchbox._
val r = RecordLoader.loadArchives("/home/nruest/tmp/auk/75/3552/warcs/ARCHIVEIT-3552-DAILY-IEDLRH-20130212071008-00001-wbgrp-crawl052.us.archive.org-6680.warc.gz", sc)


// Exiting paste mode, now interpreting.

java.lang.NoClassDefFoundError: Could not initialize class io.archivesunleashed.package$RecordLoader$
  ... 57 elided
  1. @edsu example

Expected behavior

Well, this shouldn't happen. It should work.

Environment information

  • AUT version: 0.17.0
  • OS: Ubuntu 18.04
  • Java version: Java 8
  • Apache Spark version: Tested on 2.3.0, 2.3.2, 2.4.0
  • Apache Spark w/aut: --jars
  • Apache Spark command used to run AUT: spark-shell --jars /home/nruest/Downloads/aut-0.17.0-fatjar.jar

Additional context

@edsu notes in Slack that the 0.16.0 fatjar works.

@ruebot ruebot self-assigned this Mar 21, 2019

@ruebot ruebot added the bug label Mar 21, 2019

@jrwiebe

This comment has been minimized.

Sign in to view
Copy link
Contributor

jrwiebe commented Mar 21, 2019

@ruebot, what was the build environment where the 0.17.0 release was performed? I checked out the 0.17.0 release branch, built it, and ran the same test, and did not receive an error. However, I do receive the error when using the released fatjar.

I think the problem is that the wrong SLF4J binding is being included in the fatjar. See this issue, and this background. It's not clear to me why I can't replicate the error when I build 0.17.0, though.

Side issue: I think we can drop cobertura-maven-plugin as a general dependency. It's listed as a dependency in <reporting>, which is sufficient for the mvn cobertura:cobertura target.
@ruebot

This comment has been minimized.

Sign in to view
Copy link
Member Author

ruebot commented Mar 21, 2019

@jrwiebe it would have been my laptop or desktop; Ubuntu 18.04, and OpenJDK 8 probably.

Yeah, the SLF4J binding makes since. It came up for me this morning in a quick cursory google.

...good catch on cobertura. That pom haunts me sometimes 😄
@ruebot

This comment has been minimized.

Sign in to view
Copy link
Member Author

ruebot commented Mar 21, 2019

Helping out a team here at the datathon. This worked, so maybe we're fine on master?

$ ~/spark/bin/spark-shell --jars /home/ubuntu/aut/target/aut-0.17.1-SNAPSHOT-fatjar.jar --master local[*] --driver-memory 54G --conf spark.network.timeout=100000000 --conf spark.executor.heartbeatInterval=6000s --conf spark.driver.maxResultSize=12
2019-03-21 20:23:12 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://host-192-168-69-7.openstacklocal:4040
Spark context available as 'sc' (master = local[*], app id = local-1553199803313).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0
      /_/
         
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)
Type in expressions to have them evaluated.
Type :help for more information.

scala> :paste
// Entering paste mode (ctrl-D to finish)

import io.archivesunleashed._
import io.archivesunleashed.matchbox._
val r = RecordLoader.loadArchives("/mnt/data/dcpl/dc-punk/warcs/ARCHIVEIT-10427-TEST-JOB677841-20180831145751779-00001.warc.gz", sc)

// Exiting paste mode, now interpreting.

import io.archivesunleashed._
import io.archivesunleashed.matchbox._
r: org.apache.spark.rdd.RDD[io.archivesunleashed.ArchiveRecord] = MapPartitionsRDD[2] at map at package.scala:69
@jrwiebe

This comment has been minimized.

Sign in to view
Copy link
Contributor

jrwiebe commented Mar 23, 2019

As I mentioned yesterday, when I built the 0.17.0 release I also had no problem. I tried this with OpenJDK 8 under Linux (Ubuntu 18.04 and OS X) and with Oracle JDK 8 (just OS X).

@ruebot ruebot referenced this issue Mar 23, 2019

Merged

Resolve cobertura-maven-plugin class issue; resolves #313. #314

@jrwiebe jrwiebe closed this in 8504190 Mar 23, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.