This PR improves ExtractDomain by replacing backslash with forward slash in URL before passing it into Java URL class.

GitHub issue(s):

• #269

What does this Pull Request do?

This PR improves URL parsing in ExtractDomain by replacing backslash with forward slash before passing it into Java URL class, allowing ExtractDomain to capture the true domain of an URL.

How should this be tested?

• git fetch --all
• git checkout fix-url
• mvn clean install
• Create an output directory with sub-directories:
  mkdir -p path/to/where/ever/you/can/write/output/all-text path/to/where/ever/you/can/write/output/all-domains path/to/where/ever/you/can/write/output/gephi path/to/where/ever/you/can/write/spark-jobs
• Adapt the script below:

import io.archivesunleashed._
import io.archivesunleashed.app._
import io.archivesunleashed.matchbox._
sc.setLogLevel("INFO")

val input = "/tuna1/scratch/i2milligan/warcs.archive-it.org/cgi-bin/getarcs.pl/*.gz"

val output1 = "/tuna1/scratch/aut-issue-269/derivatives/all-domains"
val output2 = "/tuna1/scratch/aut-issue-269/derivatives/all-text"
val output3 = "/tuna1/scratch/aut-issue-269/derivatives/gephi"

RecordLoader.loadArchives(input, sc).map(r => ExtractDomain(r.getUrl)).countItems().saveAsTextFile(output1)

RecordLoader.loadArchives(input, sc).keepValidPages().map(r => (r.getCrawlDate, r.getDomain, r.getUrl, RemoveHTML(r.getContentString))).saveAsTextFile(output2)

val links = RecordLoader.loadArchives(input, sc).keepValidPages().map(r => (r.getCrawlDate, ExtractLinks(r.getUrl, r.getContentString))).flatMap(r => r._2.map(f => (r._1, ExtractDomain(f._1).replaceAll("^\\s*www\\.", ""), ExtractDomain(f._2).replaceAll("^\\s*www\\.", "")))).filter(r => r._2 != "" && r._3 != "").countItems().filter(r => r._2 > 5)
WriteGraphML(links, output3)

sys.exit

Current Results

With this PR patch:

• /tuna1/scratch/aut-issue-269/derivatives/all-domains or /tuna1/scratch/aut-issue-269/derivatives/all-domains.txt (a combined version of all files in /tuna1/scratch/aut-issue-269/derivatives/all-domains)
• /tuna1/scratch/aut-issue-269/derivatives/gephi (doesn't contain backslash anymore, proper domain seetorontonow.canada-booknow.com has been extracted from URL)

Without this PR patch (master branch):

• /tuna1/scratch/aut-issue-269/derivatives/all-domains-without-patch or /tuna1/scratch/aut-issue-269/derivatives/all-domains-without-patch.txt (combined version)
• /tuna1/scratch/aut-issue-269/derivatives/gephi-without-patch (contains backslash as in URL seetorontonow.canada-booknow.com\booking_results.php)

Interested parties

@lintool @ianmilligan1 @ruebot @greebie