Tests exist for this, but are disabled because there is no NER classifier in the repo. Is it possible to include the NER classifier in-house, or is there a licensing issue with doing so?

Looks like I shouldn't have removed it here 0ec8ab1, and I don't see english.all.3class.distsim.crf.ser.gz in the Git history at all. But we have it over here with a notice. I'm happy to restore example.txt, but I'm unsure about how to handle english.all.3class.distsim.crf.ser.gz.

@lintool I assume english.all.3class.distsim.crf.ser.gz is over there and not in it the aut or warcbase repo because of licensing? That correct? If so, how do you want to handle this?

I'm interested that NER seems to be included in Tika, which we all ready use for the PDFParser. I could take this on as a November project I think. It may mean some configuration.

I think NER is basic GPL, so I don't think it's a licensing issue.

Our thought back when we designed the entity extractor was less a licensing issue, but more not wanting the repository to balloon out of control in terms of size. english.all.3class.distsim.crf.ser.gz is ~ 33MB in size, and it's just one language, so the thought was that if a user needed it then they should download it from a separate resources repository.

It was a general aversion to having too much cruft build up within the repo. @lintool

Yes, that's exactly it. My usual treatment is to create a separate repo, e.g., aut-resources and then ask the user to wget the data...

We have tika-parsers in our pom.xml. There is a version of NER in there -- maybe we could use that? https://wiki.apache.org/tika/TikaAndNER

@greebie Maven isn't going to pull that that classifier file from that I can tell.

Okay -- back to square one. I think we are stuck not being able to test this one.

Although I may be able to mock a NERClassifier .. let's keep this open until I can figure out.

NER functions require a classifier file that is not included in AUT. Potential solutions:

develop a generic NER classifier for test purposes. See #53 & #52.