Some more on the modularity and SPI topic:

• https://groups.google.com/d/msg/dataverse-community/Nc8tX0s8lo8/pphF_LKUAQAJ
• https://groups.google.com/d/msg/dataverse-community/Nc8tX0s8lo8/WVffZL-gAQAJ

Also, https://projects.iq.harvard.edu/dcm2017/agenda shows " SPIs and Modularity: Are you interested in learning about Dataverse's emergent modular architecture?"

• We'll limit this to the file level for now
• Towards to goal of small chunks, the scope of this issue is that we want an installation to be able to run Data Explorer and TwoRavens in parallel.
• @scolapasta will edit this to add additional details

Repeating part of comment from #4230:

Besides the more obvious dynamic text and links we need for external tools, a couple of aspects that we need to handle in a modular way are:

When to show the links to the external tool
How to handle the link to the external tool (i.e. wha dynamic info to send)
In both these cases, we can think of building these in incremental ways, with each step building on the previous step, building from simplest, but supporting fewer tools, to most complex, but also most flexible.

For when to show the link, we have 3 steps:
A. simple list of tools, will only support one operation (i.e. Configure)
B. Add "Operation" to list, in order to support multiple operations (i.e. Configure vs. Explore)
C. Add "Subject of Operation" in order to support operations on different kind of objects (i.e. tools that work on tabular files, like DataExplorer, vs tools that work on geospatial, like World Map.

For how to handle link we similarly have 3 steps:

1. Dictate the exact parameter names and values that an external tool will need.
2. Define parameters dynamically from a list of "reserved" terms that we define.
3. Define individual "Handler" Java classes and SPI interface, in order to handle any tool.

In order to support Modular Explore for Two Ravens and DataExplore, the minimum we technically need is A1. However, since we are working on Modular Configure first, that bumps us to B1. In addition, in this scenario we would need to contact the external tools and have them change their parameters and we would rather be more flexible. So the decision is that the definition for done for this issue is B2.

Note, that we would still not be able to support geoConnect / WorldMap in a modular way yet, until we support at least C2, but per the previous comment, that is out of scope for this issue.

Thanks @scolapasta for taking the lead on breaking this up before next week's backlog grooming session.

@michbarsinai I pinged you in Slack because I said we'll need to write a more intelligent parser. Please see 272b4c2 for a terrible kludge and tests with examples of what we need. Thanks.

@jggautier after standup we talked about if there's a chance I messed up download counts by differentiating between "TwoRavens" and "Data Explorer" or whatever. I don't think so. As I hoped it looks like the code is simply counting rows like this: "select count(o.id) from GuestbookResponse o where o.datafile_id = " + dataFileId. That's from a method called getCountGuestbookResponsesByDataFileId in GuestbookResponseServiceBean.

@scolapasta that's fine. As you may have noticed, we dragged this issue into QA this morning during standup. Maybe we could tell third-party developers that they need to ask for a UUID from us for their application to be included in our app store (or a least be listed in the guides). I'm still trying to control the scope of this issue.

@kcondon I hope this helps:

Changes to consider testing:

• TwoRavens: TwoRavens has been converted into an "external tool". Part of the installation instructions have been adjusted to include a curl command operating on a JSON file. Two database settings have been removed: :TwoRavensTabularView and :TwoRavensUrl. Note that the "subset" functionaly still has TwoRavens branding but is independent of TwoRavens.
• WorldMap: The logic for when to show the WorldMap button has been adjusted to be aware of the presence of external tools.
• Configure button: The Modular Configure code (#4230) has been refactored and should be retested. Note that external tools can now have a type "configure" vs "explore" as noted in the guides.
• Download counts and types: Instead of "Explore" we now record the names of the external tools. The rules around download counts should not have changed. I'm not sure if they are documented or not.
• Guides: The guides have been adjusted to mention external tools as appropriate but further work will be done for #4429 some day.

To think about:

• What is the definition of done for integrating Data Explorer into Dataverse (#4249)? Should we go ahead and test Data Explorer now? It seems to work for me. 😄

Known issues:

• When you launch an external tool from a popup the tool will take over the browser window rather than opening in a new tab. When there is no popup, the tool launches in a new tab, which is the desired behavior. This bug exists in production for TwoRavens and has been discussed with Gustavo, Steve, and Mike but we don't have a fix. The inconsistency is unfortunate.

I just gave a demo to of the code as of c236079 to @landreev @djbrooke @TaniaSchlatter @mheppler @dlmurphy @kcondon @jggautier @matthew-a-dunlap and @pameyer (thanks!). The code is on https://dev1.dataverse.org and the guides are at http://guides.dataverse.org/en/3657-tworavens-as-modular-explore/installation/external-tools.html

Notes to self:

1. db update script is required to run this.
2. Significant doc changes with instructions on how to add external tools, including new api endpoints?

• Revised popup title to "Dataset Terms" for download/explore workflows on dataset/file pgs.

@pdurbin to respond to your App store comparison, yes the name is important, but behind the scenes what is used (at least in Android) is the package name. Beyond multiple tools with the same name, another concern is the same tool with a changing name, i.e having two rows that should map to the same thing, but don't (or when we internationalize and remove the name from the db).

I generally prefer to tackle issues which would require future database updates that modify data (as opposed to structure) preemptively. I don't see this as an expansion of scope (rather as a good coding practice); the expansion of scope already occurred when we decided to track the specific tool.

@scolapasta as I mentioned, I'm interested in controlling the scope of this issue. I'm happy to revert c236079 to make it so we go back to persisting "Explore" rather than "Two Ravens" or "Data Explorer" or whatever. That commit was so recent I expect it will revert cleanly and easily.

@djbrooke in the demo you seemed to like that individual tools are being tracked. What would you like?

Good discussion, thanks. I understand the reasoning for IDs instead of names, but let's keep this moving for now. On to QA!

Issues found so far:
-last line in db update script is missing a ;

@kcondon whoops, good catch. I added the missing semicolon in 92e0a43

As we discussed after standup yesterday there are technically no new API endpoints but the "manifest" format (JSON) has changed in that external tool authors must specify if their tool should appear under the "Explore" button or the "Configure" button.

Thanks. So this is not new?
curl -X POST -H 'Content-type: application/json' --upload-file twoRavens.json http://localhost:8080/api/admin/externalTools

Well, twoRavens.json is new (in the future, we'd like the TwoRavens team to host this file and #4429 is related) but the API endpoint is not new. It shipped with 4.8.5 and is documented at http://guides.dataverse.org/en/4.8.5/installation/external-tools.html#making-an-external-tool-available-in-dataverse

@kcondon just found a bug (thanks!) where two "Explore" buttons were being displayed side by side (one a dropdown and one a non-drop down). I fixed this in c6c0c87 and left a detailed comment about the fix in the commit.

x-Seeing two explore buttons on a shape file that has already been mapped.

Known issues:

• When you launch an external tool from a popup the tool will take over the browser window rather than opening in a new tab. When there is no popup, the tool launches in a new tab, which is the desired behavior. This bug exists in production for TwoRavens and has been discussed with Gustavo, Steve, and Mike but we don't have a fix. The inconsistency is unfortunate.

This is now being tracked at #4742.