Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Argentine BNM #2194

Merged
merged 4 commits into from Jun 22, 2020
Merged

Add Argentine BNM #2194

merged 4 commits into from Jun 22, 2020

Conversation

@adam3smith
Copy link
Collaborator

adam3smith commented Jun 21, 2020

Requested on Twitter: https://twitter.com/Ronconi/status/1274087752905166861

Easily accessible MARC XML

function scrape(urls) {
for (let url of urls) {
// Z.debug(url);
// eslint-disable-next-line loopfunc

This comment has been minimized.

Copy link
@adam3smith

adam3smith Jun 21, 2020

Author Collaborator

@dstillman just making sure disabling this rule is OK in this context? My understanding is that it's intended to avoid re-defining a function n times, which this doesn't do, but wasn't 100% sure (the reason there's a loop here is to avoid unnecessarily downloading the catalog item page from search results.)

This comment has been minimized.

Copy link
@dstillman

dstillman Jun 21, 2020

Member

Did you mean no-loop-func? loopfunc is from JSHint, I think.

This is OK — the function doesn't reference any external variables, so there's nothing to get confused by here — but it does technically create multiple anonymous functions for no reason. You could just define var handler = function (text) { … }; before the loop and pass handler to each invocation.

adam3smith added 2 commits Jun 21, 2020
};
for (let url of urls) {
// Z.debug(url);
ZU.doGet(constructMARCurl(url), handler);

This comment has been minimized.

Copy link
@dstillman

dstillman Jun 21, 2020

Member

Actually, there's a more fundamental problem with this. When doGet() is used inside a loop, it's called simultaneously as many times as there are loops, resulting in many simultaneous requests (possibly limited to 6 by the network stack). doGet() can take an array of URLs, and it will automatically serialize those rather than trying to make all the requests at once. (In the future, it (or its successor) could also gently parallelize them.) So you can just remove the loop here and call doGet() once with an array of constructed URLs, with a single function for the callback.

I suspect this mistake is made in many translators. It's actually done in the Google Scholar translator, which might be the reason for the frequent throttling we see there.

This comment has been minimized.

Copy link
@adam3smith

adam3smith Jun 22, 2020

Author Collaborator

That makes sense -- I think I got this right in adff864 ?

@dstillman
Copy link
Member

dstillman commented Jun 22, 2020

Yep, that looks good.

@adam3smith adam3smith merged commit 6cb7001 into zotero:master Jun 22, 2020
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.