Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upAdd Argentine BNM #2194
Add Argentine BNM #2194
Conversation
function scrape(urls) { | ||
for (let url of urls) { | ||
// Z.debug(url); | ||
// eslint-disable-next-line loopfunc |
This comment has been minimized.
This comment has been minimized.
adam3smith
Jun 21, 2020
Author
Collaborator
@dstillman just making sure disabling this rule is OK in this context? My understanding is that it's intended to avoid re-defining a function n times, which this doesn't do, but wasn't 100% sure (the reason there's a loop here is to avoid unnecessarily downloading the catalog item page from search results.)
This comment has been minimized.
This comment has been minimized.
dstillman
Jun 21, 2020
Member
Did you mean no-loop-func
? loopfunc
is from JSHint, I think.
This is OK — the function doesn't reference any external variables, so there's nothing to get confused by here — but it does technically create multiple anonymous functions for no reason. You could just define var handler = function (text) { … };
before the loop and pass handler
to each invocation.
}; | ||
for (let url of urls) { | ||
// Z.debug(url); | ||
ZU.doGet(constructMARCurl(url), handler); |
This comment has been minimized.
This comment has been minimized.
dstillman
Jun 21, 2020
Member
Actually, there's a more fundamental problem with this. When doGet()
is used inside a loop, it's called simultaneously as many times as there are loops, resulting in many simultaneous requests (possibly limited to 6 by the network stack). doGet() can take an array of URLs, and it will automatically serialize those rather than trying to make all the requests at once. (In the future, it (or its successor) could also gently parallelize them.) So you can just remove the loop here and call doGet() once with an array of constructed URLs, with a single function for the callback.
I suspect this mistake is made in many translators. It's actually done in the Google Scholar translator, which might be the reason for the frequent throttling we see there.
This comment has been minimized.
This comment has been minimized.
adam3smith
Jun 22, 2020
Author
Collaborator
That makes sense -- I think I got this right in adff864 ?
Yep, that looks good. |
adam3smith commentedJun 21, 2020
Requested on Twitter: https://twitter.com/Ronconi/status/1274087752905166861
Easily accessible MARC XML