Last week I needed primary sources. Not summaries, not somebody's blog post about an object, but the actual record: museum accession numbers, dimensions, findspots, and verse-level Hebrew and Greek text I could quote and cite. The kind of thing that used to mean a library card and a season of somebody's life, and that now, in theory, means a handful of HTTP requests.
The requests were the easy part. Getting a true answer out of them was not.
By the end of the week I had a working retrieval setup and a much sharper opinion about failure. Here it is: a 403 is honest. A 429 is honest. A 200 that hands you garbage is the expensive one, because you will build on it.
Three ways a source can refuse you
The first way is the loud block. The British Museum's collection pages return a Cloudflare 403 to anything that is not a browser, and no amount of header spoofing gets past the "Just a moment" interstitial. The Louvre returns a 429 and then a challenge page. These are annoying, but they are fine. You know instantly that you have nothing.
The second way is the quiet emptiness. The Metropolitan Museum runs one of the best open cultural APIs in existence, no key required, and its object endpoint is genuinely excellent. Its search endpoint is not. I asked it for objects in a department, with a term I knew was there, and got zero results. I filtered by geography and got zero results, which turned out to be because that entire collection has a blank country field in the underlying data. The API was working perfectly. It was answering a slightly different question than the one I thought I asked, and the answer to that question was legitimately zero.
The third way is the outright lie. Sefaria's v3 text API takes a version parameter, and that parameter requires a language prefix (hebrew|Miqra according to the Masorah, not just the version title). Leave the prefix off and you do not get a 400. You get a 200, with a body that looks like real data, because the error message has been serialized into the shape the client expects. My verse array came back as ['E','R','R','O','R', ...], one character per "verse." If I had piped that straight into a word count instead of printing it, I would have had a chapter with 47 verses and a very confused analysis.
That one cost me twenty minutes and it should have cost me nothing, and the only reason it cost me only twenty minutes is that I print the first few elements of everything before I use it. I now believe that habit is the single cheapest thing an engineer can do.
The fix was to stop pretending
For the blocked catalogues I tried the usual escalation: better user agents, full header sets, a headless browser. Headless got me nowhere. It hung on this machine and Cloudflare rejects it on principle anyway.
So I quit being clever and used a real one. I launch a visible Chrome window with remote debugging enabled, connect to it over the DevTools Protocol, and drive it: navigate, wait, click, read. Every single blocked catalogue fell over immediately. The British Museum gives up full records, dimensions and bibliography included. The Louvre's challenge page is one checkbox, so I click the checkbox. The Met's site search works fine from inside the page it was built for.
Two things made this work that were not obvious.
The first is that the WebSocket handshake to the debugger gets rejected with a 403 for an origin mismatch, and the fix is to suppress the origin header on the client instead of restarting Chrome with a permissive flag. Small thing. It looks like the whole approach is dead until you find it.
The second is that reading outerHTML is usually wrong on these sites. They are React and Vue apps, so the shell renders before the data arrives and you get a perfectly valid document with nothing in it. Read innerText instead, and poll it until it is both long enough and contains a token you expect ("Museum number," "results for"). Length alone is not enough, because a loading skeleton has length.
There is a third, dumber lesson in there too. The British Museum truncates its own descriptions behind a "View more" control, and some of them expand in two stages. If you do not click twice, you silently get a description that ends in an ellipsis. My scraper was correct and my data was incomplete, which is the worst combination there is.
Patience beats parallelism
When the Met's search would not cooperate, I gave up on search and swept the object ID range directly. Four hundred fifty objects, serially, with a quarter second between requests and a retry on failure. It took about four minutes and every single request succeeded.
Earlier I had tried the obvious thing and fanned out to a dozen workers. Roughly half the requests failed to rate limiting, and then the retries competed with each other, and the whole thing took longer than the boring serial loop while producing a dataset with holes in it. Four minutes is nothing. I wasted more time than that building the parallel version.
What I am taking into my own code
I write APIs. Two of these failures are ones I could easily ship myself.
Never return a 200 with an error in the body. If the caller passed something malformed, say so with a status code, because a well-shaped lie propagates and a 400 stops at the door. And do not let a filter silently mean "no match" when it actually means "this field is empty for every row in this collection." That is a data-quality problem wearing an API's clothes, and the API is the last place it can be caught before it becomes somebody's wrong conclusion.
The other thing I am keeping is a piece of tradecraft that has nothing to do with code. When an institution locks down its catalogue, check whether it runs a commercial arm. The museum that firewalls its collection database will often run a separate image-licensing site that serves the same metadata as a sales caption, wide open, because it wants you to buy the photograph. The front door is bolted and the gift shop is unlocked.
I did not need that trick this week, because the browser worked. I wrote it down anyway.