Thanks for the report! Here's my general update on the situation that I've posted in Discord:
Hello, general app PSA for anyone worried about Cloudflare on AO3: I'm going to wait for the dust to settle on AO3 figuring out their new rate limits, but I highly suspect we will Be Okay in the end. The app obeys the (old) rate limits AO3 sets for its human users and will obey any new ones.
It's highly likely that it got hit by a stricter rate limit today the same way a whole bunch of humans also got rate limited doing perfectly normal things. Once rate limited, the app was probably put in the Timeout Sin Bin for anywhere up to an hour, which probably compounded when anyone subsequently tried to scrape and went "oh no!" We don't currently have rate limiting language telling end users in the app that the app got rate limited because the app, by design, was never ever meant to hit AO3s limits... ever. But those have changed, is all. Reports say that at least one or two scrapes have succeeded recently, so things may be calming down a bit.
Once things settle down For Real, I'll update the language on the app to watch out for rate limiting, and I imagine that the worst that will happen is that there might be times where we're less able to scrape if AO3 decides to use, say, dynamic rate limiting at hairy times. I think we'll be okay, but I'll keep my eye out.
If the app has outages over the next little while, just go grab a (long) cup of tea - re-scraping will only make things worse and will likely end up with the exchange you're rescraping getting previously-scraped requests potentially cleared out. The app does not keep database updates and cannot restore lost scrapes. I'm, alas, at work and can't get the updated handling in yet.
no subject
Thanks for the report! Here's my general update on the situation that I've posted in Discord:
Hello, general app PSA for anyone worried about Cloudflare on AO3: I'm going to wait for the dust to settle on AO3 figuring out their new rate limits, but I highly suspect we will Be Okay in the end. The app obeys the (old) rate limits AO3 sets for its human users and will obey any new ones.
It's highly likely that it got hit by a stricter rate limit today the same way a whole bunch of humans also got rate limited doing perfectly normal things. Once rate limited, the app was probably put in the Timeout Sin Bin for anywhere up to an hour, which probably compounded when anyone subsequently tried to scrape and went "oh no!" We don't currently have rate limiting language telling end users in the app that the app got rate limited because the app, by design, was never ever meant to hit AO3s limits... ever. But those have changed, is all. Reports say that at least one or two scrapes have succeeded recently, so things may be calming down a bit.
Once things settle down For Real, I'll update the language on the app to watch out for rate limiting, and I imagine that the worst that will happen is that there might be times where we're less able to scrape if AO3 decides to use, say, dynamic rate limiting at hairy times. I think we'll be okay, but I'll keep my eye out.
If the app has outages over the next little while, just go grab a (long) cup of tea - re-scraping will only make things worse and will likely end up with the exchange you're rescraping getting previously-scraped requests potentially cleared out. The app does not keep database updates and cannot restore lost scrapes. I'm, alas, at work and can't get the updated handling in yet.
Super long psa sorry haha.