Skip to main content
1 of 3
Kevin Montrose
  • 18.7k
  • 6
  • 35
  • 62

We're fixing this, shouldn't be too much longer (may even have been done by the time you read this).

Basically, this is another consequence of using a cloud computing service.

We've got a bunch of fairly scummy (and generally illegal, due to attribute violations) sites out there that like to scrape us. Now, we realize its a fools errand to try and prevent people from copy our content (and we're 100% A-OK and super happy with people who copy us and attribute correctly) but the truly egregious ones that are showing up in our logs get IP bans.

We've gotten pretty good at this, so now these sites can't just naively wget us; but of course they still try. They could build some distributing network of crawlers or get fancy with their scraping techniques, but that would require effort and money (and since the business model is to slap low rate ads on other people's content, money isn't easy).

Instead, they've been turning to GAE apparently. Recently a couple of IPs got banned due to truly egregious scraping of the sites (not the API), which unintentionally has also blocked some API access from GAE.

We're lifting those bans (and will just live with the scraping for now) until we get some additional infrastructure so we can block subsets of the network, rather than the all or nothing options we have now.

Kevin Montrose
  • 18.7k
  • 6
  • 35
  • 62