Sign up ×
Stack Apps is a question and answer site for apps, scripts, and development with the Stack Exchange API. It's 100% free, no registration required.

I'm experiencing a weird problem on StackPrinter hosted on Google App Engine.
It seems that the application can't reach several endpoints like Super User or Server Fault; Stack Overflow and other endpoints still works.

This WORK (Stack Overflow)

This DOES NOT WORK (Super User)

WORKING sites list:
Stack Overflow
Meta Stack Overflow
Role Playing Tex

NOT WORKING sites list:
Super User
Server Fault
Stack Apps
Web Applications
Electronics and Robots

It works on StackPrinter mirror hosted on Dreamhost; that indicates that it is probably a problem of connectivity from Google data centers to those particular domains/subdomains; looks weird though.

@Kevin, could you please verify if StackPrinter's calls reach SE server properly?
This can be easily tested on Top Printed page.

Anyone suffering this problem?

Follow this post on App Engine Google Groups.

Stackauth site does not work too

uhm, it still does not work for serverfault endpoint.

share|improve this question
@system yes, indeed, my projects experiencing the same problems. –  Vladislav Tserman Nov 12 '10 at 9:32
@Vladislav looks something like this –  systempuntoout Nov 12 '10 at 9:38
@system me too!!!!!1 –  Shay Erlichmen Nov 12 '10 at 12:25
@Shay pain shared is pain divided :) –  systempuntoout Nov 12 '10 at 13:21
@system my app runs on auto pilot, I didn't notice this error and it is almost exploded –  Shay Erlichmen Nov 12 '10 at 13:45
StackGuru is back online :) –  Shay Erlichmen Nov 15 '10 at 16:08

1 Answer 1

up vote 4 down vote accepted

We're fixing this, shouldn't be too much longer (may even have been done by the time you read this).

Basically, this is another consequence of using a cloud computing service.

We've got a bunch of crummy (and technically illegal, due to attribute violations) sites out there that like to scrape us. Now, we realize its a fools errand to try and prevent people from copy our content (and we're 100% A-OK and super happy with people who copy us and attribute correctly) but the truly egregious ones that are showing up in our logs get IP bans.

We've gotten pretty good at this, so now these sites can't just naively wget us; but of course they still try. They could build some distributing network of crawlers or get fancy with their scraping techniques, but that would require effort and money (and since the business model is to slap low rate ads on other people's content, money isn't easy).

Instead, they've been turning to GAE apparently. Recently a couple of IPs got banned due to truly egregious scraping of the sites (not the API), which unintentionally has also blocked some API access from GAE.

We're lifting those bans (and will just live with the scraping for now) until we get some additional infrastructure so we can block subsets of the network, rather than the all or nothing options we have now.

share|improve this answer
fixed! Thanks to you and to the sysadmins –  systempuntoout Nov 13 '10 at 9:31
+1 for the thorough explanation (and pragmatic approach), thanks! Have you looked into whether this might be solvable directly with the GAE team? Their TOS most definitely forbids this kind of abuse, either way I'm sure they won't like their platform getting discredited by such activities. Reporting each case manually via… might get too tedious of course, but maybe this can be automated somehow? Eventually a site network your size might qualify for a VIP solution or so as well? –  Steffen Opel Nov 13 '10 at 12:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .