Take the 2-minute tour ×
Stack Apps is a question and answer site for apps, scripts, and development with the Stack Exchange API. It's 100% free, no registration required.

I'm experiencing a weird problem on StackPrinter hosted on Google App Engine.
It seems that the application can't reach several endpoints like Super User or Server Fault; Stack Overflow and other endpoints still works.

This WORK (Stack Overflow)
http://www.stackprinter.com/export?question=478760&format=HTML&service=stackoverflow&linktohome=false

This DOES NOT WORK (Super User)
http://www.stackprinter.com/export?question=52483&format=HTML&service=superuser&linktohome=false

WORKING sites list:
Stack Overflow
English
Meta Stack Overflow
Programmers
Ubuntu
CSTheory
Webmasters
Photography
OnStartups
Statistical
Wordpress
Phisics
Role Playing Tex
Homebrew
Unix

NOT WORKING sites list:
Gaming
Apple
Super User
Server Fault
Stack Apps
Web Applications
Ui
Android
Electronics and Robots
Mathematics
Cooking
GIS

It works on StackPrinter mirror hosted on Dreamhost; that indicates that it is probably a problem of connectivity from Google data centers to those particular domains/subdomains; looks weird though.

@Kevin, could you please verify if StackPrinter's calls reach SE server properly?
This can be easily tested on Top Printed page.

Anyone suffering this problem?

EDIT:
Follow this post on App Engine Google Groups.

EDIT2:
Stackauth site does not work too

EDIT3:
uhm, it still does not work for serverfault endpoint.
http://www.stackprinter.com/export?question=261&format=HTML&service=serverfault&linktohome=false

share|improve this question
    
@system yes, indeed, my projects experiencing the same problems. –  Vladislav Tserman Nov 12 '10 at 9:32
    
@Vladislav looks something like this –  systempuntoout Nov 12 '10 at 9:38
    
@system me too!!!!!1 –  Shay Erlichmen Nov 12 '10 at 12:25
    
@Shay pain shared is pain divided :) –  systempuntoout Nov 12 '10 at 13:21
    
@system my app runs on auto pilot, I didn't notice this error and it is almost exploded –  Shay Erlichmen Nov 12 '10 at 13:45
    
StackGuru is back online :) –  Shay Erlichmen Nov 15 '10 at 16:08

1 Answer 1

up vote 4 down vote accepted

We're fixing this, shouldn't be too much longer (may even have been done by the time you read this).

Basically, this is another consequence of using a cloud computing service.

We've got a bunch of crummy (and technically illegal, due to attribute violations) sites out there that like to scrape us. Now, we realize its a fools errand to try and prevent people from copy our content (and we're 100% A-OK and super happy with people who copy us and attribute correctly) but the truly egregious ones that are showing up in our logs get IP bans.

We've gotten pretty good at this, so now these sites can't just naively wget us; but of course they still try. They could build some distributing network of crawlers or get fancy with their scraping techniques, but that would require effort and money (and since the business model is to slap low rate ads on other people's content, money isn't easy).

Instead, they've been turning to GAE apparently. Recently a couple of IPs got banned due to truly egregious scraping of the sites (not the API), which unintentionally has also blocked some API access from GAE.

We're lifting those bans (and will just live with the scraping for now) until we get some additional infrastructure so we can block subsets of the network, rather than the all or nothing options we have now.

share|improve this answer
    
fixed! Thanks to you and to the sysadmins –  systempuntoout Nov 13 '10 at 9:31
1  
+1 for the thorough explanation (and pragmatic approach), thanks! Have you looked into whether this might be solvable directly with the GAE team? Their TOS most definitely forbids this kind of abuse, either way I'm sure they won't like their platform getting discredited by such activities. Reporting each case manually via code.google.com/support/bin/… might get too tedious of course, but maybe this can be automated somehow? Eventually a site network your size might qualify for a VIP solution or so as well? –  Steffen Opel Nov 13 '10 at 12:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .