2

I'm experiencing a weird problem on StackPrinter hosted on Google App Engine.
It seems that the application can't reach several endpoints like Super User or Server Fault; Stack Overflow and other endpoints still works.

This WORK (Stack Overflow)
http://www.stackprinter.com/export?question=478760&format=HTML&service=stackoverflow&linktohome=false

This DOES NOT WORK (Super User)
http://www.stackprinter.com/export?question=52483&format=HTML&service=superuser&linktohome=false

WORKING sites list:
Stack Overflow
English
Meta Stack Overflow
Programmers
Ubuntu
CSTheory
Webmasters
Photography
OnStartups
Statistical
Wordpress
Phisics
Role Playing Tex
Homebrew
Unix

NOT WORKING sites list:
Gaming
Apple
Super User
Server Fault
Stack Apps
Web Applications
Ui
Android
Electronics and Robots
Mathematics
Cooking
GIS

It works on StackPrinter mirror hosted on Dreamhost; that indicates that it is probably a problem of connectivity from Google data centers to those particular domains/subdomains; looks weird though.

@Kevin, could you please verify if StackPrinter's calls reach SE server properly?
This can be easily tested on Top Printed page.

Anyone suffering this problem?

EDIT:
Follow this post on App Engine Google Groups.

EDIT2:
Stackauth site does not work too

EDIT3:
uhm, it still does not work for serverfault endpoint.
http://www.stackprinter.com/export?question=261&format=HTML&service=serverfault&linktohome=false

6
  • @system yes, indeed, my projects experiencing the same problems. Commented Nov 12, 2010 at 9:32
  • @Vladislav looks something like this Commented Nov 12, 2010 at 9:38
  • @system me too!!!!!1 Commented Nov 12, 2010 at 12:25
  • @Shay pain shared is pain divided :) Commented Nov 12, 2010 at 13:21
  • @system my app runs on auto pilot, I didn't notice this error and it is almost exploded Commented Nov 12, 2010 at 13:45
  • StackGuru is back online :) Commented Nov 15, 2010 at 16:08

1 Answer 1

4

We're fixing this, shouldn't be too much longer (may even have been done by the time you read this).

Basically, this is another consequence of using a cloud computing service.

We've got a bunch of crummy (and technically illegal, due to attribute violations) sites out there that like to scrape us. Now, we realize its a fools errand to try and prevent people from copy our content (and we're 100% A-OK and super happy with people who copy us and attribute correctly) but the truly egregious ones that are showing up in our logs get IP bans.

We've gotten pretty good at this, so now these sites can't just naively wget us; but of course they still try. They could build some distributing network of crawlers or get fancy with their scraping techniques, but that would require effort and money (and since the business model is to slap low rate ads on other people's content, money isn't easy).

Instead, they've been turning to GAE apparently. Recently a couple of IPs got banned due to truly egregious scraping of the sites (not the API), which unintentionally has also blocked some API access from GAE.

We're lifting those bans (and will just live with the scraping for now) until we get some additional infrastructure so we can block subsets of the network, rather than the all or nothing options we have now.

1
  • 1
    +1 for the thorough explanation (and pragmatic approach), thanks! Have you looked into whether this might be solvable directly with the GAE team? Their TOS most definitely forbids this kind of abuse, either way I'm sure they won't like their platform getting discredited by such activities. Reporting each case manually via code.google.com/support/bin/… might get too tedious of course, but maybe this can be automated somehow? Eventually a site network your size might qualify for a VIP solution or so as well? Commented Nov 13, 2010 at 12:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .