Google App Engine [app]s - We are all on the same quota!

Question

Noticing a couple of "This IP has exceeded the request-per-day limit" errors on StackPrinter log, I asked for a verification and the prompt answer was:

There are a number of [app]s using Google App Engine, like StackPrinter.

Unfortunately, you're all using the same pool of IPs. Thus, you're all on the same quota.

If you can introduce a proxy or forwarder of some manner to your [app], you can solve this problem.

In few words: Google App Engine app[s] are doomed.

What?
I seriously thought that the quota check was by IP+AppKey.
Let's see the faq:

What are the API request limits?

A single IP address can only make a certain number of API requests per day, depending on the presence of a valid API key.

Although this seems quite clear, this sentence does not cover the case where the API keys, consumed by the same IP, are more than one.

Looking at "my" StackPrinter quota right now I see: x-ratelimit-current - 4486
Last evening was above 9000 and during the night there were just 50/60 prints; this sadly confirm the "shared quota" problem.

People have worked hard to rig up StackApps [App]s hosted on Google App Engine.
Here are some example:

• Stack Exchange Notifier Chrome Extension
• StackGuru
• SkeetStalker
• StackAthlon
• Not a real question
• StackPrinter

This is a serious limitation of the API usage and without a proper action, our Gae Applications will starve in no time.

Proposal:

• Is it possible to move the IP quota check to a more fair IP+Key or IP+AppId or IP+User-Agent or KeyOnly check?
• Could "authenticated requests" be a solution?
• Could you raise the 10.000 threshold for the Google pools of IP to temporarily tampon these "This IP has exceeded the request-per-day limit" errors?

One last thing: the proxy idea is useless because it is slow and does not scale with Google App Engine.

^{* Feel free to edit this question to add better ideas\proposals.}

---

EDIT:
One Google App Engine IP is gone right now; a call against Stack Overflow api returns a dreadful x-ratelimit-current - 0

---

EDIT:
In response to Kevin's answer:
well, I think you are probably right in most of your thinking; I've drilled down into this topic a little bit further and all the APIkey-oriented checks I proposed (suggested more by the heat than by careful research) terribly suck.
Ok, they could be good for a toy API but not for a long term awesome project like the Stack Exchange API.

That said, I have to blame myself to have totally overlooked this aspect :).
In my defense, as previously said, faq is not particularly clear about this aspect and your answer given to this precog brilliant question was pretty misleading:

Sky: Hey, watch out! If apps are hosted on the same IP they could have problem.
Kevin: We've indicated in the past that the per-day limit for a key can be increased if need is demonstrated.

This wrong concept of "Problem with rate limit? We can increase it, no problem at all!" is sneakily reinforced in the faq:

What should I do if I need more requests per day?

Certain types of applications - services and websites to name two - can legitimately have much higher per-day request requirements than typical applications. If you can demonstrate a need for a higher request quota, contact us.

Wrong! This can be applied just for the "rich" lucky guy who has his own dedicated IP!
And remember, even moving your poor bleeding [app]s from the Cloud to Dreamhost does not solve the problem (although is less probable to have an hog request [app] on your same IP).
What is the lesson I learnt..
an API, to be a state of the art serious API to play with (for a website), must provide Authentication*; without this feature, you are asking for trouble.

I recommend you to update the faq and be clear about this danger.

What about our Google App Engine application?
Do we really need to wait an Authentication feature to port our app and move away from the Cloud?
Waiting for this feature, and still having a relatively small number of applications, is it possible to relax the key limit constraint a little?
Is a TLD+IP check a possible solution?

ok, too much thoughts.. thanks for your patience.

EDIT:
working solution here

_{* Oauth is not the only available solution; take Amazon Web Services for example, they allow authenticated REST calls passing apikey+timestamp+signature inside the query string for each request.}

Answer 1

I was wondering if the scenario of a server based app that lives in a hosted environment, where a single IP can be shared by many web sites.

As I understand the rate-limit as it stands, every site on the shared host IP, regardless of api key, will be sharing the same rate-limit of 10,000 per site per day.

more @ rate-limit per endpoint per IP VS. server apps in hosted environments

Looks like everyone in GAE either

• gets unlimited rate-limit
• rate-limit engine checks for key + ip + top level domain (or some variation) of the referrer as described in linked post
• GAE is no longer a viable platform for API apps

NOTE: this issue is not limited to the current fire - anyone on a shared platform, e.g. standard shared webhosting, is vulnerable to this issue.

Answer 2

The proposed "solutions" to this are all terrible*.

Ip+Key

Trivially forged.

IP+AppId

Trivially forged.

IP+User-Agent

Trivially forged.

KeyOnly

Surprise surprise, trivially forged.

---

Now, authentication would give us a non-trivially forged token to throttle on. This is a planned improvement in a future API version, however this workaround would require [app]s to force all users to login before using them (if our goal was to do away with IP based throttles altogether).

This problem will exist for any user-agnostic (ie. non-authenticating) [app] which shares its IP forever.

Essentially, [app]s on Google App Engine (or any other cloud service) are sitting behind a 1000-to-1 users-to-IPs order NAT, and NAT breaks the internet. If you want to avoid this problem, you'll need to get a dedicated IP somehow.

If there were a viable work around for this problem, we'd implement it, but frankly the issue is so far upstream of us there's nothing we can do.

Some other APIs with IP based throttling (in part, at least):

• Twitter
• Facebook (undocumented, approx. ~100-300 calls / 24 hours unauthenticated)
• Yahoo

There are many others, these are just a few which are easy to find references for.

_{They are all, in fact, variations of security through obscurity.}

Answer 3

I had a GAE app that queried hourly a few sites for traffic stats. I've brought it down. Thanks for the warning and enjoy your 72 extra calls per day :)

Answer 4

All this issues also apply to request throttling limits (30 requests over 5 seconds).
I'm sure that I'm throttling requests from my app properly according to api terms of usage based on a distributed lock with memcached, but still frequently getting following error "Too Many Requests, We're sorry...There are an unusual number of requests coming from this IP address.To protect our users, we can't process any more requests from this IP address right now."

Answer 5

A possible solution will be to whitelist some app keys, the whitelisting means that the ip is not checked and only the app key is check.