Take the 2-minute tour ×
Stack Apps is a question and answer site for apps, scripts, and development with the Stack Exchange API. It's 100% free, no registration required.

alt text

Noticing a couple of "This IP has exceeded the request-per-day limit" errors on StackPrinter log, I asked for a verification and the prompt answer was:

There are a number of [app]s using Google App Engine, like StackPrinter.

Unfortunately, you're all using the same pool of IPs. Thus, you're all on the same quota.

If you can introduce a proxy or forwarder of some manner to your [app], you can solve this problem.

In few words: Google App Engine app[s] are doomed.

What?
I seriously thought that the quota check was by IP+AppKey.
Let's see the faq:

What are the API request limits?

A single IP address can only make a certain number of API requests per day, depending on the presence of a valid API key.

Although this seems quite clear, this sentence does not cover the case where the API keys, consumed by the same IP, are more than one.

Looking at "my" StackPrinter quota right now I see: x-ratelimit-current - 4486
Last evening was above 9000 and during the night there were just 50/60 prints; this sadly confirm the "shared quota" problem.

People have worked hard to rig up StackApps [App]s hosted on Google App Engine.
Here are some example:

This is a serious limitation of the API usage and without a proper action, our Gae Applications will starve in no time.

Proposal:

  • Is it possible to move the IP quota check to a more fair IP+Key or IP+AppId or IP+User-Agent or KeyOnly check?
  • Could "authenticated requests" be a solution?
  • Could you raise the 10.000 threshold for the Google pools of IP to temporarily tampon these "This IP has exceeded the request-per-day limit" errors?

One last thing: the proxy idea is useless because it is slow and does not scale with Google App Engine.

* Feel free to edit this question to add better ideas\proposals.


EDIT:
One Google App Engine IP is gone right now; a call against Stack Overflow api returns a dreadful x-ratelimit-current - 0


EDIT:
In response to Kevin's answer:
well, I think you are probably right in most of your thinking; I've drilled down into this topic a little bit further and all the APIkey-oriented checks I proposed (suggested more by the heat than by careful research) terribly suck.
Ok, they could be good for a toy API but not for a long term awesome project like the Stack Exchange API.

That said, I have to blame myself to have totally overlooked this aspect :).
In my defense, as previously said, faq is not particularly clear about this aspect and your answer given to this precog brilliant question was pretty misleading:

Sky: Hey, watch out! If apps are hosted on the same IP they could have problem.
Kevin: We've indicated in the past that the per-day limit for a key can be increased if need is demonstrated.

This wrong concept of "Problem with rate limit? We can increase it, no problem at all!" is sneakily reinforced in the faq:

What should I do if I need more requests per day?

Certain types of applications - services and websites to name two - can legitimately have much higher per-day request requirements than typical applications. If you can demonstrate a need for a higher request quota, contact us.

Wrong! This can be applied just for the "rich" lucky guy who has his own dedicated IP!
And remember, even moving your poor bleeding [app]s from the Cloud to Dreamhost does not solve the problem (although is less probable to have an hog request [app] on your same IP).
What is the lesson I learnt..
an API, to be a state of the art serious API to play with (for a website), must provide Authentication*; without this feature, you are asking for trouble.

I recommend you to update the faq and be clear about this danger.

What about our Google App Engine application?
Do we really need to wait an Authentication feature to port our app and move away from the Cloud?
Waiting for this feature, and still having a relatively small number of applications, is it possible to relax the key limit constraint a little?
Is a TLD+IP check a possible solution?

ok, too much thoughts.. thanks for your patience.

EDIT:
working solution here

* Oauth is not the only available solution; take Amazon Web Services for example, they allow authenticated REST calls passing apikey+timestamp+signature inside the query string for each request.

share|improve this question
    
I Have no big hopes that this issue would be resolved anytime soon. Even Twitter claims that the majority of cloud platforms, including Google App Engine, cannot be whitelisted to access their Search API due to absence of static IP. But I continue to hope that something would be done. –  Vladislav Tserman Oct 20 '10 at 9:43
1  
@Vladislav I totally agree that a cloud platform can't be whitelisted! But Twitter offers a way to bypass this problem using authenticated requests per app and/or user. –  systempuntoout Oct 20 '10 at 9:48
1  
DOOOOOOOOOOOOOOOOMED! –  Sky Sanders Oct 20 '10 at 17:17
3  
For reference, we (App Engine) include the App ID in the user-agent header, and we don't let users set the user-agent header on outgoing requests. If the IP comes from our pool (_netblocks.google.com), you can trust that the App ID in the user-agent is accurate. –  Nick Johnson Oct 28 '10 at 15:15
    
@Nick are you sure? AFAIK user-agent can be set since SDK version 1.2.1 . I remember this because I auto answered my own question on this topic 7 month ago here –  systempuntoout Nov 22 '10 at 13:41
    
@systempuntoout I'm positive - it's always been the case that although you can set part of the user-agent string, you can't remove your App ID from it. –  Nick Johnson Sep 5 '11 at 4:47

5 Answers 5

up vote 4 down vote accepted

I was wondering if the scenario of a server based app that lives in a hosted environment, where a single IP can be shared by many web sites.

As I understand the rate-limit as it stands, every site on the shared host IP, regardless of api key, will be sharing the same rate-limit of 10,000 per site per day.

more @ rate-limit per endpoint per IP VS. server apps in hosted environments

Looks like everyone in GAE either

  • gets unlimited rate-limit
  • rate-limit engine checks for key + ip + top level domain (or some variation) of the referrer as described in linked post
  • GAE is no longer a viable platform for API apps

NOTE: this issue is not limited to the current fire - anyone on a shared platform, e.g. standard shared webhosting, is vulnerable to this issue.

share|improve this answer
2  
totally forgot that brilliant precog question (that I had upvoted months ago). From what I see, just increasing the key limit does not solve the problem. In that scenario, a key hog app could continue to eat from other dishes without any problem leaving the other poor apps starving to death. –  systempuntoout Oct 20 '10 at 13:20
    
Marked as accepted because you already knew it would be a problem –  systempuntoout Oct 22 '10 at 11:05

The proposed "solutions" to this are all terrible*.

Ip+Key

Trivially forged.

IP+AppId

Trivially forged.

IP+User-Agent

Trivially forged.

KeyOnly

Surprise surprise, trivially forged.


Now, authentication would give us a non-trivially forged token to throttle on. This is a planned improvement in a future API version, however this workaround would require [app]s to force all users to login before using them (if our goal was to do away with IP based throttles altogether).

This problem will exist for any user-agnostic (ie. non-authenticating) [app] which shares its IP forever.

Essentially, [app]s on Google App Engine (or any other cloud service) are sitting behind a 1000-to-1 users-to-IPs order NAT, and NAT breaks the internet. If you want to avoid this problem, you'll need to get a dedicated IP somehow.

If there were a viable work around for this problem, we'd implement it, but frankly the issue is so far upstream of us there's nothing we can do.

Some other APIs with IP based throttling (in part, at least):

There are many others, these are just a few which are easy to find references for.

They are all, in fact, variations of security through obscurity.

share|improve this answer
1  
How about white listing certified [app]s? Those apps will be throttled based on key only. –  Shay Erlichmen Oct 20 '10 at 18:50
1  
@Shay - how do we identify those [app]s? They'd have to have a dedicated IP already, or its just a (historically, short) matter of time before the key is leaked. Now, if an [app] is already on a dedicated IP we can raise the quota (this has already been done for some [app]s, in fact). –  Kevin Montrose Oct 20 '10 at 18:53
3  
@Kevin why do you think that the key will leak? If the code runs on the server no one will see it and its the best interest of the key owner to take care so that the key won't leak. If worse come to worse and the key leak, the owner will notice and request a new key. –  Shay Erlichmen Oct 20 '10 at 20:06
    
@Shay - that's an awfully Utopian outlook. And remember, we'd have to do that for every Google App Engine [app] (or similar service). Logistically, handling that many potential key revocations is a nightmare. –  Kevin Montrose Oct 20 '10 at 20:19
    
@Kevin - You haven't said anything about Key + Top level domain. I don't know how/if that would work, but it seems feasible... –  Dan Dumitru Oct 20 '10 at 20:30
    
@Dan - how do we get the domain? If it comes across in the request, its easily forged. We can't look it up based on the IP, because thats shared. Its equivalent to any of the other suggestions, and just as flawed. Trust me, alot of thought went into how quotas work; and we settled on "the way everyone else has been forced to do it." Its not ideal, but its the reality of the internet as it exists today. –  Kevin Montrose Oct 20 '10 at 20:33
2  
The key revocation can be done automatically by the key owner without any intervention on your behalf. The problem is how do you certified an application, I would start with ones that have a decent reputation on one of the SE sites. –  Shay Erlichmen Oct 20 '10 at 21:00
    
@Shay - How would the key owner know the key had been leaked? We can't force them to check any metrics. The burden ultimately falls on the SO team. Trying to certify [app]s (high rep on anything but SO is irrelevant, and its a stretch even on SO; security is not easy) is going to take us down the App Store rabbit hole, which is not even worth considering. –  Kevin Montrose Oct 20 '10 at 21:06
5  
AppID isn't trivially forged: We (App Engine) include the App ID in the user-agent header, and we don't let users set the user-agent header on outgoing requests. If the IP comes from our pool (_netblocks.google.com), you can trust that the App ID in the user-agent is accurate. –  Nick Johnson Oct 28 '10 at 15:14
    
@Nick - I've hoped for and expected something along these lines, so thanks for jumping in with an authoritative statement! Does this imply that @Kevins assumption of IP+AppId being trivially forged has actually been unfounded and IP+AppId is an/the appropriate solution instead, so all this would have been much ado about nothing (including the just released custom workaround)? –  Steffen Opel Oct 28 '10 at 17:28
1  
@Steffen I disagree that IP+AppId is an appropriate solution also with this statement. API should be coded without custom permission (in this case the google range of IPs.); handling specific cases with custom IF is a terrible idea. The custom workaround, although temporary, is far more generic and usable (also outside GAE). –  systempuntoout Oct 30 '10 at 16:20
    
@Steffen That's correct - distinguishing on IP plus AppId is the right way to limit per 'user' for App Engine sources. –  Nick Johnson Oct 31 '10 at 14:45
2  
@systempuntoout In an ideal world with unlimited IPs and no cloud solutions with shared egress, I'd agree. As it is, the world isn't ideal, and anything that limits on IP either needs to special case large heterogenous sources of egress traffic, or implement keys/access tokens. –  Nick Johnson Oct 31 '10 at 14:46

I had a GAE app that queried hourly a few sites for traffic stats. I've brought it down. Thanks for the warning and enjoy your 72 extra calls per day :)

share|improve this answer
1  
+1 ahahahahah escaped from a sinking boat. Thank you for the tip. –  systempuntoout Oct 20 '10 at 18:31
    
@system I don't really have a choice to be honest :) –  badp Oct 20 '10 at 18:37

All this issues also apply to request throttling limits (30 requests over 5 seconds).
I'm sure that I'm throttling requests from my app properly according to api terms of usage based on a distributed lock with memcached, but still frequently getting following error "Too Many Requests, We're sorry...There are an unusual number of requests coming from this IP address.To protect our users, we can't process any more requests from this IP address right now."

share|improve this answer
2  
also covered in an early post - too lazy to find it right now, but thanks for bringing it up. –  Sky Sanders Oct 21 '10 at 0:52

A possible solution will be to whitelist some app keys, the whitelisting means that the ip is not checked and only the app key is check.

share|improve this answer
    
@Shay Just checked: x-ratelimit-current - 2814 --> I think in few hours will be doomed again –  systempuntoout Oct 20 '10 at 10:46
    
Here is what I've just got in a request that makes just a few api calls -> X-RateLimit-Current: 2262, X-RateLimit-Current: 2002 –  Vladislav Tserman Oct 20 '10 at 11:50
    
x-ratelimit-current - 1947 –  systempuntoout Oct 20 '10 at 12:14
    
x-ratelimit-current - 1624 –  systempuntoout Oct 20 '10 at 12:20
    
x-ratelimit-current - 1340 –  systempuntoout Oct 20 '10 at 13:13
    
x-ratelimit-current - 983 wow, here we are.. –  systempuntoout Oct 20 '10 at 13:23
    
x-ratelimit-current - 439 –  systempuntoout Oct 20 '10 at 14:19
    
x-ratelimit-current - 29 –  systempuntoout Oct 20 '10 at 14:51
4  
GONE!!!!!!!!!!!! –  systempuntoout Oct 20 '10 at 14:59
1  
@system - sorry bout your luck. looks like everyone in GAE either gets unlimited rate-limit or rate-limit engine checks for key+TLD or GAE is no longer a viable platform for API apps –  Sky Sanders Oct 20 '10 at 17:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .