23

alt text

Noticing a couple of "This IP has exceeded the request-per-day limit" errors on StackPrinter log, I asked for a verification and the prompt answer was:

There are a number of [app]s using Google App Engine, like StackPrinter.

Unfortunately, you're all using the same pool of IPs. Thus, you're all on the same quota.

If you can introduce a proxy or forwarder of some manner to your [app], you can solve this problem.

In few words: Google App Engine app[s] are doomed.

What?
I seriously thought that the quota check was by IP+AppKey.
Let's see the faq:

What are the API request limits?

A single IP address can only make a certain number of API requests per day, depending on the presence of a valid API key.

Although this seems quite clear, this sentence does not cover the case where the API keys, consumed by the same IP, are more than one.

Looking at "my" StackPrinter quota right now I see: x-ratelimit-current - 4486
Last evening was above 9000 and during the night there were just 50/60 prints; this sadly confirm the "shared quota" problem.

People have worked hard to rig up StackApps [App]s hosted on Google App Engine.
Here are some example:

This is a serious limitation of the API usage and without a proper action, our Gae Applications will starve in no time.

Proposal:

  • Is it possible to move the IP quota check to a more fair IP+Key or IP+AppId or IP+User-Agent or KeyOnly check?
  • Could "authenticated requests" be a solution?
  • Could you raise the 10.000 threshold for the Google pools of IP to temporarily tampon these "This IP has exceeded the request-per-day limit" errors?

One last thing: the proxy idea is useless because it is slow and does not scale with Google App Engine.

* Feel free to edit this question to add better ideas\proposals.


EDIT:
One Google App Engine IP is gone right now; a call against Stack Overflow api returns a dreadful x-ratelimit-current - 0


EDIT:
In response to Kevin's answer:
well, I think you are probably right in most of your thinking; I've drilled down into this topic a little bit further and all the APIkey-oriented checks I proposed (suggested more by the heat than by careful research) terribly suck.
Ok, they could be good for a toy API but not for a long term awesome project like the Stack Exchange API.

That said, I have to blame myself to have totally overlooked this aspect :).
In my defense, as previously said, faq is not particularly clear about this aspect and your answer given to this precog brilliant question was pretty misleading:

Sky: Hey, watch out! If apps are hosted on the same IP they could have problem.
Kevin: We've indicated in the past that the per-day limit for a key can be increased if need is demonstrated.

This wrong concept of "Problem with rate limit? We can increase it, no problem at all!" is sneakily reinforced in the faq:

What should I do if I need more requests per day?

Certain types of applications - services and websites to name two - can legitimately have much higher per-day request requirements than typical applications. If you can demonstrate a need for a higher request quota, contact us.

Wrong! This can be applied just for the "rich" lucky guy who has his own dedicated IP!
And remember, even moving your poor bleeding [app]s from the Cloud to Dreamhost does not solve the problem (although is less probable to have an hog request [app] on your same IP).
What is the lesson I learnt..
an API, to be a state of the art serious API to play with (for a website), must provide Authentication*; without this feature, you are asking for trouble.

I recommend you to update the faq and be clear about this danger.

What about our Google App Engine application?
Do we really need to wait an Authentication feature to port our app and move away from the Cloud?
Waiting for this feature, and still having a relatively small number of applications, is it possible to relax the key limit constraint a little?
Is a TLD+IP check a possible solution?

ok, too much thoughts.. thanks for your patience.

EDIT:
working solution here

* Oauth is not the only available solution; take Amazon Web Services for example, they allow authenticated REST calls passing apikey+timestamp+signature inside the query string for each request.

6
  • I Have no big hopes that this issue would be resolved anytime soon. Even Twitter claims that the majority of cloud platforms, including Google App Engine, cannot be whitelisted to access their Search API due to absence of static IP. But I continue to hope that something would be done. Oct 20, 2010 at 9:43
  • 1
    @Vladislav I totally agree that a cloud platform can't be whitelisted! But Twitter offers a way to bypass this problem using authenticated requests per app and/or user. Oct 20, 2010 at 9:48
  • 1
    DOOOOOOOOOOOOOOOOMED! Oct 20, 2010 at 17:17
  • 3
    For reference, we (App Engine) include the App ID in the user-agent header, and we don't let users set the user-agent header on outgoing requests. If the IP comes from our pool (_netblocks.google.com), you can trust that the App ID in the user-agent is accurate. Oct 28, 2010 at 15:15
  • @Nick are you sure? AFAIK user-agent can be set since SDK version 1.2.1 . I remember this because I auto answered my own question on this topic 7 month ago here Nov 22, 2010 at 13:41
  • @systempuntoout I'm positive - it's always been the case that although you can set part of the user-agent string, you can't remove your App ID from it. Sep 5, 2011 at 4:47

5 Answers 5

5

I was wondering if the scenario of a server based app that lives in a hosted environment, where a single IP can be shared by many web sites.

As I understand the rate-limit as it stands, every site on the shared host IP, regardless of api key, will be sharing the same rate-limit of 10,000 per site per day.

more @ rate-limit per endpoint per IP VS. server apps in hosted environments

Looks like everyone in GAE either

  • gets unlimited rate-limit
  • rate-limit engine checks for key + ip + top level domain (or some variation) of the referrer as described in linked post
  • GAE is no longer a viable platform for API apps

NOTE: this issue is not limited to the current fire - anyone on a shared platform, e.g. standard shared webhosting, is vulnerable to this issue.

1
  • 2
    totally forgot that brilliant precog question (that I had upvoted months ago). From what I see, just increasing the key limit does not solve the problem. In that scenario, a key hog app could continue to eat from other dishes without any problem leaving the other poor apps starving to death. Oct 20, 2010 at 13:20
6

The proposed "solutions" to this are all terrible*.

Ip+Key

Trivially forged.

IP+AppId

Trivially forged.

IP+User-Agent

Trivially forged.

KeyOnly

Surprise surprise, trivially forged.


Now, authentication would give us a non-trivially forged token to throttle on. This is a planned improvement in a future API version, however this workaround would require [app]s to force all users to login before using them (if our goal was to do away with IP based throttles altogether).

This problem will exist for any user-agnostic (ie. non-authenticating) [app] which shares its IP forever.

Essentially, [app]s on Google App Engine (or any other cloud service) are sitting behind a 1000-to-1 users-to-IPs order NAT, and NAT breaks the internet. If you want to avoid this problem, you'll need to get a dedicated IP somehow.

If there were a viable work around for this problem, we'd implement it, but frankly the issue is so far upstream of us there's nothing we can do.

Some other APIs with IP based throttling (in part, at least):

There are many others, these are just a few which are easy to find references for.

They are all, in fact, variations of security through obscurity.

13
  • 1
    How about white listing certified [app]s? Those apps will be throttled based on key only. Oct 20, 2010 at 18:50
  • 1
    @Shay - how do we identify those [app]s? They'd have to have a dedicated IP already, or its just a (historically, short) matter of time before the key is leaked. Now, if an [app] is already on a dedicated IP we can raise the quota (this has already been done for some [app]s, in fact). Oct 20, 2010 at 18:53
  • 3
    @Kevin why do you think that the key will leak? If the code runs on the server no one will see it and its the best interest of the key owner to take care so that the key won't leak. If worse come to worse and the key leak, the owner will notice and request a new key. Oct 20, 2010 at 20:06
  • @Shay - that's an awfully Utopian outlook. And remember, we'd have to do that for every Google App Engine [app] (or similar service). Logistically, handling that many potential key revocations is a nightmare. Oct 20, 2010 at 20:19
  • @Kevin - You haven't said anything about Key + Top level domain. I don't know how/if that would work, but it seems feasible... Oct 20, 2010 at 20:30
  • @Dan - how do we get the domain? If it comes across in the request, its easily forged. We can't look it up based on the IP, because thats shared. Its equivalent to any of the other suggestions, and just as flawed. Trust me, alot of thought went into how quotas work; and we settled on "the way everyone else has been forced to do it." Its not ideal, but its the reality of the internet as it exists today. Oct 20, 2010 at 20:33
  • 2
    The key revocation can be done automatically by the key owner without any intervention on your behalf. The problem is how do you certified an application, I would start with ones that have a decent reputation on one of the SE sites. Oct 20, 2010 at 21:00
  • @Shay - How would the key owner know the key had been leaked? We can't force them to check any metrics. The burden ultimately falls on the SO team. Trying to certify [app]s (high rep on anything but SO is irrelevant, and its a stretch even on SO; security is not easy) is going to take us down the App Store rabbit hole, which is not even worth considering. Oct 20, 2010 at 21:06
  • 5
    AppID isn't trivially forged: We (App Engine) include the App ID in the user-agent header, and we don't let users set the user-agent header on outgoing requests. If the IP comes from our pool (_netblocks.google.com), you can trust that the App ID in the user-agent is accurate. Oct 28, 2010 at 15:14
  • @Nick - I've hoped for and expected something along these lines, so thanks for jumping in with an authoritative statement! Does this imply that @Kevins assumption of IP+AppId being trivially forged has actually been unfounded and IP+AppId is an/the appropriate solution instead, so all this would have been much ado about nothing (including the just released custom workaround)? Oct 28, 2010 at 17:28
  • 1
    @Steffen I disagree that IP+AppId is an appropriate solution also with this statement. API should be coded without custom permission (in this case the google range of IPs.); handling specific cases with custom IF is a terrible idea. The custom workaround, although temporary, is far more generic and usable (also outside GAE). Oct 30, 2010 at 16:20
  • @Steffen That's correct - distinguishing on IP plus AppId is the right way to limit per 'user' for App Engine sources. Oct 31, 2010 at 14:45
  • 2
    @systempuntoout In an ideal world with unlimited IPs and no cloud solutions with shared egress, I'd agree. As it is, the world isn't ideal, and anything that limits on IP either needs to special case large heterogenous sources of egress traffic, or implement keys/access tokens. Oct 31, 2010 at 14:46
2

I had a GAE app that queried hourly a few sites for traffic stats. I've brought it down. Thanks for the warning and enjoy your 72 extra calls per day :)

0
2

All this issues also apply to request throttling limits (30 requests over 5 seconds).
I'm sure that I'm throttling requests from my app properly according to api terms of usage based on a distributed lock with memcached, but still frequently getting following error "Too Many Requests, We're sorry...There are an unusual number of requests coming from this IP address.To protect our users, we can't process any more requests from this IP address right now."

1
  • 2
    also covered in an early post - too lazy to find it right now, but thanks for bringing it up. Oct 21, 2010 at 0:52
1

A possible solution will be to whitelist some app keys, the whitelisting means that the ip is not checked and only the app key is check.

10
  • @Shay Just checked: x-ratelimit-current - 2814 --> I think in few hours will be doomed again Oct 20, 2010 at 10:46
  • Here is what I've just got in a request that makes just a few api calls -> X-RateLimit-Current: 2262, X-RateLimit-Current: 2002 Oct 20, 2010 at 11:50
  • x-ratelimit-current - 1947 Oct 20, 2010 at 12:14
  • x-ratelimit-current - 1624 Oct 20, 2010 at 12:20
  • x-ratelimit-current - 1340 Oct 20, 2010 at 13:13
  • x-ratelimit-current - 983 wow, here we are.. Oct 20, 2010 at 13:23
  • x-ratelimit-current - 439 Oct 20, 2010 at 14:19
  • x-ratelimit-current - 29 Oct 20, 2010 at 14:51
  • 4
    GONE!!!!!!!!!!!! Oct 20, 2010 at 14:59
  • 1
    @system - sorry bout your luck. looks like everyone in GAE either gets unlimited rate-limit or rate-limit engine checks for key+TLD or GAE is no longer a viable platform for API apps Oct 20, 2010 at 17:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .