5

This looks like a duplicate of Throttle violation after 300 calls but this one is reported as fixed in 2012.

Today I made ~450 requests with an app key (no oauth token) and suddenly got a 502 response:

throttle_violation : too many requests from this IP, more requests available in 86281 seconds"

Did I do something wrong? The same API key used to work some weeks ago and I made several thousand requests per day without getting any errors.

Update:

Most probably my fault but I'll need to wait another day to confirm it :) I added backoff handling code also to the problematic requests and still failed. Then I noticed that the filter I use only includes .wrapper.total and backoff field is filtered out.

Update 2:

This is still happening after I started using a filter that adds the backOff and the quota_remaining properties.

I created a log file with every request that my app made and the responses. There is not a single backOff tag and the quota_remaining is not 0.

File available here

After 644 requests I suddenly get a throttle violation error without any warning like backoff or something.

I'll create something like a unit test reproducing the issue but it would need some more time since I also need to wait for 24 hours again (or register a new API key but I don't want to abuse).

Update 3:

MCVE available at: https://github.com/vap78/testissue7255

Travis CI integration job that reproduces the issue: https://travis-ci.org/vap78/testissue7255

Additional finding - looks like that the ban is much less than the 86390 seconds returned in the error message. I used the same app key several times this morning and I manage to make ~650 requests every time before I get this error.

Update 4:

I think this is really a bug. I implemented a logic that sleeps for 5 seconds and then retries the request when this error occurs - something similar to the backoff handling which just sleeps for 5 seconds. The retry was successful and the test I implemented passed.

So my best guess is that this error is returned instead of a backoff message for some reason or no backoff is returned in the previous response.

  • You need to provide an MCVE. Other high-volume apps are still going. Did you monitor and respect the backoff and quota_remaining properties? – Brock Adams Mar 17 '17 at 20:08
  • Ooops .. just noted that exactly in the used flow I did not include the backoff handling :(. When the ban lifts I'll be able to confirm that this was the issue. – vap78 Mar 18 '17 at 9:06
  • @BrockAdams please see my update. I'll add some coding reproducing the issue a bit later but at least by tracing the requests I can't understand why would I get such an error. – vap78 Apr 7 '17 at 15:41
  • Then this may be a bug; it's happened before. An MCVE is crucial. And go ahead and make new app keys. It doesn't hurt, and chasing down bugs is a good-enough reason. – Brock Adams Apr 7 '17 at 21:14
  • Also, track down any other app or script that might be using the API from that same IP address. – Brock Adams Apr 7 '17 at 21:15
  • @BrockAdams done. That's my personal computer so eventually the ISP might be using one IP for several customers to access the Internet. Still I reproduced this issue from a public CI server (see link above). – vap78 Apr 8 '17 at 8:13
  • Ugh, Java. Swore offa that over a decade ago! (Not an MCVE I personally want to mess with.) Good luck. – Brock Adams Apr 8 '17 at 9:05
  • Unfortunately bounties are wasted on Stack Apps. You might try posting on MSE when it runs out. – Brock Adams Apr 11 '17 at 11:16
  • :) well .. was worth a try – vap78 Apr 11 '17 at 11:17
3

Your app is subject to IP throttling as explained in the rate limiting documentation:

Every application is subject to an IP based concurrent request throttle. If a single IP is making more than 30 requests a second, new requests will be dropped. The exact ban period is subject to change, but will be on the order of 30 seconds to a few minutes typically. Note that exactly what response an application gets (in terms of HTTP code, text, and so on) is undefined when subject to this ban; we consider > 30 request/sec per IP to be very abusive and thus cut the requests off very harshly.

When I analyze your log file I see you managed to make 1289 calls in 1 minute and 37 seconds. While you are still within the 30 request per second, you're way over what the HA Proxy allows for any connection, coming from the API or not. You can find similar experiments to find the exact rate limit parameters on MSE but from my own experiments I find that several endpoints seem to have subtle different throttle rates.

The IP rate limit is separate and not linked or related to the dynamic throttle used by the backoff parameter. Not receiving a backoff doesn't mean you can't be IP throttled.

Pausing between calls, whether a backoff is received or not is the best advice I can give to prevent running into the IP rate limit.

  • Timing does not seem to affect the outcome. I adjusted the test case to wait 1 second after every request. Still at request 647 I got an error response. See: travis-ci.org/vap78/testissue7255/builds/310202771 – vap78 Dec 2 '17 at 17:49
  • You to wait more then ... you're still running into the IP filter. It is not a linear function... – rene Dec 2 '17 at 17:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .