Why do I get a throttle violation after 300 requests while using a key but no access_token?

Question

I'm doing some testing on my app, and I let it run on a machine this afternoon while I was away. When I got back, I was geting 503 throttle violation errors from my API calls.

I was calling the "questions" endpoint at a rate of once per minute, passing a "min" date value to limit the returned questions to just those that have been submitted since my last poll. In the case where "has_more" was true in the response, I was retrying (with the page incremented) at a rate of once every 5 seconds. To my understanding, this is far less than the established maximum number of requests per unit time. I believe I was at about 300 requests total when I hit the throttle.

I've also got code in place to honor the "backoff" attribute if it was present, but that code was untested and it's possible it doesn't work.

I lost the console output, so I don't know exactly what happened when the feces impacted the high-speed rotating blades. Now, all of my requests return 502 errors, even though it's been about 2.5 hours since the "event."

Request url:

https://api.stackexchange.com/2.0/questions?sort=creation&pagesize=5&min=1325870128&site=gaming&order=asc&key=XXXXXXXX&page=1

Where XXXXXXXX is my API key. I can't get the quota since all endpoints are returning the same throttle violation error. In earlier testing, it was reading a max quota of 10,000.

Specific data that is being returned:

{u'error_id': 502, u'error_message': u'too many requests from this IP', 
 u'error_name': u'throttle_violation'}

Here's the API usage graph for my app, I can't grab the entire runtime graph (resolution issues), but there should be enough of it there to ensure that there's nothing odd going on:

What's a good way to debug this issue? I haven't found any good tools for showing a spike in my API requests at any higher resolution than daily. The data I can get back from the API is pretty much just "you were bad, go away" which isn't terribly helpful either.

UPDATE: At about 19:10 UTC Jan 6, the condition cleared and API requests are succeeding again. I'm going to let it continue to run to see if there's further issues, but I'd still like to root cause why this happened so I can avoid it in the future.

UPDATE2: I let the app run overnight, and as soon as I'd hit 301 requests, I hit another throttle violation, even though I am using a key with quota_max = 10,000

requesting stackexchange url: https://api.stackexchange.com/2.0/questions?sort=creation&pagesize=5&min=1325913207&site=gaming&order=asc&key=REDACTED&page=1

success!  calling callback.
{u'has_more': False, u'items': [], u'quota_max': 10000, u'quota_remaining': 9699}

2 minutes later:

requesting stackexchange url: https://api.stackexchange.com/2.0/questions?sort=creation&pagesize=5&min=1325913328&site=gaming&order=asc&key=REDACTED&page=1

Error:
{u'error_id': 502, u'error_message': u'too many requests from this IP', u'error_name': u'throttle_violation'}

The two requests were made 120 seconds apart, and you'll note that the final successful response did not contain a "backoff" element.

UPDATE 3 - I tried again today with an access_token parameter, and I'm currently at around 400 requests 10 seconds apart without throttling. The requests continued to succeed until my quota was around 1717, at which point the access_token timed out:

{u'error_id': 402, u'error_message': u'expired', u'error_name': u'invalid_access_token'}

Answer 1

I believe this has been fixed.

Let me know if you run into it again.