Return to Revisions

###Throttling and Caching

In order to provide maximum data throughput and at the same time protect your application from, and obviate coding for, server errors due to throttle rate violations and/or polling abuse, Soapi.JS2 implements a caching throttle implementation, Soapi.Net.ThrottleCache, that throttles on a sliding window with a governor supplemented by a per route configurable request cache.

All http requests are handled by a static instance of ThrottleCache that lives @ Soapi.Net.Throttle.

e.g.

// internal code
Soapi.Net.Throttle = 
    new Soapi.Net.ThrottleCache(throttleWindowTime, throttleWindowCount, 
    maxActiveRequests, defaultCacheDuration);

// Allow [throttleWindowCount] requests per [throttleWindowTime] 
//       while active requests < maxActiveRequests 

// Cache all results [defaultCacheDuration] if an explicit cache rule for that route, 
// e.g. '/users/{id}' is not found.


Soapi.Net.Throttle.enqueue(request);

Throttling

As a request is accepted by the throttle, the internal request cache is checked for matching results, if found the request is satisified immediately and the workflow is complete.

If a cache hit is not found, the request is placed in a queue which is processed at a regular polling interval. The resolution of this polling timer is arbitrary and can be changed by stopping and restarting the throttle.

Soapi.Net.Throttle.suspend();
Soapi.Net.Throttle.start(pollingIntervalInMs);

At polling time, there are 3 conditions that must be met in order for a request to be sent to the API:

1. that the cache has not received matching data while the request sat in the queue. If this is the case, the request is satsified from the cache.
2. that the number of active requests does not exceed the configured maximum active request count. If the number of active requests meets or exceeds the max allowed, processing of the queue is deferred.
3. that the interval, in ms, between now and the n^th previous request, where n^th is the window count, is equal to or greater than the configured window interval, otherwise processing of the queue is deferred.

When these conditions are met, a request is popped from the queue and sent to the API.

Caching

Each successful request, after the json has been converted into domain objects, it is placed in a request cache, keyed by the url.

Each cache item is scheduled to be removed from the cache [defaultCacheDuration] milliseconds after it is placed, unless an explicit cache rule has been added for that route, e.g.

// cache stackauth.com/1.0/sites for 5 minutes
Soapi.Net.Throttle.addCacheRule('/sites', 360000); 

You can 'seed' the cache with data for fault tolerance or testing purposes by using the same mechanism used internally when a successful response is received and processed, e.g.

Soapi.Net.Throttle.signal(manuallyConstructedSites, '/sites', 'http://stackauth.com/sites?key=yourkey')

The single static instance of ThrottleCache will be used by any and all ApiContext instances you may have in play on any single page.

NOTE: the throttle-cache will be in today's (8-21) drop.

Example:

Throttle-cache log from AlphaQuickStart.htm with 30 requests per 5 second window with max 15 active requests and a 60 second default cache duration - note the cache scavenging happening as the browser was left idle.

throttle: 2010-07-20 20:27:44.830 - request processed: interval: 1282361264830ms
throttle: 2010-07-20 20:27:45.027 - request processed: interval: 197ms
throttle: 2010-07-20 20:27:45.037 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:45.047 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:45.057 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:45.067 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:45.077 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:45.087 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:45.097 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:45.107 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:45.117 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:45.128 - request processed: interval: 11ms
throttle: 2010-07-20 20:27:45.138 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:45.148 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:45.158 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:45.168 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:45.178 - active queue full
throttle: 2010-07-20 20:27:45.268 - request processed: interval: 100ms
throttle: 2010-07-20 20:27:45.278 - active queue full
throttle: 2010-07-20 20:27:45.568 - request processed: interval: 300ms
throttle: 2010-07-20 20:27:45.578 - active queue full
throttle: 2010-07-20 20:27:46.730 - request processed: interval: 1162ms
throttle: 2010-07-20 20:27:46.740 - active queue full
throttle: 2010-07-20 20:27:47.010 - request processed: interval: 280ms
throttle: 2010-07-20 20:27:47.020 - active queue full
throttle: 2010-07-20 20:27:47.040 - request processed: interval: 30ms
throttle: 2010-07-20 20:27:47.050 - active queue full
throttle: 2010-07-20 20:27:47.365 - request processed: interval: 325ms
throttle: 2010-07-20 20:27:47.376 - active queue full
throttle: 2010-07-20 20:27:47.449 - request processed: interval: 84ms
throttle: 2010-07-20 20:27:47.465 - active queue full
throttle: 2010-07-20 20:27:47.535 - request processed: interval: 86ms
throttle: 2010-07-20 20:27:47.546 - active queue full
throttle: 2010-07-20 20:27:47.746 - request processed: interval: 211ms
throttle: 2010-07-20 20:27:47.756 - active queue full
throttle: 2010-07-20 20:27:47.997 - request processed: interval: 251ms
throttle: 2010-07-20 20:27:48.007 - active queue full
throttle: 2010-07-20 20:27:48.097 - request processed: interval: 100ms
throttle: 2010-07-20 20:27:48.107 - active queue full
throttle: 2010-07-20 20:27:48.327 - request processed: interval: 230ms
throttle: 2010-07-20 20:27:48.337 - active queue full
throttle: 2010-07-20 20:27:48.898 - request processed: interval: 571ms
throttle: 2010-07-20 20:27:48.908 - active queue full
throttle: 2010-07-20 20:27:48.921 - request processed: interval: 23ms
throttle: 2010-07-20 20:27:48.939 - active queue full
throttle: 2010-07-20 20:27:49.020 -  waiting 810ms
throttle: 2010-07-20 20:27:49.832 - request processed: interval: 911ms
throttle: 2010-07-20 20:27:49.843 -  waiting 184ms
throttle: 2010-07-20 20:27:50.027 - request processed: interval: 195ms
throttle: 2010-07-20 20:27:50.037 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:50.047 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:50.057 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:50.067 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:50.077 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:50.087 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:50.097 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:50.107 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:50.117 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:50.128 - request processed: interval: 11ms
throttle: 2010-07-20 20:27:50.138 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:50.148 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:50.158 - active queue full
throttle: 2010-07-20 20:27:50.170 - request processed: interval: 22ms
throttle: 2010-07-20 20:27:50.181 - request processed: interval: 11ms
throttle: 2010-07-20 20:27:50.192 - active queue full
throttle: 2010-07-20 20:27:50.262 -  waiting 6ms
throttle: 2010-07-20 20:27:50.273 - request processed: interval: 92ms
throttle: 2010-07-20 20:27:50.284 -  waiting 284ms
cache: 2010-07-20 20:27:50.387 - served from cache: http://api.stackapps.com/1.0/users/14/answers
cache: 2010-07-20 20:27:50.388 - served from cache: http://api.stackapps.com/1.0/users/14/badges
cache: 2010-07-20 20:27:50.388 - served from cache: http://api.stackapps.com/1.0/users/14/comments
cache: 2010-07-20 20:27:50.388 - served from cache: http://api.stackapps.com/1.0/users/14/comments/18
cache: 2010-07-20 20:27:50.388 - served from cache: http://api.stackapps.com/1.0/users/14/favorites
cache: 2010-07-20 20:27:50.388 - served from cache: http://api.stackapps.com/1.0/users/14/mentioned
cache: 2010-07-20 20:27:50.389 - served from cache: http://api.stackapps.com/1.0/users/14/reputation
cache: 2010-07-20 20:27:50.389 - served from cache: http://api.stackapps.com/1.0/users/14/tags
cache: 2010-07-20 20:27:50.389 - served from cache: http://api.stackapps.com/1.0/users/14/timeline
throttle: 2010-07-20 20:27:50.575 - request processed: interval: 302ms
throttle: 2010-07-20 20:27:50.586 -  waiting 1144ms
cache: 2010-07-20 20:27:51.397 - served from cache: http://api.stackapps.com/1.0/questions/386/answers
cache: 2010-07-20 20:27:51.397 - served from cache: http://api.stackapps.com/1.0/questions/386/comments
cache: 2010-07-20 20:27:51.397 - served from cache: http://api.stackapps.com/1.0/questions/386/timeline
cache: 2010-07-20 20:27:51.398 - served from cache: http://api.stackapps.com/1.0/revisions/386
cache: 2010-07-20 20:27:51.398 - served from cache: http://api.stackapps.com/1.0/revisions/386/e452e755-a3fe-45d0-976f-e319ec460200
throttle: 2010-07-20 20:27:51.732 - request processed: interval: 1157ms
throttle: 2010-07-20 20:27:51.742 -  waiting 268ms
throttle: 2010-07-20 20:27:52.012 - request processed: interval: 280ms
throttle: 2010-07-20 20:27:52.022 -  waiting 18ms
throttle: 2010-07-20 20:27:52.042 - request processed: interval: 30ms
throttle: 2010-07-20 20:27:52.052 -  waiting 313ms
throttle: 2010-07-20 20:27:52.373 - request processed: interval: 331ms
throttle: 2010-07-20 20:27:52.383 -  waiting 66ms
throttle: 2010-07-20 20:27:52.453 - request processed: interval: 80ms
throttle: 2010-07-20 20:27:52.463 -  waiting 72ms
throttle: 2010-07-20 20:27:52.544 - request processed: interval: 91ms
throttle: 2010-07-20 20:27:52.554 -  waiting 192ms
throttle: 2010-07-20 20:27:52.755 - request processed: interval: 211ms
throttle: 2010-07-20 20:27:52.766 -  waiting 231ms
throttle: 2010-07-20 20:27:53.001 - request processed: interval: 246ms
throttle: 2010-07-20 20:27:53.011 -  waiting 86ms
throttle: 2010-07-20 20:27:53.101 - request processed: interval: 100ms
throttle: 2010-07-20 20:27:53.112 -  waiting 215ms
throttle: 2010-07-20 20:27:53.334 - request processed: interval: 233ms
throttle: 2010-07-20 20:27:53.345 -  waiting 553ms
throttle: 2010-07-20 20:27:53.903 - request processed: interval: 569ms
throttle: 2010-07-20 20:27:53.914 -  waiting 7ms
throttle: 2010-07-20 20:27:53.924 - request processed: interval: 21ms
throttle: 2010-07-20 20:27:53.935 -  waiting 897ms
throttle: 2010-07-20 20:27:54.839 - request processed: interval: 915ms
throttle: 2010-07-20 20:27:54.849 -  waiting 178ms
throttle: 2010-07-20 20:27:55.029 - request processed: interval: 190ms
throttle: 2010-07-20 20:27:55.040 - request processed: interval: 11ms
throttle: 2010-07-20 20:27:55.051 - request processed: interval: 11ms
throttle: 2010-07-20 20:27:55.062 - request processed: interval: 11ms
throttle: 2010-07-20 20:27:55.073 - request processed: interval: 11ms
throttle: 2010-07-20 20:27:55.084 - request processed: interval: 11ms
throttle: 2010-07-20 20:27:55.094 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:55.104 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:55.114 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:55.124 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:55.134 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:55.145 - request processed: interval: 11ms
throttle: 2010-07-20 20:27:55.155 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:55.166 -  waiting 4ms
throttle: 2010-07-20 20:27:55.176 - request processed: interval: 21ms
throttle: 2010-07-20 20:27:55.187 - active queue full
throttle: 2010-07-20 20:27:56.079 - request processed: interval: 903ms
throttle: 2010-07-20 20:27:56.090 - active queue full
throttle: 2010-07-20 20:27:56.212 - request processed: interval: 133ms
throttle: 2010-07-20 20:27:56.222 - request processed: interval: 10ms
throttle: 2010-07-20 20:27:56.232 -  waiting 500ms
throttle: 2010-07-20 20:27:56.735 - request processed: interval: 513ms
throttle: 2010-07-20 20:27:56.745 -  waiting 267ms
throttle: 2010-07-20 20:27:57.018 - request processed: interval: 283ms
throttle: 2010-07-20 20:27:57.028 -  waiting 14ms
throttle: 2010-07-20 20:27:57.049 - request processed: interval: 31ms
throttle: 2010-07-20 20:27:57.059 - active queue full
throttle: 2010-07-20 20:27:57.261 -  waiting 112ms
throttle: 2010-07-20 20:27:57.373 - request processed: interval: 324ms
throttle: 2010-07-20 20:27:57.383 -  waiting 70ms
throttle: 2010-07-20 20:27:57.455 - request processed: interval: 82ms
throttle: 2010-07-20 20:27:57.465 -  waiting 79ms
throttle: 2010-07-20 20:27:57.545 - request processed: interval: 90ms
throttle: 2010-07-20 20:27:57.566 -  waiting 189ms
throttle: 2010-07-20 20:27:57.774 - request processed: interval: 229ms
throttle: 2010-07-20 20:27:57.796 -  waiting 205ms
throttle: 2010-07-20 20:27:58.003 - request processed: interval: 229ms
throttle: 2010-07-20 20:27:58.023 -  waiting 78ms
throttle: 2010-07-20 20:27:58.103 - request processed: interval: 100ms
throttle: 2010-07-20 20:27:58.123 - active queue full
throttle: 2010-07-20 20:27:58.351 - request processed: interval: 248ms
throttle: 2010-07-20 20:27:58.361 - active queue full
throttle: 2010-07-20 20:27:58.412 -  waiting 491ms
throttle: 2010-07-20 20:27:58.907 - request processed: interval: 556ms
throttle: 2010-07-20 20:27:58.918 -  waiting 6ms
throttle: 2010-07-20 20:27:58.928 - request processed: interval: 21ms
throttle: 2010-07-20 20:27:58.938 -  waiting 901ms
throttle: 2010-07-20 20:27:59.842 - request processed: interval: 914ms
throttle: 2010-07-20 20:27:59.852 -  waiting 177ms
throttle: 2010-07-20 20:28:0.033 - request processed: interval: 191ms
throttle: 2010-07-20 20:28:0.043 - request processed: interval: 10ms
throttle: 2010-07-20 20:28:0.053 - request processed: interval: 10ms
throttle: 2010-07-20 20:28:0.063 - request processed: interval: 10ms
throttle: 2010-07-20 20:28:0.073 - request processed: interval: 10ms
throttle: 2010-07-20 20:28:0.083 -  waiting 1ms
throttle: 2010-07-20 20:28:0.093 - request processed: interval: 20ms
throttle: 2010-07-20 20:28:0.104 - request processed: interval: 11ms
throttle: 2010-07-20 20:28:0.115 - request processed: interval: 11ms
throttle: 2010-07-20 20:28:0.125 - request processed: interval: 10ms
throttle: 2010-07-20 20:28:0.135 - request processed: interval: 10ms
throttle: 2010-07-20 20:28:0.145 - request processed: interval: 10ms
throttle: 2010-07-20 20:28:0.155 - request processed: interval: 10ms
throttle: 2010-07-20 20:28:0.165 - request processed: interval: 10ms
throttle: 2010-07-20 20:28:0.175 - active queue full
throttle: 2010-07-20 20:28:0.216 - request processed: interval: 51ms
throttle: 2010-07-20 20:28:0.227 - active queue full
throttle: 2010-07-20 20:28:0.237 -  waiting 842ms
throttle: 2010-07-20 20:28:1.081 - request processed: interval: 865ms
throttle: 2010-07-20 20:28:1.092 -  waiting 120ms
throttle: 2010-07-20 20:28:1.215 - request processed: interval: 134ms
throttle: 2010-07-20 20:28:1.226 - request processed: interval: 11ms
throttle: 2010-07-20 20:28:1.237 -  waiting 498ms
throttle: 2010-07-20 20:28:1.735 - request processed: interval: 509ms
throttle: 2010-07-20 20:28:1.746 -  waiting 272ms
throttle: 2010-07-20 20:28:2.026 - request processed: interval: 291ms
throttle: 2010-07-20 20:28:2.036 -  waiting 13ms
throttle: 2010-07-20 20:28:2.056 - request processed: interval: 30ms
throttle: 2010-07-20 20:28:2.066 -  waiting 307ms
throttle: 2010-07-20 20:28:2.376 - request processed: interval: 320ms
throttle: 2010-07-20 20:28:2.387 -  waiting 68ms
throttle: 2010-07-20 20:28:2.457 - request processed: interval: 81ms
throttle: 2010-07-20 20:28:2.467 -  waiting 78ms
throttle: 2010-07-20 20:28:2.548 - request processed: interval: 91ms
throttle: 2010-07-20 20:28:2.559 -  waiting 215ms
throttle: 2010-07-20 20:28:2.779 - request processed: interval: 231ms
throttle: 2010-07-20 20:28:2.790 -  waiting 213ms
throttle: 2010-07-20 20:28:3.011 - request processed: interval: 232ms
throttle: 2010-07-20 20:28:3.022 -  waiting 81ms
throttle: 2010-07-20 20:28:3.112 - request processed: interval: 101ms
throttle: 2010-07-20 20:28:3.122 - active queue full
throttle: 2010-07-20 20:28:3.343 -  waiting 8ms
throttle: 2010-07-20 20:28:3.354 - request processed: interval: 242ms
throttle: 2010-07-20 20:28:3.365 - active queue full
throttle: 2010-07-20 20:28:3.385 -  waiting 522ms
throttle: 2010-07-20 20:28:3.916 - request processed: interval: 562ms
throttle: 2010-07-20 20:28:3.927 -  waiting 1ms
throttle: 2010-07-20 20:28:3.938 - request processed: interval: 22ms
throttle: 2010-07-20 20:28:3.948 -  waiting 894ms
throttle: 2010-07-20 20:28:4.846 - request processed: interval: 908ms
throttle: 2010-07-20 20:28:4.856 -  waiting 177ms
throttle: 2010-07-20 20:28:5.036 - request processed: interval: 190ms
throttle: 2010-07-20 20:28:5.046 - request processed: interval: 10ms
throttle: 2010-07-20 20:28:5.056 - request processed: interval: 10ms
throttle: 2010-07-20 20:28:5.066 - active queue full
throttle: 2010-07-20 20:28:5.106 - request processed: interval: 50ms
throttle: 2010-07-20 20:28:5.117 - active queue full
throttle: 2010-07-20 20:28:5.198 - request processed: interval: 92ms
throttle: 2010-07-20 20:28:5.208 - active queue full
throttle: 2010-07-20 20:28:5.270 - request processed: interval: 72ms
throttle: 2010-07-20 20:28:5.280 - active queue full
throttle: 2010-07-20 20:28:5.340 - request processed: interval: 70ms
throttle: 2010-07-20 20:28:7.424 - request processed: interval: 2084ms
throttle: 2010-07-20 20:28:9.329 - request processed: interval: 1905ms
throttle: 2010-07-20 20:28:13.267 - request processed: interval: 3938ms
throttle: 2010-07-20 20:28:14.359 - request processed: interval: 1092ms
cache: 2010-07-20 20:28:45.017 - removed from cache: http://stackauth.com/1.0/sites
cache: 2010-07-20 20:28:45.560 - removed from cache: http://api.stackapps.com/1.0/answers/704/comments
cache: 2010-07-20 20:28:46.720 - removed from cache: http://api.stackapps.com/1.0/badges/name
cache: 2010-07-20 20:28:47.002 - removed from cache: http://api.stackapps.com/1.0/answers/704
cache: 2010-07-20 20:28:47.037 - removed from cache: http://api.stackapps.com/1.0/badges
[-- truncated for size --]
cache: 2010-07-20 20:29:13.267 - removed from cache: http://api.stackapps.com/1.0/questions?pagesize=100&page=5
cache: 2010-07-20 20:29:14.356 - removed from cache: http://api.stackapps.com/1.0/questions?pagesize=100&page=6
cache: 2010-07-20 20:29:15.175 - removed from cache: http://api.stackapps.com/1.0/questions?pagesize=100&page=7
>>>