###Throttling and Caching
In order to provide maximum data throughput and at the same time protect your application from, and obviate coding for, server errors due to throttle rate violations and/or polling abuse, Soapi.JS2 implements a caching throttle implementation, Soapi.Net.ThrottleCache, that throttles on a sliding window with a governor supplemented by a per route configurable request cache.
All http requests are handled by a static instance of ThrottleCache that lives @ Soapi.Net.Throttle.
e.g.
// internal code
Soapi.Net.Throttle =
new Soapi.Net.ThrottleCache(throttleWindowTime, throttleWindowCount,
maxActiveRequests, defaultCacheDuration);
// Allow [throttleWindowCount] requests per [throttleWindowTime]
// while active requests < maxActiveRequests
// Cache all results [defaultCacheDuration] if an explicit cache rule for that route,
// e.g. '/users/{id}' is not found.
Soapi.Net.Throttle.enqueue(request);
Throttling
As a request is accepted by the throttle, the internal request cache is checked for matching results, if found the request is satisified immediately and the workflow is complete.
If a cache hit is not found, the request is placed in a queue which is processed at a regular polling interval. The resolution of this polling timer is arbitrary and can be changed by stopping and restarting the throttle.
Soapi.Net.Throttle.suspend();
Soapi.Net.Throttle.start(pollingIntervalInMs);
At polling time, there are 3 conditions that must be met in order for a request to be sent to the API:
- that the cache has not received matching data while the request sat in the queue. If this is the case, the request is satsified from the cache.
- that the number of active requests does not exceed the configured maximum active request count. If the number of active requests meets or exceeds the max allowed, processing of the queue is deferred.
- that the interval, in ms, between now and the nth previous request, where nth is the window count, is equal to or greater than the configured window interval, otherwise processing of the queue is deferred.
When these conditions are met, a request is popped from the queue and sent to the API.
Caching
Each successful request, after the json has been converted into domain objects, it is placed in a request cache, keyed by the url.
Each cache item is scheduled to be removed from the cache [defaultCacheDuration] milliseconds after it is placed, unless an explicit cache rule has been added for that route, e.g.
// cache stackauth.com/1.0/sites for 5 minutes
Soapi.Net.Throttle.addCacheRule('/sites', 360000);
You can 'seed' the cache with data for fault tolerance or testing purposes by using the same mechanism used internally when a successful response is received and processed, e.g.
Soapi.Net.Throttle.signal(manuallyConstructedSites, '/sites', 'http://stackauth.com/sites?key=yourkey')
The single static instance of ThrottleCache will be used by any and all ApiContext instances you may have in play on any single page.
Example:
Throttle-cache log from AlphaQuickStart.htm with 30 requests per 5 second window with max 15 active requests and a 60 second default cache duration - note the cache scavenging happening as the browser was left idle.
throttle: 2010-07-20 20:27:44.830 - request processed: interval: 1282361264830ms throttle: 2010-07-20 20:27:45.027 - request processed: interval: 197ms throttle: 2010-07-20 20:27:45.037 - request processed: interval: 10ms throttle: 2010-07-20 20:27:45.047 - request processed: interval: 10ms throttle: 2010-07-20 20:27:45.057 - request processed: interval: 10ms throttle: 2010-07-20 20:27:45.067 - request processed: interval: 10ms throttle: 2010-07-20 20:27:45.077 - request processed: interval: 10ms throttle: 2010-07-20 20:27:45.087 - request processed: interval: 10ms throttle: 2010-07-20 20:27:45.097 - request processed: interval: 10ms throttle: 2010-07-20 20:27:45.107 - request processed: interval: 10ms throttle: 2010-07-20 20:27:45.117 - request processed: interval: 10ms throttle: 2010-07-20 20:27:45.128 - request processed: interval: 11ms throttle: 2010-07-20 20:27:45.138 - request processed: interval: 10ms throttle: 2010-07-20 20:27:45.148 - request processed: interval: 10ms throttle: 2010-07-20 20:27:45.158 - request processed: interval: 10ms throttle: 2010-07-20 20:27:45.168 - request processed: interval: 10ms throttle: 2010-07-20 20:27:45.178 - active queue full throttle: 2010-07-20 20:27:45.268 - request processed: interval: 100ms throttle: 2010-07-20 20:27:45.278 - active queue full throttle: 2010-07-20 20:27:45.568 - request processed: interval: 300ms throttle: 2010-07-20 20:27:45.578 - active queue full throttle: 2010-07-20 20:27:46.730 - request processed: interval: 1162ms throttle: 2010-07-20 20:27:46.740 - active queue full throttle: 2010-07-20 20:27:47.010 - request processed: interval: 280ms throttle: 2010-07-20 20:27:47.020 - active queue full throttle: 2010-07-20 20:27:47.040 - request processed: interval: 30ms throttle: 2010-07-20 20:27:47.050 - active queue full throttle: 2010-07-20 20:27:47.365 - request processed: interval: 325ms throttle: 2010-07-20 20:27:47.376 - active queue full throttle: 2010-07-20 20:27:47.449 - request processed: interval: 84ms throttle: 2010-07-20 20:27:47.465 - active queue full throttle: 2010-07-20 20:27:47.535 - request processed: interval: 86ms throttle: 2010-07-20 20:27:47.546 - active queue full throttle: 2010-07-20 20:27:47.746 - request processed: interval: 211ms throttle: 2010-07-20 20:27:47.756 - active queue full throttle: 2010-07-20 20:27:47.997 - request processed: interval: 251ms throttle: 2010-07-20 20:27:48.007 - active queue full throttle: 2010-07-20 20:27:48.097 - request processed: interval: 100ms throttle: 2010-07-20 20:27:48.107 - active queue full throttle: 2010-07-20 20:27:48.327 - request processed: interval: 230ms throttle: 2010-07-20 20:27:48.337 - active queue full throttle: 2010-07-20 20:27:48.898 - request processed: interval: 571ms throttle: 2010-07-20 20:27:48.908 - active queue full throttle: 2010-07-20 20:27:48.921 - request processed: interval: 23ms throttle: 2010-07-20 20:27:48.939 - active queue full throttle: 2010-07-20 20:27:49.020 - waiting 810ms throttle: 2010-07-20 20:27:49.832 - request processed: interval: 911ms throttle: 2010-07-20 20:27:49.843 - waiting 184ms throttle: 2010-07-20 20:27:50.027 - request processed: interval: 195ms throttle: 2010-07-20 20:27:50.037 - request processed: interval: 10ms throttle: 2010-07-20 20:27:50.047 - request processed: interval: 10ms throttle: 2010-07-20 20:27:50.057 - request processed: interval: 10ms throttle: 2010-07-20 20:27:50.067 - request processed: interval: 10ms throttle: 2010-07-20 20:27:50.077 - request processed: interval: 10ms throttle: 2010-07-20 20:27:50.087 - request processed: interval: 10ms throttle: 2010-07-20 20:27:50.097 - request processed: interval: 10ms throttle: 2010-07-20 20:27:50.107 - request processed: interval: 10ms throttle: 2010-07-20 20:27:50.117 - request processed: interval: 10ms throttle: 2010-07-20 20:27:50.128 - request processed: interval: 11ms throttle: 2010-07-20 20:27:50.138 - request processed: interval: 10ms throttle: 2010-07-20 20:27:50.148 - request processed: interval: 10ms throttle: 2010-07-20 20:27:50.158 - active queue full throttle: 2010-07-20 20:27:50.170 - request processed: interval: 22ms throttle: 2010-07-20 20:27:50.181 - request processed: interval: 11ms throttle: 2010-07-20 20:27:50.192 - active queue full throttle: 2010-07-20 20:27:50.262 - waiting 6ms throttle: 2010-07-20 20:27:50.273 - request processed: interval: 92ms throttle: 2010-07-20 20:27:50.284 - waiting 284ms cache: 2010-07-20 20:27:50.387 - served from cache: http://api.stackapps.com/1.0/users/14/answers cache: 2010-07-20 20:27:50.388 - served from cache: http://api.stackapps.com/1.0/users/14/badges cache: 2010-07-20 20:27:50.388 - served from cache: http://api.stackapps.com/1.0/users/14/comments cache: 2010-07-20 20:27:50.388 - served from cache: http://api.stackapps.com/1.0/users/14/comments/18 cache: 2010-07-20 20:27:50.388 - served from cache: http://api.stackapps.com/1.0/users/14/favorites cache: 2010-07-20 20:27:50.388 - served from cache: http://api.stackapps.com/1.0/users/14/mentioned cache: 2010-07-20 20:27:50.389 - served from cache: http://api.stackapps.com/1.0/users/14/reputation cache: 2010-07-20 20:27:50.389 - served from cache: http://api.stackapps.com/1.0/users/14/tags cache: 2010-07-20 20:27:50.389 - served from cache: http://api.stackapps.com/1.0/users/14/timeline throttle: 2010-07-20 20:27:50.575 - request processed: interval: 302ms throttle: 2010-07-20 20:27:50.586 - waiting 1144ms cache: 2010-07-20 20:27:51.397 - served from cache: http://api.stackapps.com/1.0/questions/386/answers cache: 2010-07-20 20:27:51.397 - served from cache: http://api.stackapps.com/1.0/questions/386/comments cache: 2010-07-20 20:27:51.397 - served from cache: http://api.stackapps.com/1.0/questions/386/timeline cache: 2010-07-20 20:27:51.398 - served from cache: http://api.stackapps.com/1.0/revisions/386 cache: 2010-07-20 20:27:51.398 - served from cache: http://api.stackapps.com/1.0/revisions/386/e452e755-a3fe-45d0-976f-e319ec460200 throttle: 2010-07-20 20:27:51.732 - request processed: interval: 1157ms throttle: 2010-07-20 20:27:51.742 - waiting 268ms throttle: 2010-07-20 20:27:52.012 - request processed: interval: 280ms throttle: 2010-07-20 20:27:52.022 - waiting 18ms throttle: 2010-07-20 20:27:52.042 - request processed: interval: 30ms throttle: 2010-07-20 20:27:52.052 - waiting 313ms throttle: 2010-07-20 20:27:52.373 - request processed: interval: 331ms throttle: 2010-07-20 20:27:52.383 - waiting 66ms throttle: 2010-07-20 20:27:52.453 - request processed: interval: 80ms throttle: 2010-07-20 20:27:52.463 - waiting 72ms throttle: 2010-07-20 20:27:52.544 - request processed: interval: 91ms throttle: 2010-07-20 20:27:52.554 - waiting 192ms throttle: 2010-07-20 20:27:52.755 - request processed: interval: 211ms throttle: 2010-07-20 20:27:52.766 - waiting 231ms throttle: 2010-07-20 20:27:53.001 - request processed: interval: 246ms throttle: 2010-07-20 20:27:53.011 - waiting 86ms throttle: 2010-07-20 20:27:53.101 - request processed: interval: 100ms throttle: 2010-07-20 20:27:53.112 - waiting 215ms throttle: 2010-07-20 20:27:53.334 - request processed: interval: 233ms throttle: 2010-07-20 20:27:53.345 - waiting 553ms throttle: 2010-07-20 20:27:53.903 - request processed: interval: 569ms throttle: 2010-07-20 20:27:53.914 - waiting 7ms throttle: 2010-07-20 20:27:53.924 - request processed: interval: 21ms throttle: 2010-07-20 20:27:53.935 - waiting 897ms throttle: 2010-07-20 20:27:54.839 - request processed: interval: 915ms throttle: 2010-07-20 20:27:54.849 - waiting 178ms throttle: 2010-07-20 20:27:55.029 - request processed: interval: 190ms throttle: 2010-07-20 20:27:55.040 - request processed: interval: 11ms throttle: 2010-07-20 20:27:55.051 - request processed: interval: 11ms throttle: 2010-07-20 20:27:55.062 - request processed: interval: 11ms throttle: 2010-07-20 20:27:55.073 - request processed: interval: 11ms throttle: 2010-07-20 20:27:55.084 - request processed: interval: 11ms throttle: 2010-07-20 20:27:55.094 - request processed: interval: 10ms throttle: 2010-07-20 20:27:55.104 - request processed: interval: 10ms throttle: 2010-07-20 20:27:55.114 - request processed: interval: 10ms throttle: 2010-07-20 20:27:55.124 - request processed: interval: 10ms throttle: 2010-07-20 20:27:55.134 - request processed: interval: 10ms throttle: 2010-07-20 20:27:55.145 - request processed: interval: 11ms throttle: 2010-07-20 20:27:55.155 - request processed: interval: 10ms throttle: 2010-07-20 20:27:55.166 - waiting 4ms throttle: 2010-07-20 20:27:55.176 - request processed: interval: 21ms throttle: 2010-07-20 20:27:55.187 - active queue full throttle: 2010-07-20 20:27:56.079 - request processed: interval: 903ms throttle: 2010-07-20 20:27:56.090 - active queue full throttle: 2010-07-20 20:27:56.212 - request processed: interval: 133ms throttle: 2010-07-20 20:27:56.222 - request processed: interval: 10ms throttle: 2010-07-20 20:27:56.232 - waiting 500ms throttle: 2010-07-20 20:27:56.735 - request processed: interval: 513ms throttle: 2010-07-20 20:27:56.745 - waiting 267ms throttle: 2010-07-20 20:27:57.018 - request processed: interval: 283ms throttle: 2010-07-20 20:27:57.028 - waiting 14ms throttle: 2010-07-20 20:27:57.049 - request processed: interval: 31ms throttle: 2010-07-20 20:27:57.059 - active queue full throttle: 2010-07-20 20:27:57.261 - waiting 112ms throttle: 2010-07-20 20:27:57.373 - request processed: interval: 324ms throttle: 2010-07-20 20:27:57.383 - waiting 70ms throttle: 2010-07-20 20:27:57.455 - request processed: interval: 82ms throttle: 2010-07-20 20:27:57.465 - waiting 79ms throttle: 2010-07-20 20:27:57.545 - request processed: interval: 90ms throttle: 2010-07-20 20:27:57.566 - waiting 189ms throttle: 2010-07-20 20:27:57.774 - request processed: interval: 229ms throttle: 2010-07-20 20:27:57.796 - waiting 205ms throttle: 2010-07-20 20:27:58.003 - request processed: interval: 229ms throttle: 2010-07-20 20:27:58.023 - waiting 78ms throttle: 2010-07-20 20:27:58.103 - request processed: interval: 100ms throttle: 2010-07-20 20:27:58.123 - active queue full throttle: 2010-07-20 20:27:58.351 - request processed: interval: 248ms throttle: 2010-07-20 20:27:58.361 - active queue full throttle: 2010-07-20 20:27:58.412 - waiting 491ms throttle: 2010-07-20 20:27:58.907 - request processed: interval: 556ms throttle: 2010-07-20 20:27:58.918 - waiting 6ms throttle: 2010-07-20 20:27:58.928 - request processed: interval: 21ms throttle: 2010-07-20 20:27:58.938 - waiting 901ms throttle: 2010-07-20 20:27:59.842 - request processed: interval: 914ms throttle: 2010-07-20 20:27:59.852 - waiting 177ms throttle: 2010-07-20 20:28:0.033 - request processed: interval: 191ms throttle: 2010-07-20 20:28:0.043 - request processed: interval: 10ms throttle: 2010-07-20 20:28:0.053 - request processed: interval: 10ms throttle: 2010-07-20 20:28:0.063 - request processed: interval: 10ms throttle: 2010-07-20 20:28:0.073 - request processed: interval: 10ms throttle: 2010-07-20 20:28:0.083 - waiting 1ms throttle: 2010-07-20 20:28:0.093 - request processed: interval: 20ms throttle: 2010-07-20 20:28:0.104 - request processed: interval: 11ms throttle: 2010-07-20 20:28:0.115 - request processed: interval: 11ms throttle: 2010-07-20 20:28:0.125 - request processed: interval: 10ms throttle: 2010-07-20 20:28:0.135 - request processed: interval: 10ms throttle: 2010-07-20 20:28:0.145 - request processed: interval: 10ms throttle: 2010-07-20 20:28:0.155 - request processed: interval: 10ms throttle: 2010-07-20 20:28:0.165 - request processed: interval: 10ms throttle: 2010-07-20 20:28:0.175 - active queue full throttle: 2010-07-20 20:28:0.216 - request processed: interval: 51ms throttle: 2010-07-20 20:28:0.227 - active queue full throttle: 2010-07-20 20:28:0.237 - waiting 842ms throttle: 2010-07-20 20:28:1.081 - request processed: interval: 865ms throttle: 2010-07-20 20:28:1.092 - waiting 120ms throttle: 2010-07-20 20:28:1.215 - request processed: interval: 134ms throttle: 2010-07-20 20:28:1.226 - request processed: interval: 11ms throttle: 2010-07-20 20:28:1.237 - waiting 498ms throttle: 2010-07-20 20:28:1.735 - request processed: interval: 509ms throttle: 2010-07-20 20:28:1.746 - waiting 272ms throttle: 2010-07-20 20:28:2.026 - request processed: interval: 291ms throttle: 2010-07-20 20:28:2.036 - waiting 13ms throttle: 2010-07-20 20:28:2.056 - request processed: interval: 30ms throttle: 2010-07-20 20:28:2.066 - waiting 307ms throttle: 2010-07-20 20:28:2.376 - request processed: interval: 320ms throttle: 2010-07-20 20:28:2.387 - waiting 68ms throttle: 2010-07-20 20:28:2.457 - request processed: interval: 81ms throttle: 2010-07-20 20:28:2.467 - waiting 78ms throttle: 2010-07-20 20:28:2.548 - request processed: interval: 91ms throttle: 2010-07-20 20:28:2.559 - waiting 215ms throttle: 2010-07-20 20:28:2.779 - request processed: interval: 231ms throttle: 2010-07-20 20:28:2.790 - waiting 213ms throttle: 2010-07-20 20:28:3.011 - request processed: interval: 232ms throttle: 2010-07-20 20:28:3.022 - waiting 81ms throttle: 2010-07-20 20:28:3.112 - request processed: interval: 101ms throttle: 2010-07-20 20:28:3.122 - active queue full throttle: 2010-07-20 20:28:3.343 - waiting 8ms throttle: 2010-07-20 20:28:3.354 - request processed: interval: 242ms throttle: 2010-07-20 20:28:3.365 - active queue full throttle: 2010-07-20 20:28:3.385 - waiting 522ms throttle: 2010-07-20 20:28:3.916 - request processed: interval: 562ms throttle: 2010-07-20 20:28:3.927 - waiting 1ms throttle: 2010-07-20 20:28:3.938 - request processed: interval: 22ms throttle: 2010-07-20 20:28:3.948 - waiting 894ms throttle: 2010-07-20 20:28:4.846 - request processed: interval: 908ms throttle: 2010-07-20 20:28:4.856 - waiting 177ms throttle: 2010-07-20 20:28:5.036 - request processed: interval: 190ms throttle: 2010-07-20 20:28:5.046 - request processed: interval: 10ms throttle: 2010-07-20 20:28:5.056 - request processed: interval: 10ms throttle: 2010-07-20 20:28:5.066 - active queue full throttle: 2010-07-20 20:28:5.106 - request processed: interval: 50ms throttle: 2010-07-20 20:28:5.117 - active queue full throttle: 2010-07-20 20:28:5.198 - request processed: interval: 92ms throttle: 2010-07-20 20:28:5.208 - active queue full throttle: 2010-07-20 20:28:5.270 - request processed: interval: 72ms throttle: 2010-07-20 20:28:5.280 - active queue full throttle: 2010-07-20 20:28:5.340 - request processed: interval: 70ms throttle: 2010-07-20 20:28:7.424 - request processed: interval: 2084ms throttle: 2010-07-20 20:28:9.329 - request processed: interval: 1905ms throttle: 2010-07-20 20:28:13.267 - request processed: interval: 3938ms throttle: 2010-07-20 20:28:14.359 - request processed: interval: 1092ms cache: 2010-07-20 20:28:45.017 - removed from cache: http://stackauth.com/1.0/sites cache: 2010-07-20 20:28:45.560 - removed from cache: http://api.stackapps.com/1.0/answers/704/comments cache: 2010-07-20 20:28:46.720 - removed from cache: http://api.stackapps.com/1.0/badges/name cache: 2010-07-20 20:28:47.002 - removed from cache: http://api.stackapps.com/1.0/answers/704 cache: 2010-07-20 20:28:47.037 - removed from cache: http://api.stackapps.com/1.0/badges [-- truncated for size --] cache: 2010-07-20 20:29:13.267 - removed from cache: http://api.stackapps.com/1.0/questions?pagesize=100&page=5 cache: 2010-07-20 20:29:14.356 - removed from cache: http://api.stackapps.com/1.0/questions?pagesize=100&page=6 cache: 2010-07-20 20:29:15.175 - removed from cache: http://api.stackapps.com/1.0/questions?pagesize=100&page=7 >>>