Soapi Caching and Throttling
Throttle
All API requests made by Soapi are routed first through Soapi.Net.RequestCache and failing a cache hit, they are then routed through Soapi.Net.RequestThrottle.
The throttle, which is described in detail here, works on a sliding window, allowing maximum request rate up to the prescribed limits, currently 30 per 5 seconds, and subsequently blocking requests, in the order received, to maintain the prescribed rate.
An additional measure, that compensates for what seems to be an undocumented rate restriction, once a request passes the rate throttle, it enters the active request queue which limits the number of active requests to a configurable number, currently 10.
Ultimately this means that short bursts of request that are characteristic of an end user discovery or research tool will be serviced as quickly as they are received, improving user experience and long running processes that may involved thousands of requests can be trusted to run reliably to completion as quickly as allowed.
In testing, I have pumped 5000 requests into Soapi as fast as my machine with 50 threads will go and received all responses successfully.
Cache
Implementing the cache proved to be a challenge. My initial attempts were quite naive, simply caching successful results for a specific period of time.
Casual testing showed this to improve performance and reduce API bandwidth usage as expected.
It was when I implemented lazy loading, e.g stub hydration, that the shortcomings of this approach were exposed.
It became clear that many identical request could be pushed into the cache/throttle in immediate succession resulting in multiple identical requests being made simultaneously.
The obvious problem is that each request checks the cache before proceeding and since the initial request has not completed and cached the results for that url, multiple identical request are issued. This is a no-no for several reasons.
Ultimately, the solution is to implement smart cache items that know when they are pending, valid and expired. Thus when the initial request is recieved, an item is placed in the cache with a state of 'pending' until the request completes whereupon the cache item is populated with data and flagged as valid.
Subsequent requests that are received for that url while the request is pending register as 'waiting'. When the original request completes and signals, the waiting requests simply grab the cached data from memory and carry on.
Implementing this was an interesting challenge in thread management, especially when the Silverlight and Windows Phone environments are factored in, but ultimately, the implementation appears successful.
Fault tolerance
When the complimentary services provided by the throttle and cache are combined in a real world scenario, request failures become a big issue.
If a request is pending for a piece of data that 10 other requests are waiting for and it fails, the failure cascades.
Now, if the failure is a hard error, such as might result from bad parameters or repeatable unrecoverable error, all of the requests are going to fail in any case.
But consider transient network problems or the spurious 500 errors that the server throws occasionally. These types of failures are recoverable.
With this in mind, the requests made by Soapi, upon failure simply rest a moment and retry up to a configurable amount of times. Currently 3 is the default retry count.
This capability spans the throttle and the cache providing an impressive degree of reliability and fault tolerance.
