Return to Answer

##Soapi Caching and Throttling

Throttle

All API requests made by Soapi are routed first through Soapi.Net.RequestCache and failing a cache hit, they are then routed through Soapi.Net.RequestThrottle.

The throttle, which is described in detail here, works on a sliding window, allowing maximum request rate up to the prescribed limits, currently 30 per 5 seconds, and subsequently blocking requests, in the order received, to maintain the prescribed rate.

An additional measure, that compensates for what seems to be an undocumented rate restriction, once a request passes the rate throttle, it enters the active request queue which limits the number of active requests to a configurable number, currently 10.

Ultimately this means that short bursts of request that are characteristic of an end user discovery or research tool will be serviced as quickly as they are received, improving user experience and long running processes that may involved thousands of requests can be trusted to run reliably to completion as quickly as allowed.

In testing, I have pumped 5000 requests into Soapi as fast as my machine with 50 threads will go and received all responses successfully.

Cache

Implementing the cache proved to be a challenge. My initial attempts were quite naive, simply caching successful results for a specific period of time.

Casual testing showed this to improve performance and reduce API bandwidth usage as expected.

It was when I implemented lazy loading, e.g stub hydration, that the shortcomings of this approach were exposed.

It became clear that many identical request could be pushed into the cache/throttle in immediate succession resulting in multiple identical requests being made simultaneously.

The obvious problem is that each request checks the cache before proceeding and since the initial request has not completed and cached the results for that url, multiple identical request are issued. This is a no-no for several reasons.

Ultimately, the solution is to implement smart cache items that know when they are pending, valid and expired. Thus when the initial request is recieved, an item is placed in the cache with a state of 'pending' until the request completes whereupon the cache item is populated with data and flagged as valid.

Subsequent requests that are received for that url while the request is pending register as 'waiting'. When the original request completes and signals, the waiting requests simply grab the cached data from memory and carry on.

Implementing this was an interesting challenge in thread management, especially when the Silverlight and Windows Phone environments are factored in, but ultimately, the implementation appears successful.

Fault tolerance

When the complimentary services provided by the throttle and cache are combined in a real world scenario, request failures become a big issue.

If a request is pending for a piece of data that 10 other requests are waiting for and it fails, the failure cascades.

Now, if the failure is a hard error, such as might result from bad parameters or repeatable unrecoverable error, all of the requests are going to fail in any case.

But consider transient network problems or the spurious 500 errors that the server throws occasionally. These types of failures are recoverable.

With this in mind, the requests made by Soapi, upon failure simply rest a moment and retry up to a configurable amount of times. Currently 3 is the default retry count.

This capability spans the throttle and the cache providing an impressive degree of reliability and fault tolerance.

Soapi Caching and Throttling

Throttle

All API requests made by Soapi are routed first through Soapi.Net.RequestCache and failing a cache hit, they are then routed through Soapi.Net.RequestThrottle.

The throttle, which is described in detail here, works on a sliding window, allowing maximum request rate up to the prescribed limits, currently 30 per 5 seconds, and subsequently blocking requests, in the order received, to maintain the prescribed rate.

An additional measure, that compensates for what seems to be an undocumented rate restriction, once a request passes the rate throttle, it enters the active request queue which limits the number of active requests to a configurable number, currently 10.

Ultimately this means that short bursts of request that are characteristic of an end user discovery or research tool will be serviced as quickly as they are received, improving user experience and long running processes that may involved thousands of requests can be trusted to run reliably to completion as quickly as allowed.

In testing, I have pumped 5000 requests into Soapi as fast as my machine with 50 threads will go and received all responses successfully.

Cache

Implementing the cache proved to be a challenge. My initial attempts were quite naive, simply caching successful results for a specific period of time.

Casual testing showed this to improve performance and reduce API bandwidth usage as expected.

It was when I implemented lazy loading, e.g stub hydration, that the shortcomings of this approach were exposed.

It became clear that many identical request could be pushed into the cache/throttle in immediate succession resulting in multiple identical requests being made simultaneously.

The obvious problem is that each request checks the cache before proceeding and since the initial request has not completed and cached the results for that url, multiple identical request are issued. This is a no-no for several reasons.

Ultimately, the solution is to implement smart cache items that know when they are pending, valid and expired. Thus when the initial request is recieved, an item is placed in the cache with a state of 'pending' until the request completes whereupon the cache item is populated with data and flagged as valid.

Subsequent requests that are received for that url while the request is pending register as 'waiting'. When the original request completes and signals, the waiting requests simply grab the cached data from memory and carry on.

Implementing this was an interesting challenge in thread management, especially when the Silverlight and Windows Phone environments are factored in, but ultimately, the implementation appears successful.

Fault tolerance

When the complimentary services provided by the throttle and cache are combined in a real world scenario, request failures become a big issue.

If a request is pending for a piece of data that 10 other requests are waiting for and it fails, the failure cascades.

Now, if the failure is a hard error, such as might result from bad parameters or repeatable unrecoverable error, all of the requests are going to fail in any case.

But consider transient network problems or the spurious 500 errors that the server throws occasionally. These types of failures are recoverable.

With this in mind, the requests made by Soapi, upon failure simply rest a moment and retry up to a configurable amount of times. Currently 3 is the default retry count.

This capability spans the throttle and the cache providing an impressive degree of reliability and fault tolerance.

##Soapi Caching and Throttling

Throttle

All API requests made by Soapi are routed first through Soapi.Net.RequestCache and failing a cache hit, they are then routed through Soapi.Net.RequestThrottle.

The throttle, which is described in detail here, works on a sliding window, allowing maximum request rate up to the prescribed limits, currently 30 per 5 seconds, and subsequently blocking requests, in the order received, to maintain the prescribed rate.

An additional measure, that compensates for what seems to be an undocumented rate restriction, once a request passes the rate throttle, it enters the active request queue which limits the number of active requests to a configurable number, currently 10.

Ultimately this means that short bursts of request that are characteristic of an end user discovery or research tool will be serviced as quickly as they are received, improving user experience and long running processes that may involved thousands of requests can be trusted to run reliably to completion as quickly as allowed.

In testing, I have pumped 5000 requests into Soapi as fast as my machine with 50 threads will go and received all responses successfully.

Cache

Implementing the cache proved to be a challenge. My initial attempts were quite naive, simply caching successful results for a specific period of time.

Casual testing showed this to improve performance and reduce API bandwidth usage as expected.

It was when I implemented lazy loading, e.g stub hydration, that the shortcomings of this approach were exposed.

It became clear that many identical request could be pushed into the cache/throttle in immediate succession resulting in multiple identical requests being made simultaneously.

The obvious problem is that each request checks the cache before proceeding and since the initial request has not completed and cached the results for that url, multiple identical request are issued. This is a no-no for several reasons.

Ultimately, the solution is to implement smart cache items that know when they are pending, valid and expired. Thus when the initial request is recieved, an item is placed in the cache with a state of 'pending' until the request completes whereupon the cache item is populated with data and flagged as valid.

Subsequent requests that are received for that url while the request is pending register as 'waiting'. When the original request completes and signals, the waiting requests simply grab the cached data from memory and carry on.

Implementing this was an interesting challenge in thread management, especially when the Silverlight and Windows Phone environments are factored in, but ultimately, the implementation appears successful.

Fault tolerance

When the complimentary services provided by the throttle and cache are combined in a real world scenario, request failures become a big issue.

If a request is pending for a piece of data that 10 other requests are waiting for and it fails, the failure cascades.

Now, if the failure is a hard error, such as might result from bad parameters or repeatable unrecoverable error, all of the requests are going to fail in any case.

But consider transient network problems or the spurious 500 errors that the server throws occasionally. These types of failures are recoverable.

With this in mind, the requests made by Soapi, upon failure simply rest a moment and retry up to a configurable amount of times. Currently 3 is the default retry count.

This capability spans the throttle and the cache providing an impressive degree of reliability and fault tolerance.

##Soapi Caching and Throttling

Throttle

All API requests made by Soapi are routed first through Soapi.Net.RequestCache and failing a cache hit, they are then routed through Soapi.Net.RequestThrottle.

The throttle, which is described in detail here, works on a sliding window, allowing maximum request rate up to the prescribed limits, currently 30 per 5 seconds, and subsequently blocking requests, in the order received, to maintain the prescribed rate.

An additional measure, that compensates for what seems to be an undocumented rate restriction, once a request passes the rate throttle, it enters the active request queue which limits the number of active requests to a configurable number, currently 10.

Ultimately this means that short bursts of request that are characteristic of an end user discovery or research tool will be serviced as quickly as they are received, improving user experience and long running processes that may involved thousands of requests can be trusted to run reliably to completion as quickly as allowed.

In testing, I have pumped 5000 requests into Soapi as fast as my machine with 50 threads will go and received all responses successfully.

Cache

Implementing the cache proved to be a challenge. My initial attempts were quite naive, simply caching successful results for a specific period of time.

Casual testing showed this to improve performance and reduce API bandwidth usage as expected.

It was when I implemented lazy loading, e.g stub hydration, that the shortcomings of this approach were exposed.

It became clear that many identical request could be pushed into the cache/throttle in immediate succession resulting in multiple identical requests being made simultaneously.

The obvious problem is that each request checks the cache before proceeding and since the initial request has not completed and cached the results for that url, multiple identical request are issued. This is a no-no for several reasons.

Ultimately, the solution is to implement smart cache items that know when they are pending, valid and expired. Thus when the initial request is recieved, an item is placed in the cache with a state of 'pending' until the request completes whereupon the cache item is populated with data and flagged as valid.

Subsequent requests that are received for that url while the request is pending register as 'waiting'. When the original request completes and signals, the waiting requests simply grab the cached data from memory and carry on.

Implementing this was an interesting challenge in thread management, especially when the Silverlight and Windows Phone environments are factored in, but ultimately, the implementation appears successful.

Fault tolerance

When the complimentary services provided by the throttle and cache are combined in a real world scenario, request failures become a big issue.

If a request is pending for a piece of data that 10 other requests are waiting for and it fails, the failure cascades.

Now, if the failure is a hard error, such as might result from bad parameters or repeatable unrecoverable error, all of the requests are going to fail in any case.

But consider transient network problems or the spurious 500 errors that the server throws occasionally. These types of failures are recoverable.

With this in mind, the requests made by Soapi, upon failure simply rest a moment and retry up to a configurable amount of times. Currently 3 is the default retry count.

This capability spans the throttle and the cache providing an impressive degree of reliability and fault tolerance.

##More Information

Currently the code generated is very lean and is very simple to use but does not yet have some bells and whistles of some other wrappers, including nice identifiers and appropriate clr types for dates and such.

This is an intentional design decision. I want to maintain a consistent core as close to the metal as possible to allow for a stable platform for extension.

I have plans to generate a wrapper wrapper (;-)) library that does not affect the solid core library that will provide a higher level API for those who don't need to smell the copper.

I can expect that it would start to behave similarly to Kevin's original example with lazy loading object graphs providing dot navigation from any where to anywhere.

code generation

Some work needs to be done to bring the api parser and code generator to a point suitable for general consumption but when it does arrive at that point, consuming the products of the parser and codedom generation to produce similar wrappers in any language, while not trivial, should be an easily accomplished task.

For example, the emitter that produces the end-to-end, fully documented with intellisense support JavaScript wrapper took about 4 hours to hack in.

I fully welcome any suggestions and am especially interested in receiving patches with usage and tests.

##Soapi Caching and Throttling

Throttle

All API requests made by Soapi are routed first through Soapi.Net.RequestCache and failing a cache hit, they are then routed through Soapi.Net.RequestThrottle.

The throttle, which is described in detail here, works on a sliding window, allowing maximum request rate up to the prescribed limits, currently 30 per 5 seconds, and subsequently blocking requests, in the order received, to maintain the prescribed rate.

An additional measure, that compensates for what seems to be an undocumented rate restriction, once a request passes the rate throttle, it enters the active request queue which limits the number of active requests to a configurable number, currently 10.

Ultimately this means that short bursts of request that are characteristic of an end user discovery or research tool will be serviced as quickly as they are received, improving user experience and long running processes that may involved thousands of requests can be trusted to run reliably to completion as quickly as allowed.

In testing, I have pumped 5000 requests into Soapi as fast as my machine with 50 threads will go and received all responses successfully.

Cache

Implementing the cache proved to be a challenge. My initial attempts were quite naive, simply caching successful results for a specific period of time.

Casual testing showed this to improve performance and reduce API bandwidth usage as expected.

It was when I implemented lazy loading, e.g stub hydration, that the shortcomings of this approach were exposed.

It became clear that many identical request could be pushed into the cache/throttle in immediate succession resulting in multiple identical requests being made simultaneously.

The obvious problem is that each request checks the cache before proceeding and since the initial request has not completed and cached the results for that url, multiple identical request are issued. This is a no-no for several reasons.

Ultimately, the solution is to implement smart cache items that know when they are pending, valid and expired. Thus when the initial request is recieved, an item is placed in the cache with a state of 'pending' until the request completes whereupon the cache item is populated with data and flagged as valid.

Subsequent requests that are received for that url while the request is pending register as 'waiting'. When the original request completes and signals, the waiting requests simply grab the cached data from memory and carry on.

Implementing this was an interesting challenge in thread management, especially when the Silverlight and Windows Phone environments are factored in, but ultimately, the implementation appears successful.

Fault tolerance

When the complimentary services provided by the throttle and cache are combined in a real world scenario, request failures become a big issue.

If a request is pending for a piece of data that 10 other requests are waiting for and it fails, the failure cascades.

Now, if the failure is a hard error, such as might result from bad parameters or repeatable unrecoverable error, all of the requests are going to fail in any case.

But consider transient network problems or the spurious 500 errors that the server throws occasionally. These types of failures are recoverable.

With this in mind, the requests made by Soapi, upon failure simply rest a moment and retry up to a configurable amount of times. Currently 3 is the default retry count.

This capability spans the throttle and the cache providing an impressive degree of reliability and fault tolerance.