Take the 2-minute tour ×
Stack Apps is a question and answer site for apps, scripts, and development with the Stack Exchange API. It's 100% free, no registration required.

I've been playing with some of the APIs that have paged results.

I've been doing a first call with pagesize=0 to determine how many pages there are then running a loop querying each page.

In my case I've never needed more than 8 pages at once so far, but I'm wondering if I were playing with a bigger SE site or when my site matures and I needed to request many more pages, are there some limits I must respect?

How many simultaneous requests is OK? Should I put in occasional delays, doing smaller batches at a time? If so how long a delay, etc?

I'm also interested in looking how any Stack Exchange API frameworks might have already implemented methods to query all pages in such "friendly" ways.

share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

The official rate limits are posted here.

One thing you mentioned in your post does strike me as a little odd:

"I've been doing a first call with pagesize=0 to determine how many pages there are then running a loop querying each page."

Why the need to fetch an empty page to determine the total? Why not fetch the first page (which contains the total) and then continue from there?

"I'm also interested in looking how any Stack Exchange API frameworks might have already implemented methods to query all pages in such "friendly" ways."

I wrote a PHP wrapper for the API that does this in a rather unique way. Instead of exposing pagination to the user (although pagination data can be retrieved if needed), a stream of items is provided:

$site = API::Site('stackapps');
$response = $site->Questions()->Unanswered()->Exec();

// The $response variable takes care of pagination for us.
// We simply call Fetch() over and over until it returns FALSE.

while($item = $response->Fetch())
    // Do something with $item

The Fetch() method automatically retrieves new pages as items are requested. It also takes the rate limit into account, pausing to comply with the limit if necessary.

share|improve this answer
    
"Why the need to fetch an empty page to determine the total? Why not fetch the first page (which contains the total) and then continue from there?" Well the first call needs to be synchronous anyway since you need the total and the code seems cleaner for the pagesize=0 special case than for the treat-the-first-page-differently special case. –  hippietrail Jan 8 '12 at 12:18
    
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .