Prefered way to retrieve all search results

Question

I'm trying to figure out a reliable way to retrieve all the results from a search query (in subsequent pages. My use case includes cases with more than 100 results (the maximum page size).

Looking at the possibilities, the safest way seems to be to retrieve questions by increasing creation date. This way, if new questions are posted between successive requests, they will only appear on the last page, and not cause questions to appear twice or never in the results due to shifting page alignment.

But what happens if a question is deleted during my successive requests, say between search?…&page=1 and search?…&page=2? If the deleted question was on page 1, will I ever see the 30th question (assuming a page size of 30)?

I notice that http://api.stackoverflow.com/1.1/search?tagged=c&pagesize=100 retrieves only 99 questions (the total is >2000 in this case). Is this actually due to a deleted question? If deleted questions count in the total, successive requests might actually be safe — are they?

Another problem is when to stop: since the total might change between requests, what can I trust? Should I iterate until I get an empty page?

Summary: What is a reliable way of retrieving all the results of a search, across multiple result pages?

Answer 1

A possible solution to this finally hit me out of the blue like a ton of bricks:

Instead of retrieving successive pages by page number, why not retrieve them by date range? Here is a quick rundown of a possible procedure for retrieving questions this way (there is one caveat, which I'll explain afterward):

1. Sort by descending creation date and grab the first page. This page will contain 100 questions.
2. Grab the creation timestamp of the last question on the page.
3. Make an identical query but pass the timestamp from step #2 minus 1 as the max parameter. This will return all questions up until the timestamp of the last question.
4. Repeat steps #2 - #3 until a response contains fewer than 100 results.

This technique will ensure that all non-deleted questions from the beginning to the point in time that you made the first request are returned.

I mentioned that there was a caveat. Since you are fetching questions by timestamp (which is not unique), it is possible that some questions might have identical timestamps and therefore their order would be undefined. A workaround for this is to skip the subtraction of 1 in step #3 and check for duplicate question IDs at the end of the requests. This will ensure that 100% of the questions are always returned.

Answer 2

I'll try to answer your questions here:

"This way, if new questions are posted between successive requests, they will only appear on the last page, and not cause questions to appear twice or never in the results due to shifting page alignment."

You don't have to worry about that at all if you specify min and max parameters. Be sure to specify a date for min that occurs before the site existed; for max, simply use the current date/time.

That way, you are limiting yourself to a fixed range and don't have to worry about new items being added.

"But what happens if a question is deleted during my successive requests..."

Each request comes with pagination information at the top - which might help you if one gets deleted because the total item will be one less on a successive request.

Although this won't help you determine which item got deleted, you'll at least know that it occurred.