I'm trying to figure out a reliable way to retrieve all the results from a search query (in subsequent pages. My use case includes cases with more than 100 results (the maximum page size).
Looking at the possibilities, the safest way seems to be to retrieve questions by increasing creation date. This way, if new questions are posted between successive requests, they will only appear on the last page, and not cause questions to appear twice or never in the results due to shifting page alignment.
But what happens if a question is deleted during my successive requests, say between
search?…&page=2? If the deleted question was on page 1, will I ever see the 30th question (assuming a page size of 30)?
I notice that http://api.stackoverflow.com/1.1/search?tagged=c&pagesize=100 retrieves only 99 questions (the total is >2000 in this case). Is this actually due to a deleted question? If deleted questions count in the total, successive requests might actually be safe — are they?
Another problem is when to stop: since the total might change between requests, what can I trust? Should I iterate until I get an empty page?
Summary: What is a reliable way of retrieving all the results of a search, across multiple result pages?