5

I'm trying to figure out a reliable way to retrieve all the results from a search query (in subsequent pages. My use case includes cases with more than 100 results (the maximum page size).

Looking at the possibilities, the safest way seems to be to retrieve questions by increasing creation date. This way, if new questions are posted between successive requests, they will only appear on the last page, and not cause questions to appear twice or never in the results due to shifting page alignment.

But what happens if a question is deleted during my successive requests, say between search?…&page=1 and search?…&page=2? If the deleted question was on page 1, will I ever see the 30th question (assuming a page size of 30)?

I notice that http://api.stackoverflow.com/1.1/search?tagged=c&pagesize=100 retrieves only 99 questions (the total is >2000 in this case). Is this actually due to a deleted question? If deleted questions count in the total, successive requests might actually be safe — are they?

Another problem is when to stop: since the total might change between requests, what can I trust? Should I iterate until I get an empty page?

Summary: What is a reliable way of retrieving all the results of a search, across multiple result pages?

2 Answers 2

3

A possible solution to this finally hit me out of the blue like a ton of bricks:

Instead of retrieving successive pages by page number, why not retrieve them by date range? Here is a quick rundown of a possible procedure for retrieving questions this way (there is one caveat, which I'll explain afterward):

  1. Sort by descending creation date and grab the first page. This page will contain 100 questions.
  2. Grab the creation timestamp of the last question on the page.
  3. Make an identical query but pass the timestamp from step #2 minus 1 as the max parameter. This will return all questions up until the timestamp of the last question.
  4. Repeat steps #2 - #3 until a response contains fewer than 100 results.

This technique will ensure that all non-deleted questions from the beginning to the point in time that you made the first request are returned.

I mentioned that there was a caveat. Since you are fetching questions by timestamp (which is not unique), it is possible that some questions might have identical timestamps and therefore their order would be undefined. A workaround for this is to skip the subtraction of 1 in step #3 and check for duplicate question IDs at the end of the requests. This will ensure that 100% of the questions are always returned.

4
  • Ok, this technique works to avoid skipping an item in the middle. It assumes there aren't 100 questions with the same date (this is unlikely to happen naturally, but what about a mass import?). Your termination condition is wrong, there can be less than 100 results at any time. This technique is rather costly; as I understand it, querying page=2 after page=1 is cheaper in the backend (because the data is already sitting in some cache, I guess), and from the user's point of view it's cheaper because it doesn't count as a separate request against the API usage quota. Oct 26, 2011 at 19:28
  • @Gilles: It doesn't count as a separate request when you're fetching more than one page? ...and why would there by fewer than 100 results on anything but the last page? Oct 26, 2011 at 19:49
  • Good point with the ">100 with the same timestamp" problem though. I hadn't thought of that. Oct 26, 2011 at 19:50
  • See the question: if there's a deleted result, it's counted in the 100. I guess deleted results are eliminated late in the chain. I thought …/page=2 just after …/page=1 was counted differently, but I can't find a reference to that now, I might have imagined it or confused with some other site's API. Oct 26, 2011 at 19:58
1

I'll try to answer your questions here:

"This way, if new questions are posted between successive requests, they will only appear on the last page, and not cause questions to appear twice or never in the results due to shifting page alignment."

You don't have to worry about that at all if you specify min and max parameters. Be sure to specify a date for min that occurs before the site existed; for max, simply use the current date/time.

That way, you are limiting yourself to a fixed range and don't have to worry about new items being added.

"But what happens if a question is deleted during my successive requests..."

Each request comes with pagination information at the top - which might help you if one gets deleted because the total item will be one less on a successive request.

Although this won't help you determine which item got deleted, you'll at least know that it occurred.

2
  • Er, so, does that mean I should start again from the beginning if I see a deleted item? Feb 28, 2011 at 20:23
  • @Gilles: It looks like that's the only option... I can't think of a way around it. Feb 28, 2011 at 20:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .