Timeline for The /questions endpoint only returns around 30,000 items where 400,000 are expected
Current License: CC BY-SA 4.0
18 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 3, 2023 at 19:34 | vote | accept | cottinR | ||
| Jan 2, 2023 at 20:59 | history | edited | rene♦ | CC BY-SA 4.0 |
reworded a bit to focus on the has_more issue instead of the key or authentication problems.
|
| Jan 2, 2023 at 20:37 | comment | added | Makyen |
@rene You're correct. The stackr library does reply on has_more to end requesting pages. To account for the bug in the has_more field returned by the SE API, that end condition will need to be changed to be that items is a valid array, but contains no values. The cost for that will be one additional SE API request at the end of each sequence.
|
|
| Jan 2, 2023 at 20:35 | answer | added | rene♦ | timeline score: 3 | |
| Jan 2, 2023 at 20:18 | comment | added | Makyen | @rene Jsut FYI: The error which was reported in earlier questions from these users indicated that they had run out of quota, and had done so in < 2 hours. It's possible that is the end result of lots of testing, and wasn't the actual problem, but it's the error they reported as being the reason they stopped receiving results at < 300 requests "Collection issue for stackexchange api". | |
| Jan 2, 2023 at 20:17 | comment | added | cottinR | @Makyen do you think you could give an example? | |
| Jan 2, 2023 at 20:15 | comment | added | Makyen |
Thus, to avoid all of the issues, it's probably best to either just keep requesting pages until a valid response is received, but with no items, or get the total once, so you know the ballpark of number of requests,and then just keep requesting until getting a valid response, but with items empty. You could also test for has_more: false in addition to an empty (not missing) items. I haven't looked at the library that's being used in detail, but it's likely that it assumes that has_more is always valid, when it's actually unreliable.
|
|
| Jan 2, 2023 at 20:14 | comment | added | Makyen |
A compounding problem is that requesting total can be an expensive operation from the SE API's point of view. Doing so in SD was a strong contributor to the intermittent issues we have been seeing since May and the massive issues we saw 21 hours after having made changes to request Staging Ground posts. Overall, requesting total shouldn't be done with every request.
|
|
| Jan 2, 2023 at 20:14 | comment | added | Makyen |
@rene I had no problem fetching questions in the R tag. I did manually stop fetching after 1008 requests (100,800 questions) in order to not burn the additional 3.7k quota. However, the code I use determines the number of expected pages from the total and does not pay attention to has_more, so isn't affected by the intermittent error in what's returned in the has_more field. In the request logs, I do see at least one invalid has_more: false at around page 381.
|
|
| Jan 2, 2023 at 19:32 | comment | added | rene♦ |
@Makyen I can somehow reproduce this. I get 313 pages after which has_more returns false. It looks similar to this bug: stackapps.com/questions/8356 with the only difference that I don't get an CORS error. I'll check if I have other means to get past that has_more bug by using the steps offered by Brock.
|
|
| Jan 2, 2023 at 17:13 | comment | added | Makyen |
Frankly, the first thing I'd do is figure out some way to see what the library is sending to the SE API and verify if it's actually sending the key which you've intended to set. I don't know R or this library, so really can't tell you how to do that. The evidence you've provided so far, the limit to using a small number of requests and the error message you two have shown elsewhere, strongly suggests that it's not sending a valid key value with the request. So that's what I'd check first, either by looking at the values it's using to construct the request or the actual request sent.
|
|
| Jan 2, 2023 at 17:08 | comment | added | Makyen | For questions, I'd suggest focusing on one of: debugging the code you have (which either requires someone to duplicate what you have and do the debugging of finding out what the library is actually doing, or you finding out what the library is actually doing, primarily by looking at what it's actually sending to the SE API, or as close to that as you can, which could be just logging the values it's using to construct the requests), OR ask a general question of "how to do X" in R. | |
| Jan 2, 2023 at 17:01 | comment | added | cottinR | @Makygen I ask how I can receive all questions for the r tag from the stackoverflow site | |
| Jan 2, 2023 at 16:57 | comment | added | Makyen |
Asking about if you should use an access_token is, probably, a red herring. If the issue is that you are running out of quota at 300 requests, which is what # of requests & error message you two have included elsewhere tend to indicate, then using an access_token won't help you to solve that problem, because you must use an SE API key to use an access_token and what you're seeing tends to indicate the requests are being send without a valid key that matches the access_token, because each access_token represents authorization for a specific application + specific user.
|
|
| Jan 2, 2023 at 16:55 | comment | added | Makyen | This looks like a question which is an XY problem. Please describe in more detail what it is you are attempting to achieve (not how you're attempting to achieve it). | |
| Jan 2, 2023 at 16:31 | history | edited | cottinR | CC BY-SA 4.0 |
added 625 characters in body
|
| S Jan 2, 2023 at 16:11 | review | First questions | |||
| Jan 2, 2023 at 16:17 | |||||
| S Jan 2, 2023 at 16:11 | history | asked | cottinR | CC BY-SA 4.0 |