Timeline for Get everything touched by a user?
Current License: CC BY-SA 4.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 29, 2018 at 19:35 | comment | added | Brock Adams | For me, if I try a long series of sustained fetches at more than about 2 per second, I start getting backoffs. Limiting to 1 per second seldom sees problems. Theoretically, if you want to push it, you could get 2767 pages in 2 minutes. | |
| May 29, 2018 at 18:45 | comment | added | ashleedawg | Great idea, thanks. Out of curiosity, on what did you base the estimate for fetch time? Is there a standard avg you use? | |
| May 29, 2018 at 17:47 | comment | added | Brock Adams | If you fetch the bulk with BigQuery, then you can limit the SE API calls to just that data that occurred since the last data dump. | |
| May 29, 2018 at 10:03 | vote | accept | ashleedawg | ||
| May 29, 2018 at 10:03 | comment | added | ashleedawg |
...BigQuery -- which has an API very interesting... I'm going to check that out, although I eventually need to get this working with the realtime data.
|
|
| May 27, 2018 at 18:33 | comment | added | Brock Adams |
Also, to reiterate, the /timeline route solves your "intermittent missing comments problem", in this case.
|
|
| May 27, 2018 at 18:30 | comment | added | Brock Adams | Data Dump data can be fetched programmatically from Google's BigQuery -- which has an API. | |
| May 27, 2018 at 18:27 | comment | added | Brock Adams | JonSkeet has over 2767 pages of data (based on wanting to get the parent question for every comment, answer, and mention). That's over a quarter of your bandwidth and would take 20 to 50 minutes to fetch if you somehow didn't get rate limited. | |
| May 27, 2018 at 13:36 | comment | added | ashleedawg | I don't foresee a problem with API limits, because 10k calls should be enough to get 7 @JonSkeet's/day. (I hope he doesn't mind being a unit of measurement!) The average user has 10 posts and 26 comments. | |
| May 27, 2018 at 13:32 | comment | added | ashleedawg | I need to do this programmatically, hence the API, unless there's a way to do this from SEDE? I'd much rather retrieve 50,000 rows at a time instead of 100. Meanwhile, ` /users/{ids}/timeline` may be a quicker way for me to retrieve my list of "every question a user has touched". For example, if "User X" came to this page and added a comment anywhere, I's now want to include not only the comment in the dataset for "User X", but also the source Question, all of its Answers, and all comments on the question, or any of its' answers. | |
| May 27, 2018 at 7:17 | history | answered | Brock Adams | CC BY-SA 4.0 |