3

I wrote a small program querying the SE API to grab the public scoring data of all SO answers (excluding the last 3 days).

Basically the request I do gets me this:

 Questions[]:
    Id
    Title
    Tags
    CreationDate
    Owner:
        Id
        Name
    Answers[]:
        Id
        CreationDate
        Owner:
            Id
            Name
        Score

Here's an example of the queries I do (API key removed):

http://api.stackexchange.com/2.2/questions?site=stackoverflow&filter=!OfYUQYtgCaZ9JBeJyrvLd85AXer(WSNHQacu))0iZzl&sort=creation&order=asc&page=2580&fromdate=1229368998&todate=1433795169

The problem is: This processus is too slow to get me everything. I receive 30 questions per response of the API. I'm allowed 10,000 requests per day. So it would take me a long time to get up with the site and have all questions.

I'm also asked to backoff for 10 seconds after every request, so I suppose it's heavy for the server.

Here are my questions:

  • Is some part of my request too heavy ?
  • Is there a better approach ?
  • Thanks to a hint given by Benjamin Gruenbaum, I'm downloading a data dump (27GB) to try to do an initial filling of my DB. – Denys Séguret Jun 12 '15 at 7:50
  • Note that the dump wouldn't really solve the problem, only make it a little smaller. – Denys Séguret Jun 18 '15 at 15:57

You must log in to answer this question.

Browse other questions tagged .