3

I wrote a small program querying the SE API to grab the public scoring data of all SO answers (excluding the last 3 days).

Basically the request I do gets me this:

 Questions[]:
    Id
    Title
    Tags
    CreationDate
    Owner:
        Id
        Name
    Answers[]:
        Id
        CreationDate
        Owner:
            Id
            Name
        Score

Here's an example of the queries I do (API key removed):

http://api.stackexchange.com/2.2/questions?site=stackoverflow&filter=!OfYUQYtgCaZ9JBeJyrvLd85AXer(WSNHQacu))0iZzl&sort=creation&order=asc&page=2580&fromdate=1229368998&todate=1433795169

The problem is: This processus is too slow to get me everything. I receive 30 questions per response of the API. I'm allowed 10,000 requests per day. So it would take me a long time to get up with the site and have all questions.

I'm also asked to backoff for 10 seconds after every request, so I suppose it's heavy for the server.

Here are my questions:

  • Is some part of my request too heavy ?
  • Is there a better approach ?
Improve this question
2
  • Thanks to a hint given by Benjamin Gruenbaum, I'm downloading a data dump (27GB) to try to do an initial filling of my DB. Commented Jun 12, 2015 at 7:50
  • Note that the dump wouldn't really solve the problem, only make it a little smaller. Commented Jun 18, 2015 at 15:57

0

Reset to default

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.