1

I am creating an R wrapper for the API. The use case for this is for doing data analysis on the SO data.

As I am building it, it seems like I would need to iterate over many pages in order to get all the data that I would want for an analysis. For instance, let's say I wanted to pull all SO users. Then I can call the users method and step through every page.

Is that an appropriate way to do this, or should I be using the data dump instead? I read through "Conscientious use of the API", but in this case I would be making large one-off requests (not polling).

Improve this question

4 Answers 4

Reset to default
1

You can use the API for data analysis, provided you stay within the limits we define for the API. Just like any other [app].

You'll be restricted on the number of requests/day, you shouldn't make frequent identical requests, and the like.

Really rapidly burning through your entire days quota would also raise some alarms, especially if you're the only user of your [app] key...

Improve this answer
2
  • Thanks Kevin. So if I don't exceed the 10,000 requests per day (with a key) and don't repeatedly hit the same page I should be ok. Even if I go ahead and cycle through every page of the same method once (e.g. download all users). Out of curiousity: if someone makes a successful iphone app, I presume that will exceed 10k a day. Will each instance of that app somehow have its own key?
    – Shane
    Commented Jul 9, 2010 at 18:50
  • 2
    @Shane 10k is for each IP/key combination so if you are using the API directly from each phone you won't have an issue.
    – carson
    Commented Jul 9, 2010 at 19:25
4

I don't think the API is well suited to data analysis.

If you want to do analysis of the SE dataset(s), you should download the latest torrent file. Depending on which, and how much, data you need for your analysis, you might be able to combine the dump with some judicious use of the API to pull current metadata from the site.

Improve this answer
1
  • That's a bummer because there's no easy way to automate pulling/loading the torrent file into R. So if this isn't kosher, then this rules out easy analysis of SO data by a major community of sophisticated users.
    – Shane
    Commented Jul 9, 2010 at 16:12
2

I've tried to maintain an synced version of the data dump since the API beta. At first I thought I could do it but I kept running into issues. The first issue I hit was that the dumps come out late so you have to make up a few days just to get the initial sync. Then there is the issue of rate limits and how quickly you hit them when you are trying to track the stack overflow data.

Improve this answer
2

To extend on Dave Swerskys answer: if you just have problems using the data from the torrent as such, but are fine with the implied gap of two to five weeks regarding the actuality of the data (see last FAQ item), you have another option, that is using the OData API instead: it's a versatile, fully RESTful API design and (IMHO) more thorough and mature already (despite being relatively young in itself) regarding several significant topics (e.g. HTTP cache control) than the custom Stack Overflow API as of version 1.0. As mentioned, and unfortunately, you won't get access to live data yet - I really hope this will change some day, but right now I wouldn't hold my breath, so Daves suggestion to combine this with some judicious use of the custom API is a very tempting one ;)

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .