6

We are small company in Malaysia and intended to do a small research to analyze technology trends. We would like to use the Stack Exchange API for this purpose but we are afraid it might get detected as offensive by your back end server.

Is there a way we can avoid this outcome? Or any other suggestion to assist us to achieve our goals?

Improve this question

1 Answer 1

Reset to default
6

For analyzing large chunks of data, don't use the API. Use the Stack Exchange Data Explorer (SEDE), that's what it's there for. SEDE data lags real-time by 1 to 5 weeks, but that's no big problem for most statistical purposes.

For intense "Big data" analysis beyond even SEDE's capabilities, use the quarterly data dumps. (Thanks, Tim Stone, for the reminder.)

If you really must use the API, follow these guidelines:

  1. Register your app.
  2. Make sure you do not bust your quotas.
  3. Monitor and respect the backoff flag.
  4. For anything more, contact the API team and negotiate for special access.
Improve this answer
5
  • Thanks! I didn't know SEDE exist. It seems like SQL interface which I could try!
    – Chris Liaw
    Commented May 12, 2014 at 0:24
  • 2
    @ChrisLiaw The quarterly data dumps are also an option if you need to work with the data in a more intensive way than Data Explorer supports.
    – Tim Stone ♦
    Commented May 12, 2014 at 2:44
  • @TimStone, the data dumps includes all previous data? I mean if I grab some GBs of dump from the site, those including all previous data up to dump date?
    – Chris Liaw
    Commented May 12, 2014 at 14:53
  • @ChrisLiaw Yes, that's correct (excluding deleted data, of course)
    – Tim Stone ♦
    Commented May 12, 2014 at 15:02
  • Wow! Thanks! That's very useful for offline analysis!
    – Chris Liaw
    Commented May 13, 2014 at 5:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .