Stack Apps is a question and answer site for apps, scripts, and development with the Stack Exchange API. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We are small company in Malaysia and intended to do a small research to analyze technology trends. We would like to use the Stack Exchange API for this purpose but we are afraid it might get detected as offensive by your back end server.

Is there a way we can avoid this outcome? Or any other suggestion to assist us to achieve our goals?

share|improve this question
up vote 5 down vote accepted

For analyzing large chunks of data, don't use the API. Use the Stack Exchange Data Explorer (SEDE), that's what it's there for. SEDE data lags real-time by 1 to 5 weeks, but that's no big problem for most statistical purposes.

For intense "Big data" analysis beyond even SEDE's capabilities, use the quarterly data dumps. (Thanks, Tim Stone, for the reminder.)

If you really must use the API, follow these guidelines:

  1. Register your app.
  2. Make sure you do not bust your quotas.
  3. Monitor and respect the backoff flag.
  4. For anything more, contact the API team and negotiate for special access.
share|improve this answer
Thanks! I didn't know SEDE exist. It seems like SQL interface which I could try! – ChrisLiaw May 12 '14 at 0:24
@ChrisLiaw The quarterly data dumps are also an option if you need to work with the data in a more intensive way than Data Explorer supports. – Tim Stone May 12 '14 at 2:44
@TimStone, the data dumps includes all previous data? I mean if I grab some GBs of dump from the site, those including all previous data up to dump date? – ChrisLiaw May 12 '14 at 14:53
@ChrisLiaw Yes, that's correct (excluding deleted data, of course) – Tim Stone May 12 '14 at 15:02
Wow! Thanks! That's very useful for offline analysis! – ChrisLiaw May 13 '14 at 5:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .