Take the 2-minute tour ×
Stack Apps is a question and answer site for apps, scripts, and development with the Stack Exchange API. It's 100% free, no registration required.

We are small company in Malaysia and intended to do a small research to analyze technology trends. We would like to use the Stack Exchange API for this purpose but we are afraid it might get detected as offensive by your back end server.

Is there a way we can avoid this outcome? Or any other suggestion to assist us to achieve our goals?

share|improve this question

1 Answer 1

up vote 4 down vote accepted

For analyzing large chunks of data, don't use the API. Use the Stack Exchange Data Explorer (SEDE), that's what it's there for. SEDE data lags real-time by 1 to 5 weeks, but that's no big problem for most statistical purposes.

For intense "Big data" analysis beyond even SEDE's capabilities, use the quarterly data dumps. (Thanks, Tim Stone, for the reminder.)


If you really must use the API, follow these guidelines:

  1. Register your app.
  2. Make sure you do not bust your quotas.
  3. Monitor and respect the backoff flag.
  4. For anything more, contact the API team and negotiate for special access.
share|improve this answer
    
Thanks! I didn't know SEDE exist. It seems like SQL interface which I could try! –  ChrisLiaw May 12 at 0:24
2  
@ChrisLiaw The quarterly data dumps are also an option if you need to work with the data in a more intensive way than Data Explorer supports. –  Tim Stone May 12 at 2:44
    
@TimStone, the data dumps includes all previous data? I mean if I grab some GBs of dump from the site, those including all previous data up to dump date? –  ChrisLiaw May 12 at 14:53
    
@ChrisLiaw Yes, that's correct (excluding deleted data, of course) –  Tim Stone May 12 at 15:02
    
Wow! Thanks! That's very useful for offline analysis! –  ChrisLiaw May 13 at 5:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .