I am creating an R wrapper for the API. The use case for this is for doing data analysis on the SO data.
As I am building it, it seems like I would need to iterate over many pages in order to get all the data that I would want for an analysis. For instance, let's say I wanted to pull all SO users. Then I can call the users method and step through every page.
Is that an appropriate way to do this, or should I be using the data dump instead? I read through "Conscientious use of the API", but in this case I would be making large one-off requests (not polling).
I don't think the API is well suited to data analysis.
If you want to do analysis of the SE dataset(s), you should download the latest torrent file. Depending on which, and how much, data you need for your analysis, you might be able to combine the dump with some judicious use of the API to pull current metadata from the site.
I've tried to maintain an synced version of the data dump since the API beta. At first I thought I could do it but I kept running into issues. The first issue I hit was that the dumps come out late so you have to make up a few days just to get the initial sync. Then there is the issue of rate limits and how quickly you hit them when you are trying to track the stack overflow data.
To extend on Dave Swerskys answer: if you just have problems using the data from the torrent as such, but are fine with the implied gap of two to five weeks regarding the actuality of the data (see last FAQ item), you have another option, that is using the OData API instead: it's a versatile, fully RESTful API design and (IMHO) more thorough and mature already (despite being relatively young in itself) regarding several significant topics (e.g. HTTP cache control) than the custom Stack Overflow API as of version 1.0. As mentioned, and unfortunately, you won't get access to live data yet - I really hope this will change some day, but right now I wouldn't hold my breath, so Daves suggestion to combine this with some judicious use of the custom API is a very tempting one ;)