Get a great many counts of questions, by select tags for a date range?

Question

I am trying to get the number of questions associated with a specific tag for a particular date range. I initially assumed the fromdate and todate parameters would do this. However, it appears that filters based on the creation date of the tag itself. So not what I want.

It appears I could use the /search route to do this, using a query like:
        /search?fromdate=1420070400&todate=1454025600&sort=creation&tagged=python

This is not ideal because I am trying to get ~2k tags, month by month, over a few years so I will quickly hit the rate limit.

I want to get how many questions of a particular tag were created every month for the last five years, this is 60 unique date ranges. I want to do this for ~2k tags. So with the available syntax of the search route I would need to make ~120k requests.

Answer 1

You can do this on the API with far less than 120K requests. Just use a different approach.
Instead of getting a total for every single month, offload that work to your app. Use a /search query to get the questions for the whole interval in question. EG:

• /search?pagesize=100&fromdate=1329523200&todate=1487376000&sort=creation&tagged=python
  (You will need to request each page of data until has_more is false.)

Then use creation_date to compute the monthly counts yourself.

You can further reduce the number of API calls by combining tags; EG:
          ...&tagged=python;java...
and then separating the tags back out again in your own code.

HOWEVER, even this approach will not work for high volume sites & tags. For example, in 5 years there have been about a million Java questions. It would take your full 10K quota just to fetch that data (10K calls times 100 questions per call).

So for super high volumes, the API is not the best tool for this.
The API was never meant for this kind of data analysis.

Other tools are provided for that, mainly the data dumps and the Data Explorer (SEDE).
For example, here is a SEDE query that graphs question growth, by month, for select tags.

         

SEDE also has an API now.