My filter is not returning any results. How to create a minimal filter?

Question

I'm having trouble getting the SO API Filters to cooperate. I've gone over the documentation as well as similar questions like this and this.

I'll use this call as an example since it successfully returns 100 [complete] records:

https://api.stackexchange.com/2.2/questions?order=desc&sort=creation&site=stackoverflow&pagesize=100&tagged=vba&key= < MY KEY >

But suppose I'm interested in many records but only two fields (ie., tags & creation_date).

---

So, following my interpretation of the Filter documentation:

• I go to /docs/create-filter, expand the ▲Filter menu
• check Make Unsafe (only testing at this point)
• click Unselect All → Show 40 types not returned...

• expand Question, choose the 2 fields I want to see, and Save.

  _{(Screenshots of above actions. Click to enlarge.)}
     

I'm given _]9 which I plug into my URL:

https://api.stackexchange.com/2.2/questions?order=desc&sort=creation&site=stackoverflow&pagesize=100&tagged=vba&filter=!)8aDT7YJ0t91pUS&key= < MY KEY >

No error; no results. I tried selecting Unsafe (my preference) got a different filter, but still no results.

---

As I understand it, my other option is to use the interface with a base filter of none:

When building a filter from scratch, the none built-in filter is useful. _(Source)

Result: no filter returned.

_{(Click to enlarge)}
I also tried -(4DDaV0k4t15m16 as my base filter" (instead of the word none):

_{(Click to enlarge)}
Great, a filter! So I try that one and get no results.

What am I missing here?

Answer 1

The key mistake here is neglecting the Wrapper Object, and this is a big flaw with the filter documentation.

The wrapper is the box that everything you get from the API comes in. You always get the wrapper object except in the grossest of server/API errors. ^{_{(In your case you were getting a wrapper of {}.)}}

The stuff you care about, the payload, is the total and/or in the items properties.

For this reason, every filter should have .wrapper.items and/or .wrapper.total selected. More on this, below.

Furthermore, your filter must return the error properties and the quota remaining, or your app/script WILL get restricted or denied by the API at some point.

A safe/proper base filter always includes:

1. backoff
2. error_id
3. error_message
4. error_name
5. quota_remaining

-- You need to monitor, and handle, errors and depleted quota, or bad things will be done to your app and/or your account.

Plus one of items or total.

This is the stuff you care about, and without items, the non-wrapper filters are effectively turned off.

The reason I say "one of" is because of experience and this bit of the Paging doc:

Fetching total can be equally as expensive as fetching items. Put another way, an application fetching total when not needed is potentially halving its performance. It is for this reason that total is not returned by default.

I've observed^{(But admittedly did not collect data)} that filters that have both total and items tend to bring the backoff hammer that much quicker. See below for more on paging.

Recommended properties to also always include:

1. has_more
2. quota_max

-- these are useful sanity checks for debug, development, and API errors. And they do not seem to "cost" anything.

Note that the type property can be useful for libraries, but is seldom needed otherwise.

---

For the above reasons, the Built In Filters from the Filters doc have terrible advice.
Never use the none or total built in filters as these leave you vulnerable to undetected and unhandled errors.

---

Filters for paging:

For obvious reason, if your app expects lots of results, your filter should also the page and has_more properties. I also recommend the page_size property.
Note that several paging bugs are still unresolved, so cross checking is recommended.

For paging, I make an initial call for just the total (plus the other base properties). Then I make addition items calls^{(without the total)} to page through the results, cross checking page numbers and has_more against the previous total.

---

List of canned, useful filters:

In practice, none of the "built in" filters are good.

• none and total omit crucial wrapper properties.
• default and withbody include too much cruft for every scenario I've seen.

Here's some filters I recommend:

1. Minimum base: !GeDDagKb4Bn(8 or with paging: !.UDo6l2k)Rw2_cSv
2. Getting just the total of something: !w-*Ytm8YGWVl9..xoC
3. General, most useful fields (includes paging): !pL7rWh2ni(rrzFy9DMm2uoGFIuY63wHG.FjpGSDliVkU0C_MimSYK(5-U4x9PWW7Hb

To use these, refer to this other answer:

1. Open the appropriate doc page for the path you are using. EG /questions doc.
   This automatically shows you only the filter sections that are relevant for that API path/query.

2. Click the filter [edit] control and paste in the minimum base filter from step 1 above. Press enter for it to take effect.

3. Now, add back in the payload properties that you care about. In your example they are questions.tags and questions.creation_date. (But you probably need questions.question_id too.)

4. Be sure to add back the page and page_size properties if more than one item (or total) is expected.

5. Press the save button.

6. Carefully copy the resulting filter string for use in your app.

Note: Don't bother with /filters/create. It is more trouble than it's worth and only useful if your app needs to generate filters programmatically -- which is rare.