How do I see the differences between two or more SE API filters and merge into a filter which includes all requested properties (i.e. is a superset)?

Question

For SmokeDetector, we have several SE API filter values in use. Notably, there are separate filters for the /answers/*, /questions/* and /posts/* routes. From an organizational standpoint, I'd like to merge all these filter definitions into one filter value which is used for all requests, so we can be sure we're getting a consistent set of data for all requests.

Further complicating this is that, in the past, we've had multiple pull requests (PRs) making changes to these filters at the same time. Such PRs are normally separate in that each PR is adding or removing different properties to each of the multiple filters, but, usually, the PR is adding/removing properties consistently to/from all filters. The PRs usually don't depend on each other and are often submitted by different people, so the filters values are changed separately. The PRs, as is normal, include text changes to the filter values. Given how filters are generated, it's not possible to just merge the text change which is made by each PR, as adding or removing even a single property from a filter is not a progressive textual change. As a result, the actual changes to the filters need to be understood and applied to a new updated value. This can be quite inconvenient with multiple PRs changing multiple values (e.g. the resulting changes can involve > 10 different filter values).

I'm aware of being able to see what the filter values contain interactively using the various endpoint pages in the SE API documentation. I'm also aware of the filters/{filters} and filters/create endpoints which enable decoding a filter and creating a filter respectively.

How can I easily compare two, or more, filters, see what the differences are and generate a merged filter which is a superset of both, or all, of the filters?

Answer 1

To make it easier to merge filters, I wrote the following snippet. The detailed differences between each two filter combinations are a bit unwieldy to display in the snippet space, so they are output to the browser console.

function logSEAPIFilterDiffAndMerge(filtersIn, sourceAPIVersion, resultAPIVersion, unsafe) {
  return new Promise((resolve) => {
    'use strict'
    const key = 'YS5Q71XoOjfKU9m5LZHdqQ(('
    const seAPI = 'https://api.stackexchange.com/';
    $.get(`${seAPI}${sourceAPIVersion}/filters/${filtersIn.join(';')}?key=${key}&filter=default`)
      .then((response) => {
        const items = response.items;
        //Put the received items in the same order as the original list.
        items.sort((a, b) => filtersIn.indexOf(a.filter) - filtersIn.indexOf(b.filter));
        console.log('Input filter descriptions from SE API:', items);
        const itemFilters = items.map(({filter}) => filter);
        console.log('Filters:', itemFilters);
        const sets = items.map((item) => new Set(item.included_fields));
        const allFields = [].concat.apply([], items.map(({included_fields}) => included_fields));
        const uniqueFieldsSet = new Set(allFields);
        const uniqueFieldsArray = Array.from(uniqueFieldsSet.values());
        console.log('Unique fields:', uniqueFieldsArray);
        items.forEach((testFilter, testIndex) => {
          items.forEach((againstFilter, againstIndex) => {
            if (testIndex === againstIndex) {
              return;
            } //else
            const inTestNotInAgainst = testFilter.included_fields
              .filter((field) => !sets[againstIndex].has(field));
            console.log(`Fields in filter ${testIndex} but not in filter ${againstIndex}:`, inTestNotInAgainst);
          });
        });
        const postData = {
          key,
          //Leaving `base` blank appears to use the default filter, which isn't what we want.
          base: filtersIn[0],
          include: uniqueFieldsArray.join(';'),
          unsafe,
          filter: 'default',
        };
        if (!key) {
          delete postData.key;
        }
        $.post(`${seAPI}${resultAPIVersion}/filters/create`, postData)
          .then((postResponse) => {
            const mergedFilter = postResponse.items[0].filter;
            console.log('Merged filter:', mergedFilter);
            resolve(mergedFilter);
          });
      });
  });
}

function insertDefaults() {
  const defaultFiltersToMerge = [
    '!4z6S)cPO)zvpuDWsWTAUW(kaV6K6thsqi1tlYa', //apigetpost.py "answer"
    '!m)9.UaQrI5-DZXtlTpWhv2HroYRgS3dPhv.2vxV7fpGT*27rEHM.BKV1', //apigetpost.py "question" Does not get answer data
    '!1rs)sUKylwB)8isvCRk.xNu71LnaxjnPS12*pX*CEOKbPFwVFdHNxiMa7GIVgzDAwMa', //bodyfetcher.py /questions/{}
    '!fsv5ng(IaK_MBkZYCDWuA.U2DqLwdl*YEL_', //chatcommands.py /users/{}/posts
  ];
  $('#input-filters').val(defaultFiltersToMerge.join('\n'));
  $('#input-api-version').val('2.2');
  $('#result-api-version').val('2.3');
  $('#result-is-unsafe').prop('checked', false);
}

function getDataFromUI() {
  const filtersToMerge = $('#input-filters').val().split(/[\n\r;]+/g);
  const inputAPIVersion = $('#input-api-version').val();
  const resultAPIVersion = $('#result-api-version').val();
  const resultIsUnsafe = $('#result-is-unsafe').is(':checked');
  return {
    filtersToMerge,
    inputAPIVersion,
    resultAPIVersion,
    resultIsUnsafe,
  }
}

function mergeUIBasedFilters() {
  const {filtersToMerge, inputAPIVersion, resultAPIVersion, resultIsUnsafe} = getDataFromUI();
  logSEAPIFilterDiffAndMerge(filtersToMerge, inputAPIVersion, resultAPIVersion, resultIsUnsafe)
    .then((mergedFilter) => {
      $('#merged-filter-result').text(mergedFilter);
      $('#merged-filter-result-container').text(mergedFilter);
      $('.result-additional-text-container').text('See the browser console for detailed filter differences.');
    });
}

$('#use-default').on('click', insertDefaults);
$('#merge-filters').on('click', mergeUIBasedFilters);

textarea {
    width: calc(100vw - 20px);;
}
.api-version-input {
    width: 2em;
    margin: 5px 15px 5px 5px;
}
.action-button-container {
    display: flex;
    justify-content: space-between;
    margin-top: 5px;
}
#merged-filter-result {
    font-weight: bold;
}
.merged-filter-result-container {
    margin-top: 10px;
}

<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<div class="se-api-selection-container">SE API version:
    <label for="input-api-version">input:</label><input id="input-api-version" value="2.3" class="api-version-input">
    <label for="result-api-version">result:</label><input id="result-api-version" value="2.3" class="api-version-input">
    <label for="result-is-unsafe">unsafe result:</label><input id="result-is-unsafe" type="checkbox">
</div>
<label for="input-filters">Filters to merge (and report differences in the console):</label>
<textarea id="input-filters" value="" placeholder="Add a list of filters here. They can be on individual lines or separated by semicolons."></textarea><br/>
<div class="action-button-container">
    <button id="use-default" title="Fill the input and textarea fields with the values for SmokeDetector as of when this snippet was updated. These can be used for testing/demoing the snippet.">Use SmokeDetector filters</button><button id="merge-filters">Merge filters</button>
</div>
<div class="merged-filter-result-container">Merged filter: <span id="merged-filter-result"></span></div>
<div class="result-additional-text-container"></div>