After successfully retrieving 180 pages, the API gracelessly, semi-silently, fails

Question

I'm trying to synchronize tags using the Stack Exchange API and this works for about 180 pages, but around 180 pages it always ends with an exception that seems to indicate that there is no HTTP response. It logs the URL that fails, when I open it in a browser it does return a correct JSON response.

We use:

• Spring boot
• Feign
• ApacheHttpClient (Because our default http client did not support GZIP responses)

And do honor the following API rules:

• Throttle, maximum of 29 requests in 1 second (in practice, our request rate is much lower)
• Honor the backoff response (non found in response)
• We use a key and have enough credits

This is an example request URL that seems to fail:

https://api.stackexchange.com/2.2/tags?site=stackoverflow&key=USINGAVALIDKEY&order=asc&sort=name&page=181&pagesize=100

And the response:

[StackoverflowRepository#getTags] Accept: application/json;charset=UTF-8
[StackoverflowRepository#getTags] Content-Type: application/json;charset=UTF-8
[StackoverflowRepository#getTags] Accept-Encoding: gzip
[StackoverflowRepository#getTags] Accept-Encoding: deflate
[StackoverflowRepository#getTags] ---> END HTTP (0-byte body)
[StackoverflowRepository#getTags] <--- ERROR NoHttpResponseException: api.stackexchange.com:443 failed to respond (82ms)
[StckoverflowRepository#getTags] org.apache.http.NoHttpResponseException: api.stackexchange.com:443 failed to respond
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
    at feign.httpclient.ApacheHttpClient.execute(ApacheHttpClient.java:85)
    at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:98)
    at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:77)
    at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:102)
    at com.sun.proxy.$Proxy376.getTags(Unknown Source)

Previous response does not contain a backoff parameter:

Accept: application/json;charset=UTF-8
Content-Type: application/json;charset=UTF-8
Accept-Encoding: gzip
Accept-Encoding: deflate
---> END HTTP (0-byte body)
<--- HTTP/1.1 200 OK (117ms)
access-control-allow-credentials: false
access-control-allow-methods: GET, POST
access-control-allow-origin: *
cache-control: private
content-type: application/json; charset=utf-8
date: Thu, 20 Jun 2019 09:28:22 GMT
x-content-type-options: nosniff

    {
      "items": [
        {
          "has_synonyms": false,
          "is_moderator_only": false,
          "is_required": false,
          "count": 34,
          "name": "genbank"
        },

        [OTHER 99 RESULTS HERE]

      ],
      "has_more": true,
      "quota_max": 10000,
      "quota_remaining": 9451
    }

And last but not least, the repository used:

@FeignClient(
        name = "stackoverflowRepository",
        url = "${stackoverflow.api.client.endpoint}",
        decode404 = true,
        configuration = StackoverflowClientFeignConfiguration.class
)
public interface StackoverflowRepository {
    /**
     *
     * @param filter
     * @param site
     * @param key
     * @param order
     * @param sort
     * @param page starts from 1. Passing 0 will raise an error on the API side
     * @param pageSize
     * @return
     */
    @GetMapping(
            value = "/tags",
            produces = MediaType.APPLICATION_JSON_UTF8_VALUE,
            consumes = MediaType.APPLICATION_JSON_UTF8_VALUE)
    ResponseEntity<StackoverflowResponse<Tag>> getTags(
            @RequestParam(name = "filter", required = false) String filter,
            @RequestParam(name = "site", required = false) String site,
            @RequestParam(name = "key", required = false) String key,
            @RequestParam(name = "order", required = false) String order,
            @RequestParam(name = "sort", required = false) String sort,
            @RequestParam(name = "fromdate", required = false) Long fromDate,
            @RequestParam(name = "todate", required = false) Long toDate,
            @RequestParam(name = "page", required = false) int page,
            @RequestParam(name = "pagesize", required = false) int pageSize);

}

Answer 1

Yes, this is a serious bug. You can replicate it from any browser using the code below.

Every time I run it, it fetches 180 pages at an average rate of 5.51 calls per second.

On the 181st call I get:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://api.stackexchange.com/2.2/tags?page=181&pagesize=100&sort=name&order=desc&site=stackoverflow&key=redacted.
(Reason: CORS header ‘Access-Control-Allow-Origin’ missing).

TypeError: NetworkError when attempting to fetch resource.

Fetching the 181st page singly works normally.

It is not a particular tag that does it as fetching tags by name (ascending) fails at page 181 the same as fetching tags by name (descending).
(Note that there are currently 553 pages of tags, so fetching 181 pages desc gives all different values than fetching 181 pages asc.)

Worse, after only 382 calls on today's 10K quota, I got:

We're sorry...

There are an unusual number of requests coming from this IP address.

To protect our users, we can't process any more requests from this IP address right now.

We'll restore access as quickly as possible, so try again soon.

If you believe you have reached this page in error, contact us.

Not once was backoff, error_id, or error_name returned by the API.

---

Complete Code to Replicate:

const sortOrder   = "desc";  // "asc" or "desc"
const cYourKey    = "&key=U4DMV*8nvpm3EOpvf69Rxw(("; // best to use your own
const cMaxPages   = 300;
const tagStats    = {};
let PageNm      = 1;
let timeDiffSum = 0;
let timeDiffCnt = 0;
let lastTime    = null;

getPageOfTags (PageNm);
function sessionStat () {
    console.log (`Fetched ${PageNm} pages from the API.`);
    console.log (`Found ${Object.keys(tagStats).length} tag names.`);
    console.log (`Average API rate was ${ (1000.0 * timeDiffCnt / timeDiffSum).toFixed(2) } calls per second.`);
}
function getPageOfTags (pageNum) {
    fetch (
        `https://api.stackexchange.com/2.2/tags?page=${pageNum}&pagesize=100&sort=name&order=${sortOrder}&site=stackoverflow${cYourKey}`
    )
    .then (function (response) {
        if (response.ok)  return response.json();
        throw new Error (`API problem: Error ${response.status}, ${response.statusText}`);
    } )
    .then (function (rJsn) {
        if (rJsn.backoff  ||  rJsn.error_id) {
            throw new Error (`From API: Backoff ${rJsn.backoff}, Error: ${rJsn.error_id}, ${rJsn.error_message}, ${rJsn.error_name}`);
        }
        let timenow = (new Date() ).getTime();
        if (lastTime) {
            let timediff    = timenow - lastTime;
            timeDiffCnt++;
            timeDiffSum    += timediff;
        }
        lastTime    = timenow;
        console.log (`Page:${ (''+PageNm).padStart(4)}, Quota remaining: ${rJsn.quota_remaining}; Items found: ${rJsn.items.length}`, timenow);

        let allDone = ! rJsn.has_more;
        let zTags   = rJsn.items;
        for (let zTag of zTags) {
            tagStats[zTag.name]  = zTag.count;
        }
        if (rJsn.has_more) {
            if (PageNm >= cMaxPages)
                allDone = true;
            else {
                PageNm++;
                getPageOfTags (PageNm);
            }
        }
        if (allDone) {
            //--- Do final processing here...
            console.log ("All done. Running sessionStat()...");
            sessionStat ();
        }
    } )
    .catch (function (error) {
        console.log (`Page:${ ('' + PageNm).padStart(4)}, ERROR:`);
        console.error (error);
    } );
};

Answer 2

This bug has actually been fixed, minutes after it was reported on MSE:

Thanks for reporting! A fix for this was just deployed, and the CORS headers are getting sent again.

A more detailed explanation is available on an answer to the same bug report on MSO.

The snippet provided by Brock Adams correctly fetches all 300 pages without gettting any CORS errors.