7

I'm trying to synchronize tags using the Stack Exchange API and this works for about 180 pages, but around 180 pages it always ends with an exception that seems to indicate that there is no HTTP response. It logs the URL that fails, when I open it in a browser it does return a correct JSON response.

We use:

  • Spring boot
  • Feign
  • ApacheHttpClient (Because our default http client did not support GZIP responses)

And do honor the following API rules:

  • Throttle, maximum of 29 requests in 1 second (in practice, our request rate is much lower)
  • Honor the backoff response (non found in response)
  • We use a key and have enough credits

This is an example request URL that seems to fail:

https://api.stackexchange.com/2.2/tags?site=stackoverflow&key=USINGAVALIDKEY&order=asc&sort=name&page=181&pagesize=100

And the response:

[StackoverflowRepository#getTags] Accept: application/json;charset=UTF-8
[StackoverflowRepository#getTags] Content-Type: application/json;charset=UTF-8
[StackoverflowRepository#getTags] Accept-Encoding: gzip
[StackoverflowRepository#getTags] Accept-Encoding: deflate
[StackoverflowRepository#getTags] ---> END HTTP (0-byte body)
[StackoverflowRepository#getTags] <--- ERROR NoHttpResponseException: api.stackexchange.com:443 failed to respond (82ms)
[StckoverflowRepository#getTags] org.apache.http.NoHttpResponseException: api.stackexchange.com:443 failed to respond
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
    at feign.httpclient.ApacheHttpClient.execute(ApacheHttpClient.java:85)
    at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:98)
    at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:77)
    at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:102)
    at com.sun.proxy.$Proxy376.getTags(Unknown Source)

Previous response does not contain a backoff parameter:

Accept: application/json;charset=UTF-8
Content-Type: application/json;charset=UTF-8
Accept-Encoding: gzip
Accept-Encoding: deflate
---> END HTTP (0-byte body)
<--- HTTP/1.1 200 OK (117ms)
access-control-allow-credentials: false
access-control-allow-methods: GET, POST
access-control-allow-origin: *
cache-control: private
content-type: application/json; charset=utf-8
date: Thu, 20 Jun 2019 09:28:22 GMT
x-content-type-options: nosniff

    {
      "items": [
        {
          "has_synonyms": false,
          "is_moderator_only": false,
          "is_required": false,
          "count": 34,
          "name": "genbank"
        },

        [OTHER 99 RESULTS HERE]

      ],
      "has_more": true,
      "quota_max": 10000,
      "quota_remaining": 9451
    }

And last but not least, the repository used:

@FeignClient(
        name = "stackoverflowRepository",
        url = "${stackoverflow.api.client.endpoint}",
        decode404 = true,
        configuration = StackoverflowClientFeignConfiguration.class
)
public interface StackoverflowRepository {
    /**
     *
     * @param filter
     * @param site
     * @param key
     * @param order
     * @param sort
     * @param page starts from 1. Passing 0 will raise an error on the API side
     * @param pageSize
     * @return
     */
    @GetMapping(
            value = "/tags",
            produces = MediaType.APPLICATION_JSON_UTF8_VALUE,
            consumes = MediaType.APPLICATION_JSON_UTF8_VALUE)
    ResponseEntity<StackoverflowResponse<Tag>> getTags(
            @RequestParam(name = "filter", required = false) String filter,
            @RequestParam(name = "site", required = false) String site,
            @RequestParam(name = "key", required = false) String key,
            @RequestParam(name = "order", required = false) String order,
            @RequestParam(name = "sort", required = false) String sort,
            @RequestParam(name = "fromdate", required = false) Long fromDate,
            @RequestParam(name = "todate", required = false) Long toDate,
            @RequestParam(name = "page", required = false) int page,
            @RequestParam(name = "pagesize", required = false) int pageSize);

}
6
  • Is that the real URL which is failing? Your getTags method has some other parameters but am I correct in assuming they're null?
    – Glorfindel
    Jun 20, 2019 at 10:24
  • Yes you are right. The URL posted is the URL requested, the other parameters are optional and are excluded when they are null
    – guido
    Jun 20, 2019 at 10:59
  • OK, just checking. I have no experience with this API method, but I'm heavily using Java as well. You don't need an 3rd party HTTP client per sé; Java has built-in support for gzip (see e.g. softwareengineering.stackexchange.com/a/392969/173647 for how I call the API). But I'm not sure if switching to that will solve your problem.
    – Glorfindel
    Jun 20, 2019 at 11:02
  • Thank you for thinking along, but I feel that using using standard Java is a bit low level for API communication as using a framework as Feign takes away quite a bit of the boilerplate code. Also, I feel the problem is in a different direction, because it works fine for the first 180 pages. That said, it might be worth trying out your code in a sandbox to see if I get different results and/or it might give more insight into what is causing this problem.
    – guido
    Jun 20, 2019 at 11:49
  • 1
    Status-reviewed on MSE earlier: The API is returning incorrect values for has_more
    – rene
    Jan 6, 2023 at 12:44
  • 1

2 Answers 2

7

Yes, this is a serious bug. You can replicate it from any browser using the code below.

Every time I run it, it fetches 180 pages at an average rate of 5.51 calls per second.

On the 181st call I get:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://api.stackexchange.com/2.2/tags?page=181&pagesize=100&sort=name&order=desc&site=stackoverflow&key=redacted.
(Reason: CORS header ‘Access-Control-Allow-Origin’ missing).

TypeError: NetworkError when attempting to fetch resource.

Fetching the 181st page singly works normally.

It is not a particular tag that does it as fetching tags by name (ascending) fails at page 181 the same as fetching tags by name (descending).
(Note that there are currently 553 pages of tags, so fetching 181 pages desc gives all different values than fetching 181 pages asc.)

Worse, after only 382 calls on today's 10K quota, I got:

We're sorry...

There are an unusual number of requests coming from this IP address.

To protect our users, we can't process any more requests from this IP address right now.

We'll restore access as quickly as possible, so try again soon.

If you believe you have reached this page in error, contact us.

Not once was backoff, error_id, or error_name returned by the API.


Complete Code to Replicate:

const sortOrder   = "desc";  // "asc" or "desc"
const cYourKey    = "&key=U4DMV*8nvpm3EOpvf69Rxw(("; // best to use your own
const cMaxPages   = 300;
const tagStats    = {};
let PageNm      = 1;
let timeDiffSum = 0;
let timeDiffCnt = 0;
let lastTime    = null;

getPageOfTags (PageNm);
function sessionStat () {
    console.log (`Fetched ${PageNm} pages from the API.`);
    console.log (`Found ${Object.keys(tagStats).length} tag names.`);
    console.log (`Average API rate was ${ (1000.0 * timeDiffCnt / timeDiffSum).toFixed(2) } calls per second.`);
}
function getPageOfTags (pageNum) {
    fetch (
        `https://api.stackexchange.com/2.2/tags?page=${pageNum}&pagesize=100&sort=name&order=${sortOrder}&site=stackoverflow${cYourKey}`
    )
    .then (function (response) {
        if (response.ok)  return response.json();
        throw new Error (`API problem: Error ${response.status}, ${response.statusText}`);
    } )
    .then (function (rJsn) {
        if (rJsn.backoff  ||  rJsn.error_id) {
            throw new Error (`From API: Backoff ${rJsn.backoff}, Error: ${rJsn.error_id}, ${rJsn.error_message}, ${rJsn.error_name}`);
        }
        let timenow = (new Date() ).getTime();
        if (lastTime) {
            let timediff    = timenow - lastTime;
            timeDiffCnt++;
            timeDiffSum    += timediff;
        }
        lastTime    = timenow;
        console.log (`Page:${ (''+PageNm).padStart(4)}, Quota remaining: ${rJsn.quota_remaining}; Items found: ${rJsn.items.length}`, timenow);

        let allDone = ! rJsn.has_more;
        let zTags   = rJsn.items;
        for (let zTag of zTags) {
            tagStats[zTag.name]  = zTag.count;
        }
        if (rJsn.has_more) {
            if (PageNm >= cMaxPages)
                allDone = true;
            else {
                PageNm++;
                getPageOfTags (PageNm);
            }
        }
        if (allDone) {
            //--- Do final processing here...
            console.log ("All done. Running sessionStat()...");
            sessionStat ();
        }
    } )
    .catch (function (error) {
        console.log (`Page:${ ('' + PageNm).padStart(4)}, ERROR:`);
        console.error (error);
    } );
};

5
  • The API endpoint is also behind the HA proxy which is probably the culprit here. I never considered to be 100% guaranteed of a certain request rate only based on back-off. IP throttling is just another fail case that your code needs to handle. See also meta.stackexchange.com/questions/243773/… and meta.stackexchange.com/questions/249893/…
    – rene
    Jul 3, 2019 at 8:31
  • 2
    @rene, note that at no time did we receive a 503 error, nor a backoff (nor any other error from the API itself). It's not clear that this is the same thing as your referenced comments since they at least had some indication. This is not any kind of legit IP throttling. When the API fails to deliver necessary results and per its own documentation, that is a bug. (Or probably 2 related bugs in this case.) Jul 3, 2019 at 15:11
  • A similar issue was reported on MSE about 1/2 hour ago. The statement in an answer is that a fix has been deployed. I have not yet verified using the above code that the problem reported here is actually fixed. I probably won't be able to do so for a couple/few hours.
    – Makyen
    Nov 15, 2019 at 0:05
  • After testing, the issue was not resolved by the change which was made as a result of the bug report post on MSE.
    – Makyen
    Nov 16, 2019 at 19:24
  • However, when I tested a second time, after adding a &key=<SE API key from other script> to the request, the code correctly fetched all 300 pages to which it was limited. Adding the &key was necessary in order to test a second time due to requests without a key being limited to 300/day, which means it's only possible to test the 180 requests once per day without a key.
    – Makyen
    Nov 16, 2019 at 19:34
1

This bug has actually been fixed, minutes after it was reported on MSE:

Thanks for reporting! A fix for this was just deployed, and the CORS headers are getting sent again.

A more detailed explanation is available on an answer to the same bug report on MSO.

The snippet provided by Brock Adams correctly fetches all 300 pages without gettting any CORS errors.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .