I'm trying to synchronize tags using the Stack Exchange API and this works for about 180 pages, but around 180 pages it always ends with an exception that seems to indicate that there is no HTTP response. It logs the URL that fails, when I open it in a browser it does return a correct JSON response.

We use:

• Spring boot
• Feign
• ApacheHttpClient (Because our default http client did not support GZIP responses)

And do honor the following API rules:

• Throttle, maximum of 29 requests in 1 second (in practice, our request rate is much lower)
• Honor the backoff response (non found in response)
• We use a key and have enough credits

This is an example request URL that seems to fail:

https://api.stackexchange.com/2.2/tags?site=stackoverflow&key=USINGAVALIDKEY&order=asc&sort=name&page=181&pagesize=100

And the response:

[StackoverflowRepository#getTags] Accept: application/json;charset=UTF-8
[StackoverflowRepository#getTags] Content-Type: application/json;charset=UTF-8
[StackoverflowRepository#getTags] Accept-Encoding: gzip
[StackoverflowRepository#getTags] Accept-Encoding: deflate
[StackoverflowRepository#getTags] ---> END HTTP (0-byte body)
[StackoverflowRepository#getTags] <--- ERROR NoHttpResponseException: api.stackexchange.com:443 failed to respond (82ms)
[StckoverflowRepository#getTags] org.apache.http.NoHttpResponseException: api.stackexchange.com:443 failed to respond
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
    at feign.httpclient.ApacheHttpClient.execute(ApacheHttpClient.java:85)
    at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:98)
    at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:77)
    at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:102)
    at com.sun.proxy.$Proxy376.getTags(Unknown Source)

Previous response does not contain a backoff parameter:

Accept: application/json;charset=UTF-8
Content-Type: application/json;charset=UTF-8
Accept-Encoding: gzip
Accept-Encoding: deflate
---> END HTTP (0-byte body)
<--- HTTP/1.1 200 OK (117ms)
access-control-allow-credentials: false
access-control-allow-methods: GET, POST
access-control-allow-origin: *
cache-control: private
content-type: application/json; charset=utf-8
date: Thu, 20 Jun 2019 09:28:22 GMT
x-content-type-options: nosniff

    {
      "items": [
        {
          "has_synonyms": false,
          "is_moderator_only": false,
          "is_required": false,
          "count": 34,
          "name": "genbank"
        },
        
        [OTHER 99 RESULTS HERE]
        
      ],
      "has_more": true,
      "quota_max": 10000,
      "quota_remaining": 9451
    }

And last but not least, the repository used:

@FeignClient(
        name = "stackoverflowRepository",
        url = "${stackoverflow.api.client.endpoint}",
        decode404 = true,
        configuration = StackoverflowClientFeignConfiguration.class
)
public interface StackoverflowRepository {
    /**
     *
     * @param filter
     * @param site
     * @param key
     * @param order
     * @param sort
     * @param page starts from 1. Passing 0 will raise an error on the API side
     * @param pageSize
     * @return
     */
    @GetMapping(
            value = "/tags",
            produces = MediaType.APPLICATION_JSON_UTF8_VALUE,
            consumes = MediaType.APPLICATION_JSON_UTF8_VALUE)
    ResponseEntity<StackoverflowResponse<Tag>> getTags(
            @RequestParam(name = "filter", required = false) String filter,
            @RequestParam(name = "site", required = false) String site,
            @RequestParam(name = "key", required = false) String key,
            @RequestParam(name = "order", required = false) String order,
            @RequestParam(name = "sort", required = false) String sort,
            @RequestParam(name = "fromdate", required = false) Long fromDate,
            @RequestParam(name = "todate", required = false) Long toDate,
            @RequestParam(name = "page", required = false) int page,
            @RequestParam(name = "pagesize", required = false) int pageSize);

}