Why Content-Encoding gzip rather than Transfer-Encoding gzip

Question

The API responses are returned in gzipped format. But by using Content-Encoding gzip, the burden of unzipping is on the user of the API. On the other hand if the server uses Transfer-Encoding gzip, many http client libraries (for eg. Python's urllib) will do the unzipping automatically. The server gets the same bandwidth savings.

Answer 1

It's the same as much of the web.

As a rule, you don't want Content-Encoding: gzip or Content-Encoding: deflate a good 99.9999% of the time (to underestimate a bit).

What you want is the actual content, exactly as it would be without the content-encoding, but to get it faster. That is to say, you want Transfer-Encoding: gzip.

Transfer-Encoding: gzip also makes better logical sense on the web, since it is hop-to-hop, so proxies can apply it for capable clients talking to non-capable servers, or undo it for non-capable clients talking to capable servers. At the same time, the burden of the hop-to-hop nature is nil, as a pass-through has the correct effect (while conceptually a proxy will undo and then redo the compression, the simplest way to have the same result is just to pass it through). It's also nice to get the final size rather than the compressed size, in the header, and there's no need to change e-tags between the different versions of the document (something that causes bugs in a lot of cases, e.g. last time I looked, mod-gzip correctly changed the e-tag when it should but had a bug in the implementation of doing so resulting in invalid e-tags), and all in all it works better in every way.

Content-Encoding: gzip should really only have been used when someone wanted to actually get the gzip-encoded stream as such (e.g. to save as a .gz file).

So all in all, Transfer-Encoding is the blatantly obvious choice both here and in just about every other case where Content-Encoding is used.

However, the support for Content-Encoding is vastly greater than for Transfer-Encoding, with tool-kits, apis and libraries for both client and server giving support for the former vastly outnumbering those that do for the latter, which results in a spiral - it's not worth servers supporting it when clients don't use it and you can kludge it with Content-Encoding, and it's not worth clients supporting it when servers don't use it and you can kludge it with Content-Encoding.

Which makes it a case of "if all the webservers, browsers and web-client toolkits jumped off a bridge, would you do it too?" To which the answer is "since we have to interoperate with those toolkits, yes".

Answer 2

Transfer-Encoding is hop-by-hop, while Content-Encoding is end-to-end.

This means that if there is a proxy involved, anywhere, the proxy will see the TE gzip, unzip it, and not necessarily forward the request as TE gzip.

So, the choices are

• CE gzip and always know what you will be getting, requiring logic to decompress the response.
• TE gzip and never know what you will be getting requiring logic to decide whether to decompress the response and the logic to decompress it when required.

The logical choice is CE gzip.