Stack Apps is a question and answer site for apps, scripts, and development with the Stack Exchange API. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

The API responses are returned in gzipped format. But by using Content-Encoding gzip, the burden of unzipping is on the user of the API. On the other hand if the server uses Transfer-Encoding gzip, many http client libraries (for eg. Python's urllib) will do the unzipping automatically. The server gets the same bandwidth savings.

share|improve this question
I dont know much about this area, but if this is true this would be very helpful for python API calls. Currently you need to read the request into a temporary file-like buffer then do the decompression with a separate gzip call -- it is kind of a pain. – swanson Jun 28 '10 at 4:09
No, this is not the case. Look at the python wrappers here. – Nathan Osman Jun 28 '10 at 4:54
+ for a good question. – Sky Sanders Jun 28 '10 at 18:31
Related:… – Jo Liss Sep 29 '13 at 21:41
up vote 8 down vote accepted

It's the same as much of the web.

As a rule, you don't want Content-Encoding: gzip or Content-Encoding: deflate a good 99.9999% of the time (to underestimate a bit).

What you want is the actual content, exactly as it would be without the content-encoding, but to get it faster. That is to say, you want Transfer-Encoding: gzip.

Transfer-Encoding: gzip also makes better logical sense on the web, since it is hop-to-hop, so proxies can apply it for capable clients talking to non-capable servers, or undo it for non-capable clients talking to capable servers. At the same time, the burden of the hop-to-hop nature is nil, as a pass-through has the correct effect (while conceptually a proxy will undo and then redo the compression, the simplest way to have the same result is just to pass it through). It's also nice to get the final size rather than the compressed size, in the header, and there's no need to change e-tags between the different versions of the document (something that causes bugs in a lot of cases, e.g. last time I looked, mod-gzip correctly changed the e-tag when it should but had a bug in the implementation of doing so resulting in invalid e-tags), and all in all it works better in every way.

Content-Encoding: gzip should really only have been used when someone wanted to actually get the gzip-encoded stream as such (e.g. to save as a .gz file).

So all in all, Transfer-Encoding is the blatantly obvious choice both here and in just about every other case where Content-Encoding is used.

However, the support for Content-Encoding is vastly greater than for Transfer-Encoding, with tool-kits, apis and libraries for both client and server giving support for the former vastly outnumbering those that do for the latter, which results in a spiral - it's not worth servers supporting it when clients don't use it and you can kludge it with Content-Encoding, and it's not worth clients supporting it when servers don't use it and you can kludge it with Content-Encoding.

Which makes it a case of "if all the webservers, browsers and web-client toolkits jumped off a bridge, would you do it too?" To which the answer is "since we have to interoperate with those toolkits, yes".

share|improve this answer
If you wanted to gzip and chunk encoding the content and you wanted to use Transfer-Encoding and not Content-Encoding. What would you put in the Transfer-Encoding header? Would it be gzip or chunked or something like gzip chunked? – CMCDragonkai May 25 '15 at 11:24
@CMCDragonkai gzip chunked to indicate that first GZip had been applied, and then chunked. (Which incidentally is a rule in itself; if chunked is applied along with another encoding, it must be the last applied, though it doesn't really make sense to do anything else anyway). – Jon Hanna May 25 '15 at 13:25
I checked some place else and it seemed that the correct notation is gzip, chunked. Does it need a comma? – CMCDragonkai May 25 '15 at 14:02
@CMCDragonkai sorry, yes: gzip, chunked. The lack of comma above was just a typo, and it is required. For the reasons I give though, you might find just using Content-Encoding for the gzip works better in practice. – Jon Hanna May 25 '15 at 14:46

Transfer-Encoding is hop-by-hop, while Content-Encoding is end-to-end.

This means that if there is a proxy involved, anywhere, the proxy will see the TE gzip, unzip it, and not necessarily forward the request as TE gzip.

So, the choices are

  • CE gzip and always know what you will be getting, requiring logic to decompress the response.
  • TE gzip and never know what you will be getting requiring logic to decide whether to decompress the response and the logic to decompress it when required.

The logical choice is CE gzip.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .