The API responses are returned in gzipped format. But by using Content-Encoding gzip, the burden of unzipping is on the user of the API. On the other hand if the server uses Transfer-Encoding gzip, many http client libraries (for eg. Python's urllib) will do the unzipping automatically. The server gets the same bandwidth savings.
asked Jun 28, 2010 at 3:59
-
I dont know much about this area, but if this is true this would be very helpful for python API calls. Currently you need to read the request into a temporary file-like buffer then do the decompression with a separate gzip call -- it is kind of a pain.– swansonCommented Jun 28, 2010 at 4:09
-
1No, this is not the case. Look at the python wrappers here.– Nathan OsmanCommented Jun 28, 2010 at 4:54
-
3+ for a good question.– Sky SandersCommented Jun 28, 2010 at 18:31
-
1Related: stackoverflow.com/questions/11641923/…– Jo LissCommented Sep 29, 2013 at 21:41
Add a comment
|
2 Answers
It's the same as much of the web.
As a rule, you don't want Content-Encoding: gzip or Content-Encoding: deflate a good 99.9999% of the time (to underestimate a bit).
What you want is the actual content, exactly as it would be without the content-encoding, but to get it faster. That is to say, you want Transfer-Encoding: gzip.
Transfer-Encoding: gzip also makes better logical sense on the web, since it is hop-to-hop, so proxies can apply it for capable clients talking to non-capable servers, or undo it for non-capable clients talking to capable servers. At the same time, the burden of the hop-to-hop nature is nil, as a pass-through has the correct effect (while conceptually a proxy will undo and then redo the compression, the simplest way to have the same result is just to pass it through). It's also nice to get the final size rather than the compressed size, in the header, and there's no need to change e-tags between the different versions of the document (something that causes bugs in a lot of cases, e.g. last time I looked, mod-gzip correctly changed the e-tag when it should but had a bug in the implementation of doing so resulting in invalid e-tags), and all in all it works better in every way.
Content-Encoding: gzip should really only have been used when someone wanted to actually get the gzip-encoded stream as such (e.g. to save as a .gz file).
So all in all, Transfer-Encoding is the blatantly obvious choice both here and in just about every other case where Content-Encoding is used.
However, the support for Content-Encoding is vastly greater than for Transfer-Encoding, with tool-kits, apis and libraries for both client and server giving support for the former vastly outnumbering those that do for the latter, which results in a spiral - it's not worth servers supporting it when clients don't use it and you can kludge it with Content-Encoding, and it's not worth clients supporting it when servers don't use it and you can kludge it with Content-Encoding.
Which makes it a case of "if all the webservers, browsers and web-client toolkits jumped off a bridge, would you do it too?" To which the answer is "since we have to interoperate with those toolkits, yes".
-
If you wanted to gzip and chunk encoding the content and you wanted to use Transfer-Encoding and not Content-Encoding. What would you put in the Transfer-Encoding header? Would it be
gziporchunkedor something likegzip chunked? Commented May 25, 2015 at 11:24 -
1@CMCDragonkai
gzip chunkedto indicate that first GZip had been applied, and then chunked. (Which incidentally is a rule in itself; if chunked is applied along with another encoding, it must be the last applied, though it doesn't really make sense to do anything else anyway). Commented May 25, 2015 at 13:25 -
I checked some place else and it seemed that the correct notation is
gzip, chunked. Does it need a comma? Commented May 25, 2015 at 14:02 -
@CMCDragonkai sorry, yes:
gzip, chunked. The lack of comma above was just a typo, and it is required. For the reasons I give though, you might find just usingContent-Encodingfor the gzip works better in practice. Commented May 25, 2015 at 14:46 -
1I have to disagree with the opinion that TE is always preferred. TE makes sense when you are compressing on-the-fly. CE makes sense when you're serving already-compressed content directly from storage. For your textual static content (JS, CSS, SVG, maybe HTML), you should have already-compressed copies to serve; you don't want to re-compress them on every request. In that case use CE. Similarly, caches benefit from storing the compressed content if it will be requested repeatedly. Ideally you use TE only for dynamic content -- because that's the only content compressed on-the-fly. Commented Jan 17, 2017 at 6:24
-
1@KentonVarda the difference between static and on-the-fly is an implementation detail. The distinction is one that simply isn't present as far as the transfer protocol goes. Commented Jan 17, 2017 at 9:52
-
@JonHanna It's not an implementation detail. The question to ask is: are the bytes exactly the same, or merely equivalent. TE does not imply that the (compressed) bytes are exactly the same as served by any other request. CE does (assuming matching non-weak ETag). This is important for caches and especially important when serving range requests. If you are compressing on-the-fly, you can't really guarantee identical bytes; e.g. an update to the compression code might change things. Commented Jan 18, 2017 at 2:35
-
1@KentonVarda at the level of the protocol the difference between "static" and "on-the-fly" simply doesn't exist, so yes it is an implementation detail. You're correct that it can have that have that effect on byte-identity (though you could still use no, or a weak ETag) though it need not. That's another reason to favour TE in such cases, though in no way does it make CE better. Commented Jan 18, 2017 at 9:56
Transfer-Encoding is hop-by-hop, while Content-Encoding is end-to-end.
This means that if there is a proxy involved, anywhere, the proxy will see the TE gzip, unzip it, and not necessarily forward the request as TE gzip.
So, the choices are
- CE gzip and always know what you will be getting, requiring logic to decompress the response.
- TE gzip and never know what you will be getting requiring logic to decide whether to decompress the response and the logic to decompress it when required.
The logical choice is CE gzip.
answered Jun 28, 2010 at 5:39
