2

I'm attempting to request some information using the Stack Exchange API. Supposedly, it returns responses compress with GZip. Using Microsoft's System.IO.Compression.GZipStream or Ionic.Zlib.GZipStream, I'm always told the data is not in GZip format... Even though the response's Content-Encoding is "gzip".

I've also written the bytes from the response to a file and transferred it to a Linux box and attempted to gunzip it, but that also returned an error.

What am I doing wrong? I am sending Accept-Encoding: gzip and getting Content-Encoding: gzip...

System.IO.InvalidDataException - The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.

Here's is runnable code:

internal static readonly string TagSearchUrl = "http://api.stackexchange.com/2.1/tags?site=stackoverflow&inname=c#";

internal static string FetchURL(string url) {
    var req = HttpWebRequest.Create(url);
    req.Headers["Accept-Encoding"] = "gzip";
    var resp = req.GetResponse();
    var contentType = resp.Headers["Content-Encoding"];
    var responseString = new StreamReader(resp.GetResponseStream()).ReadToEnd();
    return UnZip(responseString);
}

public static string UnZip(string value) {
    byte[] byteArray = StringToByteArray(value);

    using (var ms = new System.IO.MemoryStream(byteArray)) {

        //using (var sr = new Ionic.Zlib.GZipStream(ms, Ionic.Zlib.CompressionMode.Decompress)) {
        using (var sr = new System.IO.Compression.GZipStream(ms, System.IO.Compression.CompressionMode.Decompress)) {
            byteArray = new byte[byteArray.Length];
            try {
                var count = sr.Read(byteArray, 0, byteArray.Length);
            }
            finally {
                sr.Close();
                ms.Close();
            }
        }
    }

    return ByteArrayToString(byteArray);
}

private static string ByteArrayToString(byte[] byteArray) {
    return System.Text.Encoding.UTF8.GetString(byteArray);
}

private static byte[] StringToByteArray(string value) {
    return System.Text.Encoding.UTF8.GetBytes(value);
}

1 Answer 1

4

You're treating the response as a UTF-8 encoded string. That's bound to fail with binary data like GZip. A quick fix is to use Encoding.GetEncoding("iso-8859-1") instead of UTF8 in StringToByteArray, and also pass this encoding to the StreamReader constructor in FetchURL.

That encoding maps bytes to codepoints one-to-one; see this answer on Stack Overflow.

A cleaner way would be to not switch back and forth between unicode characters (string, StreamReader) and binary data (byte[], MemoryStream) in the first place. Until you have unzipped the data, it does not constitute a "string" in any meaningful way.

3
  • The response has a ContentType of application/json; charset=utf-8.
    – Steve
    Jul 14, 2013 at 3:45
  • I was able to get it to work. Using UTF-8 encoding works fine. I had to change StreamReader and now work with byte[]s only until after I've unzipped the response.
    – Steve
    Jul 14, 2013 at 4:00
  • Yes, that's what I said. Of course the JSON is UTF8-encoded. But treating the raw gzip data as UTF8 is what makes no sense.
    – balpha Staff
    Jul 14, 2013 at 4:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .