Take the 2-minute tour ×
Stack Apps is a question and answer site for apps, scripts, and development with the Stack Exchange API. It's 100% free, no registration required.

I'm attempting to request some information using the Stack Exchange API. Supposedly, it returns responses compress with GZip. Using Microsoft's System.IO.Compression.GZipStream or Ionic.Zlib.GZipStream, I'm always told the data is not in GZip format... Even though the response's Content-Encoding is "gzip".

I've also written the bytes from the response to a file and transferred it to a Linux box and attempted to gunzip it, but that also returned an error.

What am I doing wrong? I am sending Accept-Encoding: gzip and getting Content-Encoding: gzip...

System.IO.InvalidDataException - The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.

Here's is runnable code:

internal static readonly string TagSearchUrl = "http://api.stackexchange.com/2.1/tags?site=stackoverflow&inname=c#";

internal static string FetchURL(string url) {
    var req = HttpWebRequest.Create(url);
    req.Headers["Accept-Encoding"] = "gzip";
    var resp = req.GetResponse();
    var contentType = resp.Headers["Content-Encoding"];
    var responseString = new StreamReader(resp.GetResponseStream()).ReadToEnd();
    return UnZip(responseString);
}

public static string UnZip(string value) {
    byte[] byteArray = StringToByteArray(value);

    using (var ms = new System.IO.MemoryStream(byteArray)) {

        //using (var sr = new Ionic.Zlib.GZipStream(ms, Ionic.Zlib.CompressionMode.Decompress)) {
        using (var sr = new System.IO.Compression.GZipStream(ms, System.IO.Compression.CompressionMode.Decompress)) {
            byteArray = new byte[byteArray.Length];
            try {
                var count = sr.Read(byteArray, 0, byteArray.Length);
            }
            finally {
                sr.Close();
                ms.Close();
            }
        }
    }

    return ByteArrayToString(byteArray);
}

private static string ByteArrayToString(byte[] byteArray) {
    return System.Text.Encoding.UTF8.GetString(byteArray);
}

private static byte[] StringToByteArray(string value) {
    return System.Text.Encoding.UTF8.GetBytes(value);
}
share|improve this question
add comment

1 Answer 1

up vote 2 down vote accepted

You're treating the response as a UTF-8 encoded string. That's bound to fail with binary data like GZip. A quick fix is to use Encoding.GetEncoding("iso-8859-1") instead of UTF8 in StringToByteArray, and also pass this encoding to the StreamReader constructor in FetchURL.

That encoding maps bytes to codepoints one-to-one; see this answer on Stack Overflow.

A cleaner way would be to not switch back and forth between unicode characters (string, StreamReader) and binary data (byte[], MemoryStream) in the first place. Until you have unzipped the data, it does not constitute a "string" in any meaningful way.

share|improve this answer
    
The response has a ContentType of application/json; charset=utf-8. –  Steve Jul 14 '13 at 3:45
    
I was able to get it to work. Using UTF-8 encoding works fine. I had to change StreamReader and now work with byte[]s only until after I've unzipped the response. –  Steve Jul 14 '13 at 4:00
    
Yes, that's what I said. Of course the JSON is UTF8-encoded. But treating the raw gzip data as UTF8 is what makes no sense. –  balpha Jul 14 '13 at 4:35
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .