Bad format returned by Stack Exchange API, not GZip?

Question

I'm attempting to request some information using the Stack Exchange API. Supposedly, it returns responses compress with GZip. Using Microsoft's System.IO.Compression.GZipStream or Ionic.Zlib.GZipStream, I'm always told the data is not in GZip format... Even though the response's Content-Encoding is "gzip".

I've also written the bytes from the response to a file and transferred it to a Linux box and attempted to gunzip it, but that also returned an error.

What am I doing wrong? I am sending Accept-Encoding: gzip and getting Content-Encoding: gzip...

System.IO.InvalidDataException - The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.

Here's is runnable code:

internal static readonly string TagSearchUrl = "http://api.stackexchange.com/2.1/tags?site=stackoverflow&inname=c#";

internal static string FetchURL(string url) {
    var req = HttpWebRequest.Create(url);
    req.Headers["Accept-Encoding"] = "gzip";
    var resp = req.GetResponse();
    var contentType = resp.Headers["Content-Encoding"];
    var responseString = new StreamReader(resp.GetResponseStream()).ReadToEnd();
    return UnZip(responseString);
}

public static string UnZip(string value) {
    byte[] byteArray = StringToByteArray(value);

    using (var ms = new System.IO.MemoryStream(byteArray)) {

        //using (var sr = new Ionic.Zlib.GZipStream(ms, Ionic.Zlib.CompressionMode.Decompress)) {
        using (var sr = new System.IO.Compression.GZipStream(ms, System.IO.Compression.CompressionMode.Decompress)) {
            byteArray = new byte[byteArray.Length];
            try {
                var count = sr.Read(byteArray, 0, byteArray.Length);
            }
            finally {
                sr.Close();
                ms.Close();
            }
        }
    }

    return ByteArrayToString(byteArray);
}

private static string ByteArrayToString(byte[] byteArray) {
    return System.Text.Encoding.UTF8.GetString(byteArray);
}

private static byte[] StringToByteArray(string value) {
    return System.Text.Encoding.UTF8.GetBytes(value);
}

Answer 1

You're treating the response as a UTF-8 encoded string. That's bound to fail with binary data like GZip. A quick fix is to use Encoding.GetEncoding("iso-8859-1") instead of UTF8 in StringToByteArray, and also pass this encoding to the StreamReader constructor in FetchURL.

That encoding maps bytes to codepoints one-to-one; see this answer on Stack Overflow.

A cleaner way would be to not switch back and forth between unicode characters (string, StreamReader) and binary data (byte[], MemoryStream) in the first place. Until you have unzipped the data, it does not constitute a "string" in any meaningful way.