.net - Why does BCL GZipStream (with StreamReader) not reliably detect Data Errors with CRC32?

Question

The the other day I ran into the question GZipStream doesn't detect corrupt data (even CRC32 passes)? (Of which this might very well be a "duplicate", I have mixed feelings on the subject. I was also the one who added the CRC32 to the title, but in retrospect that feels out of place with the remainder of the post). After exploring the problem a bit on my own, I think that the issue is far greater than the other question initially portrays.

I expanded upon the other question and made the test code runnable under LINQPad and attempt to better showcase the CRC32 (Cyclic Redundancy Check) issue, if it does indeed exist. (Since the code is only a slight modification based upon the original it is possible that the test setup/methodology is flawed or there is another strange quirk/PEBCAK both.)

The results are odd because the corrupt data is not always causing an (any!) Exception to be raised. Note that only sometimes does the CRC32 check seem to actually be "working". The corrupt bytes that cause the index-out-of-range/bad header/bad footer can be ignored because we can assume these are killing the decompression prior to the CRC32 check (which is perfectly understandable, even if the IndexOutOfRangeException should likely be wrapped by a InvalidDataException) so,

Why is the CRC32 check significantly less reliable than it ought to be? (Why is it the case that there is "Invalid data (No Exception)" below?)

Since the GZip footer contains both the CRC32 and length of the uncompressed data is seems that the error detection rate should be "significantly higher" -- that is, I would not expect a single failing case below, much less a number of undetected corrupted streams. (Of course it's nice to detect a corrupt steam ASAP: but the final safe-guard checksum seems to be downright ignored in cases.)

Format is CorruptByteIndex+FailedDetections: Message:

0+0: System.IO.InvalidDataException:The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.
1+0: System.IO.InvalidDataException:The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.
2+0: System.IO.InvalidDataException:The compression mode specified in GZip header is unknown.
3+0: Good data (No Exception)
4+0: Good data (No Exception)
5+0: Good data (No Exception)
6+0: Good data (No Exception)
7+0: Good data (No Exception)
8+0: Good data (No Exception)
9+0: Good data (No Exception)
10+0: System.IO.InvalidDataException:Unknown block type. Stream might be corrupted.
11+1: Invalid data (No Exception)
12+1: System.IO.InvalidDataException:Found invalid data while decoding.
13+1: System.IO.InvalidDataException:Found invalid data while decoding.
14+1: System.IO.InvalidDataException:Found invalid data while decoding.
15+1: System.IO.InvalidDataException:Found invalid data while decoding.
16+1: System.IO.InvalidDataException:Found invalid data while decoding.
17+2: Invalid data (No Exception)
18+2: System.IO.InvalidDataException:Found invalid data while decoding.
19+2: System.IndexOutOfRangeException:Index was outside the bounds of the array.
20+2: System.IndexOutOfRangeException:Index was outside the bounds of the array.
21+3: Invalid data (No Exception)
22+3: System.IndexOutOfRangeException:Index was outside the bounds of the array.
23+3: System.IndexOutOfRangeException:Index was outside the bounds of the array.
24+4: Invalid data (No Exception)
25+4: System.IndexOutOfRangeException:Index was outside the bounds of the array.
26+4: System.IndexOutOfRangeException:Index was outside the bounds of the array.
27+4: System.IndexOutOfRangeException:Index was outside the bounds of the array.
28+4: System.IndexOutOfRangeException:Index was outside the bounds of the array.
29+5: Invalid data (No Exception)
30+5: System.IndexOutOfRangeException:Index was outside the bounds of the array.
31+6: Invalid data (No Exception)
32+7: Invalid data (No Exception)
33+7: System.IndexOutOfRangeException:Index was outside the bounds of the array.
34+7: System.IndexOutOfRangeException:Index was outside the bounds of the array.
35+7: System.IndexOutOfRangeException:Index was outside the bounds of the array.
36+8: Invalid data (No Exception)
37+8: System.IndexOutOfRangeException:Index was outside the bounds of the array.
38+8: System.IndexOutOfRangeException:Index was outside the bounds of the array.
39+9: Invalid data (No Exception)
40+9: System.IndexOutOfRangeException:Index was outside the bounds of the array.
41+9: System.IndexOutOfRangeException:Index was outside the bounds of the array.
42+10: Invalid data (No Exception)
43+10: System.IO.InvalidDataException:Found invalid data while decoding.
44+10: System.IndexOutOfRangeException:Index was outside the bounds of the array.
45+10: System.IO.InvalidDataException:Found invalid data while decoding.
46+11: Invalid data (No Exception)
47+11: System.IndexOutOfRangeException:Index was outside the bounds of the array.
48+11: System.IndexOutOfRangeException:Index was outside the bounds of the array.
49+11: System.IndexOutOfRangeException:Index was outside the bounds of the array.
50+12: Invalid data (No Exception)
51+12: System.IndexOutOfRangeException:Index was outside the bounds of the array.
52+12: System.IndexOutOfRangeException:Index was outside the bounds of the array.
53+13: Invalid data (No Exception)
54+13: System.IndexOutOfRangeException:Index was outside the bounds of the array.
55+14: Invalid data (No Exception)
56+14: System.IndexOutOfRangeException:Index was outside the bounds of the array.
57+15: Invalid data (No Exception)
58+15: System.IndexOutOfRangeException:Index was outside the bounds of the array.
59+15: System.IndexOutOfRangeException:Index was outside the bounds of the array.
60+16: Invalid data (No Exception)
61+17: Invalid data (No Exception)
62+18: Invalid data (No Exception)
63+19: Invalid data (No Exception)
64+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
65+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
66+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
67+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
68+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
69+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
70+19: System.IO.InvalidDataException:Found invalid data while decoding.
71+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
72+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
73+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
74+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
75+19: System.IO.InvalidDataException:Found invalid data while decoding.
76+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
77+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
78+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
79+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
80+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
81+19: System.IO.InvalidDataException:Found invalid data while decoding.
82+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
83+20: Invalid data (No Exception)
84+21: Invalid data (No Exception)
85+22: Invalid data (No Exception)
86+22: System.IndexOutOfRangeException:Index was outside the bounds of the array.
87+23: Invalid data (No Exception)
88+24: Invalid data (No Exception)
89+25: Invalid data (No Exception)
90+25: System.IndexOutOfRangeException:Index was outside the bounds of the array.
91+26: Invalid data (No Exception)
92+26: System.IO.InvalidDataException:Found invalid data while decoding.
93+26: System.IndexOutOfRangeException:Index was outside the bounds of the array.
94+27: Invalid data (No Exception)
95+27: System.IndexOutOfRangeException:Index was outside the bounds of the array.
96+27: System.IndexOutOfRangeException:Index was outside the bounds of the array.
97+28: Invalid data (No Exception)
98+28: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
99+28: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
100+29: Invalid data (No Exception)
101+30: Invalid data (No Exception)
102+31: Invalid data (No Exception)
103+32: Invalid data (No Exception)
104+32: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
105+33: Invalid data (No Exception)
106+34: Invalid data (No Exception)
107+35: Invalid data (No Exception)
108+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
109+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
110+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
111+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
112+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
113+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
114+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
115+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
116+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
117+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
118+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
119+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
120+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
121+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
122+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
123+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
124+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
125+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
126+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
127+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated 

Answer 1

Preface

.NET [4 and previous] users should not use the Microsoft-provided GZipStream or DeflateStream classes under any circumstances, unless Microsoft replaces them completely with something that works. Use the DotNetZip library instead.

Update to Preface

The .NET Framework 4.5 and later have fixed the compression problem, and GZipStream and DeflateStream use zlib in those versions. I do not know if the CRC problem referenced below has been fixed.

Another Update

The CRC bug is not only not fixed, but Microsoft has decided that they won't fix it!

 

My response in Why does my C# gzip produce a larger file than Fiddler or PHP? shows that this behavior does not reflect a proper implementation of gzip corruption detection. In all of the cases tested, the error would be detected by a proper implementation. (That response also notes why seven of the cases result in good data.)

Another question is: how does this "compression" method produce 174 bytes of output from a 94-byte string? Especially seeing as how the string is compressible -- gzip reduces it to 84-bytes, even with the overhead of an 18-byte header and trailer. Can you provide a hex dump of that 174 bytes?