c# - HttpWebResponse LastModified

Question

Is the HttpWebResponse.LastModified accurate? Is it always present? My project is to create a sort of a focused web crawler and I am stucked if I will use the hash value of a resource or just the HttpWebResponse.LastModified property to check the resource's "freshness".

Using the hash value means streaming the resource every time it's checked. This has a big impact on overall performance.

If I will just check the HttpWebResponse.LastModified, is it accurate?

Answer 1

HttpWebResponse.LastModified returns the value of the HTTP Last-Modified response header.

HTTP response headers are set by the HTTP server sending the response. It's completely up to the server if it sets the Last-Modified response header, and whether it sets it to an accurate value or not.

The Last-Modified response header is part of the Validation Model for Caching in HTTP. It is usually used in conjunction with the If-Modified-Since request header. You might want to read HTTP/1.1, part 6: Caching for the details.