Cache fails to update due to missing ETAG header field

[dduenker]

Hi, thanks for this library. :)

I am currently implementing a web scraper, which utilizes it and have come across the following issue:
The web server i am contacting does not seem to provide a ETAG Header (https://www.koblenz.de/rathaus/verwaltung/pressemeldungen/).
The initial caching worked and i was happy. But then the content of the website got updated and my scraper never got the new content, as cached_path seems to think, that the cached content is still up to date.

Can you confirm this behavior or might this be something i did wrong?

Edit: I am using version 0.5.1

[epwalsh]

Hi @dduenker, this is expected given that the server doesn't provide an ETAG header. That said, you could use the freshness_lifetime setting to ensure the cache is periodically updated. Would that work for your use case?

[dduenker]

Thank you, i will try that.
Is there any reason not to use the last-modified and cache-control headers?

[epwalsh]

Is there any reason not to use the last-modified and cache-control headers?

No reason, that's a great idea actually. It wouldn't be hard to implement.. we could just treat either of those as the ETAG when the ETAG isn't present. Would you be interested in making a PR for that?

[dduenker]

Is there any reason not to use the last-modified and cache-control headers?

No reason, that's a great idea actually. It wouldn't be hard to implement.. we could just treat either of those as the ETAG when the ETAG isn't present. Would you be interested in making a PR for that?

Yes, i would be interested in doing this. :)

[epwalsh]

Great! Let me know if you have any questions

[epwalsh]

FYI it is possible to store an expiration time in the meta JSON file: https://github.com/epwalsh/rust-cached-path/blob/master/src/meta.rs#L20