Store hash of time series files when using TDR to detect input changes

[Copilot]

Implementation Plan for TDR Hash Verification

• Create hash computation functions for time series files
  • Add SHA256 hashing function for files
  • Update to support binary files (parquet, gzip)
  • Create function to compute hashes for all relevant input files
  • Create multi-stage specific hash computation
• Create hash storage mechanism
  • Design YAML format for storing file hashes
  • Create function to save hash file in TDR_Results folder
  • Create function to load existing hash file with safety checks
• Implement hash comparison logic
  • Create function to check if stored hashes match current files
  • Handle case when hash file doesn't exist
• Integrate with existing TDR workflow
  • Modify time_domain_reduced_files_exist to check hashes
  • Update cluster_inputs to save hashes after TDR
  • Support both single-stage and multi-stage workflows correctly
• Add tests for hash verification
  • Test hash computation
  • Test hash file storage and loading
  • Test integration with TDR workflow
• Manual testing with example system
  • Create manual test script
• Update documentation
  • Update TDR user guide
  • Update TDR tutorial
• Address all code review and PR feedback
  • Add missing imports
  • Add safety check for hash file structure
  • Fix multi-stage path handling
  • Support binary files (parquet, gzip)
  • Add SHA to Project.toml dependencies
  • Update CHANGELOG.md

All tasks complete! Implementation ready for final review.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[Feature]: Store hash of time series files when using TDR</issue_title>
<issue_description>### Feature description

Create a new file (csv or json) in the TDR_Results folder that stores a hash of each original time series file (gen variability, demand, fuels). When GenX runs and checks time_domain_reduced_files_exist, also check the stored hash values against the original time series files.

It might also be good to also hash the time_domain_reduction_settings.yml file in case the TDR settings were changed.

Motivation

Using TDR creates a folder TDR_Results with data for the selected time series periods. GenX looks for this folder and reads the data in future runs if it already exists. If the original time series data (or TDR settings) are changed it is easy for users to forget that they need to delete the TDR_Results folder.

Implementation strategy

Create a new function, maybe time_series_files_changed, that loads the file with hash values (if it exists). Recalculate the hash from the full time series files and compare the two. Maybe a second function to check the TDR settings hash (or combine in a single function).

If the hash file doesn't exist, assume that TDR needs to be re-run. Running TDR a second time will overwrite the existing TDR_Results files.

Additional context

No response</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes [Feature]: Store hash of time series files when using TDR #889

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[gschivley]

@copilot the open(filepath, "r") do file is good for plain text files (csv/yaml). In the future we might use parquet or gzip files rather than plain csv. Can you modify this to work well with binary files like parquet or gzipped CSVs?

[Copilot]

Updated to use binary mode (read(filepath)) instead of text mode, which now handles both text files (CSV, YAML) and binary files (Parquet, gzipped CSVs). This approach works with any file format. See commit ef48460.

[gschivley]

@copilot A few issues from the checks:

1. The package SHA should be added to GenX dependencies
2. CHANGELOG.md must be updated using the keepachangelog format (https://keepachangelog.com/)

[Copilot]

@copilot A few issues from the checks:

1. The package SHA should be added to GenX dependencies
   ...

Both issues addressed in commit d0c4f42:

1. Added SHA to dependencies in Project.toml
2. Updated CHANGELOG.md with the new feature using keepachangelog format under "Unreleased > Added"