DM-52542: Reduce storage and transmission size of historical channel table data

[ugyballoons]

Instead of saving objects or strings that contain the whole bucket keys for individual files and downloading every metadata file wholesale, store the sequence numbers and extensions in a way that can be sent over websocket and recreated in the client and donwnload historical metadata when requested.
This should both speed up the historical data polling task and the reduce the size of the transfer of data via websocket.

[Copilot]

Pull Request Overview

This PR optimizes storage and transmission of historical channel table data by replacing full bucket key storage with a structured format storing only sequence numbers and file extensions. Instead of downloading all metadata files upfront, metadata is now retrieved on-demand when requested by clients.

Key changes:

• Introduced structured data storage format using sequence numbers and extension patterns
• Implemented extension deduplication strategy with default extensions and exceptions
• Moved metadata retrieval from polling to on-demand fetching
• Added client-side reconstruction of table data from structured format

Reviewed Changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
python/lsst/ts/rubintv/background/historicaldata.py	Core implementation of structured event storage and extension optimization
python/lsst/ts/rubintv/models/models.py	New data structures for structured data, extension info, and specialized page data classes
python/lsst/ts/rubintv/handlers/api.py	Updated API endpoint to return structured data and fetch metadata on-demand
src/js/modules/convertTableData.ts	Client-side conversion of structured data back to table format
src/js/components/TableApp.tsx	Integration of client-side table reconstruction
tests/background/historicaldata_test.py	Comprehensive test suite for new structured storage
tests/models/dataclass_test.py	Tests for new dataclass structures
src/js/modules/tests/convertTableData.test.js	Client-side conversion tests

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.