2026 Storage overhaul #5226
Description
Context
- v2 Storage and Distribution System Specification #2224
- Gateway Specification #2089
- CIJTF Meeting Tracker Issue #4455
- How to do data fetching in Joystream applications #1096
- Distributor node - current progress and future plans #2553
Requirements
- A new infra release:
Origo#4766- Make Argus/Colossus/Apps(Orion) aware of the Operational status of the storage/distributior-nodes #4816
- Add support to mark storage-node/distrubtion-node as under maintenance by the Node operators #4765
- Improving Storage/Distribution related integration tests #4815
- Argus/Colossus QueryNode result caching to improve latency #4424
- Metaprotocol replacement of storage pallet #4940
- QN/ Mataptorocol Backwards Compatibility #4699
- Content directory smart contracts: Splitting the logic #1922
Scope
The general goal is to improve the content storage system independently from runtime constrains.
Prototype colossus / argus changes to implement listed features using a flexible DAO hosted API.
Develop meta-protocols to reflect necessary communication on chain for apps to be able to interoperate using preferably chain data to avoid SPOF.
Error reporting and tracking
- Lead infrastructure (error) monitoring #4452
- Proof-of-storage #2555
Lighthouse- An automated storage interrogation service #4270- QN: Improve Metaprotocol transactions error handling #4573
Update Access Policies
- Verified status for channels #2586 Channel Verification #4487 Explore channels as interoperable NFT assets #2925
- Standardise
Argusaccess policy signals in metadata #4332 - Uploader authentication #2815 Object Upload fails but still set as accepted on chain #5097 [Colossus] Resumable uploads #5043
- Runtime enforcement of channel storage limits #4819
- Argus: content delivery authentication #3152
- Argus: screen non-public content #4325
- Review: How Top CDN manage content authentication? #4438
- Standardise
Argusaccess policy signals in metadata #4332 - Argus: timed non-publicness #4331
Argus&Colossusprivate blacklists #4588- Infrastructure authentication #4414
- Argus: content access authentication scheme #4415
- Authentication between Argus and Colossus nodes #4420
Colossus / Argus refactor
- Make sure the apps don't exceed GraphQL request size limit #4624
- [Colossus] Add timeout enhancements to
superagentrequests based on the object sizes #4999 - Colossus state api re-factor #5059
- Minimise use and processing of large arrays #5051
- [Colossus] Trying to hash
File not foundresponse #5045 - colossus: support specifying multiple endpoints in operator metadata #4982
- New storage buckets analytics command #4444
- Update reverse proxy recommendations for Argus and Colossus to support HTTP/2 #3829 Question: HTTP/2 for Argus & Colossus #3808
Enable third-party actors to enrich (on-chain) content
- WIP: Probabilistic Intra-Infrastructure Payments #2757
- Delayed video publishing metaprotocol accomodation #4558
- Unlock third party post-processing for uploaders #2788
- Create Adaptive Streaming Specification #4720
- Store videos in different qualities, audio-only version #3806
- Separate audo & video tracks #4702
- QN Subtitles #3246
Enable viewers to publish playlists
- Requirements for playlists from Atlas perspective #3417
- QN Playlists #3245
- Idea: Paid Playlists (WIP) #2055
Update tooling