feat: use partman and s3 for archival partitions

[maschad]

As it stands

docker exec atoma-node-postgres-db-1 psql -U atoma -c "SELECT pg_size_pretty(pg_total_relation_size('stacks')) as size, pg_size_pretty(pg_relation_size('stacks')) as table_size, pg_size_pretty(pg_total_relation_size('stacks') - pg_relation_size('stacks')) as index_size, reltuples::bigint as estimated_rows FROM pg_class WHERE relname = 'stacks';"
  size   | table_size | index_size | estimated_rows 
---------+------------+------------+----------------
 7314 GB | 584 kB     | 7314 GB    |           1108
(1 row)

reveals

• Total Size: 7314 GB (approximately 7.3 TB)
• Table Size: 584 kB (actual data)
• Index Size: 7314 GB (indexes)
• Estimated Row Count: 1,108 rows

Which shows our indexes are excessively large, whilst this will partly be resolved by #609 to stem the growth, we will still need to clear any bloat that may be stored in TOAST.

This means will need a back up of records as well as going forward once fiat payments are integrated we will also need to store user stack related for the foreseeable future, which means it will need to be archived at intervals to avoid deletion.

This integrates https://github.com/pgpartman/pg_partman to manage archiving via s3

[Copilot]

Pull Request Overview

This PR introduces database maintenance enhancements by integrating pg_partman for partition management and S3 for archival and restoration of data. Key changes include:

• New scripts for database maintenance operations (including partition initialization, archival to S3, and restoration).
• Configuration updates in docker-compose and Ofelia for scheduling maintenance tasks.
• Prometheus monitoring integration for the archival process.

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
scripts/db-maintenance/run-maintenance.sh	Adds a script to execute maintenance tasks, including installing extensions and running SQL scripts.
scripts/db-maintenance/restore-from-s3.sql	Implements a PL/pgSQL function to restore data from S3.
scripts/db-maintenance/init-partioning.sql	Provides partitioning initialization using pg_partman; note the operator spacing issue.
scripts/db-maintenance/archive-partitions.sql	Creates functions and configurations for archiving partitions to S3.
scripts/db-maintenance/.env.example	Supplies environment variables required for S3 configuration.
prometheus.yml	Configures a new Prometheus job for monitoring the archival process.
ofelia.ini	Defines a scheduled job for monthly archive operations.
docker-compose.yaml	Updates service definitions to include maintenance tasks and ensure proper volume mappings.

[Copilot]

There is an extra space in the parameter assignment ('= >' instead of '=>'). This syntax error could prevent the partition maintenance function from executing correctly. Please update to use '=>' consistently.

Suggested change

	p_template_table = > NULL,
	p_retention = > '36 months',
	p_start_partition = > date_trunc('month', CURRENT_DATE - interval '3 years') :: text
	p_template_table => NULL,
	p_retention => '36 months',
	p_start_partition => date_trunc('month', CURRENT_DATE - interval '3 years') :: text

[Copilot]

Please remove the extra space in the parameter operator; replace ' = >' with '=>' as required by the function's syntax.

Suggested change

	p_template_table = > NULL,
	p_retention = > '36 months',
	p_start_partition = > date_trunc('month', CURRENT_DATE - interval '3 years') :: text
	p_template_table => NULL,
	p_retention => '36 months',
	p_start_partition => date_trunc('month', CURRENT_DATE - interval '3 years') :: text