diff --git a/README.md b/README.md index 274871c..a54ddb9 100644 --- a/README.md +++ b/README.md @@ -1,70 +1,476 @@ -# Python script to push and pull application config +# croudtech-bootstrap -This script is used to push and pull secrets, s3 config and ssm values for use in applications. +A Python CLI tool for managing application configuration across multiple environments using AWS services (S3, SSM Parameter Store, and Secrets Manager). -## Usage +## Overview -### Pushing Config +croudtech-bootstrap provides centralized configuration management for applications deployed across multiple environments. It supports: +- **Multi-environment configuration**: Manage separate configurations for production, staging, development, etc. +- **Shared configuration**: Define common values used by all applications in an environment +- **Secret management**: Securely store sensitive values in AWS Secrets Manager +- **Automatic Redis database allocation**: Multi-tenant Redis database management +- **Multiple output formats**: JSON, YAML, shell environment variables + +## Installation + +```bash +pip install croudtech-bootstrap ``` -Usage: croudtech-bootstrap put-config [OPTIONS] VALUES_PATH -Options: - --prefix TEXT The path prefix (Name prefix used when storing secrets and SSM values) - --region TEXT The AWS region (Defaults to the current region set using AWS_DEFAULT_REGION or AWS_REGION env vars) - --delete-first Delete the values in this path before pushing (useful for cleanup) This will remove any values with the current path prefix that aren't included in the files we're pushing. - --help Show this message and exit. +## Quick Start + +### 1. Initialize the bootstrap infrastructure + +```bash +croudtech-bootstrap init --environment-name production --region eu-west-2 +``` + +### 2. Create your configuration files + +``` +config/ +├── production/ +│ ├── common.yaml # Shared config for all apps +│ ├── common.secret.yaml # Shared secrets +│ ├── myapp.yaml # App-specific config +│ └── myapp.secret.yaml # App-specific secrets +└── staging/ + ├── common.yaml + ├── common.secret.yaml + ├── myapp.yaml + └── myapp.secret.yaml +``` + +### 3. Push configuration to AWS + +```bash +croudtech-bootstrap put-config ./config ``` -The put-config command requires the following file structure: +### 4. Retrieve configuration +```bash +croudtech-bootstrap get-config --environment-name production --app-name myapp --output-format json ``` -├── ENVIRONMENT_NAME_1 -│   ├── common.yaml -│   ├── common.secret.yaml -│   ├── AppConfig1.yaml -│   ├── AppConfig1.secret.yaml -│   ├── AppConfig2.yaml -│   └── AppConfig2.secret.yaml -├── ENVIRONMENT_NAME_2 -│   ├── common.yaml -│   ├── common.secret.yaml -│   ├── AppConfig1.yaml -│   ├── AppConfig1.secret.yaml -│   ├── AppConfig2.yaml -│   └── AppConfig2.secret.yaml + +## How Values are Stored and Retrieved + +### Storage Architecture + +Configuration is stored in three AWS services: + +| Storage | Purpose | Path Format | +| ----------------------- | ------------------------ | ---------------------------------------- | +| **S3** | Raw YAML file backup | `s3://{bucket}/{environment}/{app}.yaml` | +| **SSM Parameter Store** | Non-sensitive parameters | `/{environment}/{app}/{key}` | +| **Secrets Manager** | Sensitive values | `{environment}/{app}/{key}` | + +### Secret Tagging + +When secrets are pushed to AWS Secrets Manager, they are tagged with metadata to enable filtering during retrieval: + +| Tag Key | Tag Value | Example | +| ------------- | ---------------- | ------------ | +| `Environment` | Environment name | `production` | +| `App` | Application name | `myapp` | + +**Why tags matter:** When retrieving secrets, the system queries Secrets Manager using these tags as filters rather than listing all secrets. This ensures: + +- Secrets are correctly associated with their app/environment +- Efficient retrieval without scanning all secrets in the account +- Proper isolation between applications and environments + +### Retrieval Process + +When you retrieve configuration using `get-config`: + +1. **Fetch app-specific values** from S3 (`{environment}/{app}.yaml`) +2. **Fetch app-specific secrets** from Secrets Manager (tagged with `Environment` and `App`) +3. **If `--include-common` is enabled** (default), fetch common values and secrets +4. **Merge configurations**: App-specific values override common values +5. **Flatten nested structures**: Convert nested YAML to flat key-value pairs +6. **Process special values**: Auto-allocate Redis databases if configured + +### Value Flattening + +Nested YAML structures are automatically flattened using underscore separators: + +```yaml +# Input (myapp.yaml) +database: + host: localhost + port: 5432 + credentials: + username: admin + +# Output (flattened) +database_host: localhost +database_port: 5432 +database_credentials_username: admin ``` -Running `python -m croudtech-bootstrap put-config CONFIG_FILES_PATH` will create config for AppConfig1 and AppConfig2 in both defined environments. +### Configuration Merging + +Common configuration is merged with app-specific configuration, with app values taking precedence: -common.yaml and common.secret.yaml files contain shared config that will be used for all applications. +```yaml +# common.yaml +LOG_LEVEL: INFO +DATABASE_HOST: shared-db.example.com -### Pulling config +# myapp.yaml +LOG_LEVEL: DEBUG # Overrides common +API_KEY: abc123 +# Result +LOG_LEVEL: DEBUG # From myapp.yaml (overrides common) +DATABASE_HOST: shared-db.example.com # From common.yaml +API_KEY: abc123 # From myapp.yaml ``` -Usage: croudtech-bootstrap get-config [OPTIONS] + +## Configuration File Structure + +### Directory Layout + +``` +VALUES_PATH/ +├── ENVIRONMENT_1/ +│ ├── common.yaml # Shared config (all apps inherit) +│ ├── common.secret.yaml # Shared secrets (all apps inherit) +│ ├── app1.yaml # App1 configuration +│ ├── app1.secret.yaml # App1 secrets +│ ├── app2.yaml # App2 configuration +│ └── app2.secret.yaml # App2 secrets +├── ENVIRONMENT_2/ +│ ├── common.yaml +│ ├── common.secret.yaml +│ ├── app1.yaml +│ └── app1.secret.yaml +``` + +### File Naming Conventions + +| File Pattern | Purpose | Storage Location | +| -------------------- | --------------------------- | ------------------------ | +| `{app}.yaml` | Non-sensitive configuration | S3 + SSM Parameter Store | +| `{app}.secret.yaml` | Sensitive configuration | Secrets Manager | +| `common.yaml` | Shared non-sensitive config | Merged with all apps | +| `common.secret.yaml` | Shared secrets | Merged with all apps | + +### Example Configuration Files + +**production/common.yaml** - Shared environment settings: + +```yaml +AWS_REGION: eu-west-2 +LOG_LEVEL: INFO +REDIS_HOST: redis.example.com +REDIS_PORT: 6379 +REDIS_DB: auto # Auto-allocate database number +``` + +**production/myapp.yaml** - Application-specific settings: + +```yaml +APP_NAME: myapp +API_VERSION: v2 +database: + host: db.example.com + port: 5432 + name: myapp_production +``` + +**production/myapp.secret.yaml** - Sensitive values: + +```yaml +database: + password: supersecretpassword +API_SECRET_KEY: abc123xyz +``` + +## CLI Commands + +### Global Options + +```bash +croudtech-bootstrap [OPTIONS] COMMAND Options: - --environment-name TEXT The environment name [required] - --app-name TEXT The app name [required] - --prefix TEXT The path prefix - --region TEXT The AWS region - --include-common / --ignore-common - Include shared variables + --endpoint-url TEXT AWS API endpoint URL (for testing with LocalStack) + --put-metrics Enable CloudWatch metrics tracking (default: True) + --bucket-name TEXT S3 bucket name (default: app-bootstrap-{AWS_ACCOUNT_ID}) +``` + +### init + +Initialize bootstrap infrastructure by creating the S3 bucket. + +```bash +croudtech-bootstrap init --environment-name ENVIRONMENT --region REGION +``` + +### put-config + +Push local configuration files to AWS. + +```bash +croudtech-bootstrap put-config [OPTIONS] VALUES_PATH + +Options: + --prefix TEXT SSM parameter path prefix (default: /appconfig) + --region TEXT AWS region (default: eu-west-2) + --delete-first Remove orphaned parameters/secrets before pushing +``` + +**Example:** + +```bash +croudtech-bootstrap put-config ./config --region eu-west-2 --delete-first +``` + +### get-config + +Retrieve configuration for a specific application and environment. + +```bash +croudtech-bootstrap get-config [OPTIONS] + +Options: + --environment-name TEXT Environment name (required) + --app-name TEXT Application name (required) + --prefix TEXT SSM path prefix (default: /appconfig) + --region TEXT AWS region (default: eu-west-2) + --include-common/--ignore-common + Include shared config (default: include) --output-format [json|yaml|environment|environment-export] - --parse-redis-param / --ignore-redis-param - Parse redis host and allocate a redis - database number. Requires network access to the redis instance - --help Show this message and exit. + Output format (default: json) + --parse-redis-param/--ignore-redis-param + Auto-allocate Redis DB (default: parse) ``` -Using the put-config example above we can pull the config as follows +**Output Format Examples:** + +```bash +# JSON output (default) +croudtech-bootstrap get-config --environment-name production --app-name myapp + +# YAML output +croudtech-bootstrap get-config --environment-name production --app-name myapp --output-format yaml +# Shell variables (for sourcing) +croudtech-bootstrap get-config --environment-name production --app-name myapp --output-format environment +# Output: DATABASE_HOST="db.example.com" + +# Shell exports (for subshells) +croudtech-bootstrap get-config --environment-name production --app-name myapp --output-format environment-export +# Output: export DATABASE_HOST="db.example.com" ``` -croudtech-bootstrap get-config --environment-name ENVIRONMENT_NAME_1 --app-name AppConfig1 --output-format environment + +### list-apps + +List all applications stored in S3 across all environments. + +```bash +croudtech-bootstrap list-apps --region eu-west-2 ``` -## Installation +### cleanup-secrets + +Remove orphaned secrets from AWS Secrets Manager. + +```bash +croudtech-bootstrap cleanup-secrets VALUES_PATH --region eu-west-2 +``` + +### manage-redis + +Redis database allocation management commands. + +```bash +# Show allocated database for an app +croudtech-bootstrap manage-redis show-db \ + --environment-name production \ + --app-name myapp + +# Show all database allocations +croudtech-bootstrap manage-redis show-dbs \ + --redis-host redis.example.com \ + --redis-port 6379 + +# Manually allocate a database +croudtech-bootstrap manage-redis allocate-db \ + --redis-host redis.example.com \ + --redis-port 6379 \ + --environment-name production \ + --app-name myapp + +# Remove a database allocation +croudtech-bootstrap manage-redis deallocate-db \ + --redis-host redis.example.com \ + --redis-port 6379 \ + --environment-name production \ + --app-name myapp +``` + +## Redis Database Auto-Allocation + +When multiple applications share a single Redis instance, croudtech-bootstrap can automatically allocate unique database numbers to each application. + +### How It Works + +1. **Database 15** is reserved for storing allocation metadata +2. **Databases 0-14** are available for applications +3. Allocations are tracked using keys in format `{environment}_{app_name}` +4. When `REDIS_DB=auto` is configured, a database is automatically allocated + +### Configuration + +In your `common.yaml` or `{app}.yaml`: + +```yaml +REDIS_HOST: redis.example.com +REDIS_PORT: 6379 +REDIS_DB: auto # Triggers auto-allocation +``` + +### Output + +When configuration is retrieved with Redis auto-allocation: + +```json +{ + "REDIS_HOST": "redis.example.com", + "REDIS_PORT": 6379, + "REDIS_DB": 3, + "REDIS_URL": "redis://redis.example.com:6379/3" +} +``` + +## Programmatic Usage + +You can use croudtech-bootstrap programmatically in your Python applications: + +```python +from croudtech_bootstrap_app.bootstrap import BootstrapParameters + +# Initialize the parameters retriever +params = BootstrapParameters( + environment_name="production", + app_name="myapp", + bucket_name="app-bootstrap-123456789", + region="eu-west-2", + include_common=True, + parse_redis=True, +) + +# Get flattened parameters +config = params.get_params() +print(config["DATABASE_HOST"]) +print(config["REDIS_URL"]) + +# Get parameters preserving nested structure +raw_config = params.get_raw_params() +print(raw_config["database"]["host"]) + +# Export as environment variables +env_string = params.params_to_env(export=True) +# Returns: export DATABASE_HOST="db.example.com"\nexport API_KEY="..." +``` + +## AWS Permissions + +The following IAM permissions are required: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:CreateBucket", + "s3:PutObject", + "s3:GetObject", + "s3:ListBucket" + ], + "Resource": [ + "arn:aws:s3:::app-bootstrap-*", + "arn:aws:s3:::app-bootstrap-*/*" + ] + }, + { + "Effect": "Allow", + "Action": [ + "ssm:PutParameter", + "ssm:DeleteParameter", + "ssm:DescribeParameters", + "ssm:GetParameter", + "ssm:AddTagsToResource" + ], + "Resource": "*" + }, + { + "Effect": "Allow", + "Action": [ + "secretsmanager:CreateSecret", + "secretsmanager:UpdateSecret", + "secretsmanager:DeleteSecret", + "secretsmanager:GetSecretValue", + "secretsmanager:ListSecrets" + ], + "Resource": "*" + }, + { + "Effect": "Allow", + "Action": ["cloudwatch:PutMetricData"], + "Resource": "*" + } + ] +} +``` + +## Environment Variables + +| Variable | Description | Default | +| ------------------------------------ | -------------------------------------------- | ----------- | +| `AWS_DEFAULT_REGION` or `AWS_REGION` | AWS region | `eu-west-2` | +| `AWS_ENDPOINT_URL` | Custom AWS endpoint (for LocalStack testing) | None | +| `LOG_LEVEL` | Logging verbosity | `INFO` | + +## Testing with LocalStack + +For local development and testing: + +```bash +# Start LocalStack +docker-compose up -d localstack + +# Use croudtech-bootstrap with LocalStack +croudtech-bootstrap --endpoint-url http://localhost:4566 init --environment-name test +croudtech-bootstrap --endpoint-url http://localhost:4566 put-config ./config +``` + +## Troubleshooting + +### Common Issues + +**"Parameter value is too large to store"** + +- SSM Parameter Store has a 4096 byte limit for standard parameters +- Consider using Secrets Manager for larger values or splitting configuration + +**"Couldn't allocate Redis Database"** + +- Ensure REDIS_HOST is correctly configured +- Check that the Redis instance is accessible +- Verify databases 0-14 aren't all allocated + +**"Bucket already exists but is not owned by you"** + +- The default bucket name uses your AWS account ID +- Specify a custom bucket name with `--bucket-name` -`pip install croudtech-bootstrap` +## License +MIT License diff --git a/croudtech_bootstrap_app/bootstrap.py b/croudtech_bootstrap_app/bootstrap.py index 486cfbd..b842f9c 100644 --- a/croudtech_bootstrap_app/bootstrap.py +++ b/croudtech_bootstrap_app/bootstrap.py @@ -1,7 +1,6 @@ from __future__ import annotations import sys - from typing import TYPE_CHECKING, Any import botocore.exceptions @@ -9,14 +8,18 @@ if TYPE_CHECKING: from mypy_boto3_s3 import S3Client from mypy_boto3_secretsmanager import SecretsManagerClient + from mypy_boto3_ssm import SSMClient else: S3Client = object + SSMClient = object import json import logging import os +import re import shutil import tempfile +import time import typing from collections.abc import MutableMapping @@ -24,8 +27,6 @@ import botocore import click import yaml -import time -import re from croudtech_bootstrap_app.logging import init as initLogs @@ -38,13 +39,69 @@ class Utils: + """Utility functions for the bootstrap application.""" + @staticmethod def chunk_list(data, chunk_size): + """ + Split a list into chunks of a specified size. + + Args: + data: The list to split. + chunk_size: The maximum size of each chunk. + + Yields: + List chunks of the specified size. + """ for i in range(0, len(data), chunk_size): yield data[i : i + chunk_size] class BootstrapParameters: + """ + High-level interface for retrieving application configuration parameters. + + This class provides a simplified API for fetching configuration values and secrets + for a specific application and environment combination. It handles merging common + configuration with app-specific configuration and supports automatic Redis database + allocation. + + Value Storage and Retrieval: + Configuration is stored in AWS using a hierarchical structure: + - S3: Raw YAML files stored at s3://{bucket}/{environment}/{app}.yaml + - Secrets Manager: Sensitive values stored at {environment}/{app}/{key} + - SSM Parameter Store: Non-sensitive values (used during push) + + When retrieving parameters: + 1. App-specific values are fetched from S3 + 2. If include_common=True, common values are also fetched + 3. Values are merged (app-specific overrides common) + 4. Nested YAML is flattened using underscore separators + e.g., {db: {host: "localhost"}} becomes {"db_host": "localhost"} + 5. If parse_redis=True and REDIS_DB=auto, a Redis DB is allocated + + Example: + >>> params = BootstrapParameters( + ... environment_name="production", + ... app_name="myapp", + ... bucket_name="app-bootstrap-123456789" + ... ) + >>> config = params.get_params() + >>> print(config["DATABASE_URL"]) + + Args: + environment_name: The target environment (e.g., "production", "staging"). + app_name: The application name (must match the YAML filename without extension). + bucket_name: S3 bucket name where configuration is stored. + click: Click module for CLI output (default: click module). + prefix: SSM parameter path prefix (default: "/appconfig"). + region: AWS region (default: "eu-west-2"). + include_common: Whether to merge common.yaml values (default: True). + use_sns: Whether to use SNS notifications (default: True). + endpoint_url: Custom AWS endpoint URL for testing (default: from env). + parse_redis: Whether to auto-allocate Redis databases (default: True). + """ + def __init__( self, environment_name, @@ -73,6 +130,7 @@ def __init__( @property def bootstrap_manager(self) -> BootstrapManager: + """Lazily instantiated BootstrapManager for AWS operations.""" if not hasattr(self, "_bootstrap_manager"): self._bootstrap_manager = BootstrapManager( prefix=self.prefix, @@ -86,6 +144,7 @@ def bootstrap_manager(self) -> BootstrapManager: @property def environment(self) -> BootstrapEnvironment: + """Lazily instantiated BootstrapEnvironment for the target environment.""" if not hasattr(self, "_environment"): self._environment = BootstrapEnvironment( name=self.environment_name, path=None, manager=self.bootstrap_manager @@ -94,6 +153,7 @@ def environment(self) -> BootstrapEnvironment: @property def app(self) -> BootstrapApp: + """Lazily instantiated BootstrapApp for the target application.""" if not hasattr(self, "_app"): self._app = BootstrapApp( name=self.app_name, path=None, environment=self.environment @@ -102,6 +162,7 @@ def app(self) -> BootstrapApp: @property def common_app(self) -> BootstrapApp: + """Lazily instantiated BootstrapApp for retrieving common configuration.""" if not hasattr(self, "_common_app"): self._common_app = BootstrapApp( name="common", path=None, environment=self.environment @@ -109,11 +170,30 @@ def common_app(self) -> BootstrapApp: return self._common_app def get_redis_db(self): + """ + Get the allocated Redis database number for this application. + + Returns: + tuple: (redis_db, redis_host, redis_port) or (None, None, None) if not configured. + """ parameters = self.get_params() redis_db, redis_host, redis_port = self.find_redis_config(parameters) return redis_db, redis_host, redis_port def find_redis_config(self, parameters, allocate=False): + """ + Extract Redis configuration from parameters and optionally allocate a database. + + If REDIS_DB is not set or is set to "auto", this method will look up or allocate + a Redis database number from the shared Redis allocation system. + + Args: + parameters: Dictionary of configuration parameters. + allocate: If True, allocate a new database if one doesn't exist. + + Returns: + tuple: (redis_db, redis_host, redis_port) or (None, None, None) if not configured. + """ if "REDIS_DB" not in parameters or parameters["REDIS_DB"] == "auto": redis_host = ( parameters["REDIS_HOST"] if "REDIS_HOST" in parameters else False @@ -122,7 +202,7 @@ def find_redis_config(self, parameters, allocate=False): parameters["REDIS_PORT"] if "REDIS_PORT" in parameters else 6379 ) - if redis_host != None: + if redis_host is not None: redis_config_instance = RedisConfig( redis_host=redis_host, redis_port=redis_port, @@ -135,6 +215,21 @@ def find_redis_config(self, parameters, allocate=False): return None, None, None def parse_params(self, parameters): + """ + Post-process parameters to handle special values like Redis auto-allocation. + + If parse_redis is enabled and REDIS_HOST is configured, this will allocate + a Redis database and generate a REDIS_URL. + + Args: + parameters: Dictionary of configuration parameters. + + Returns: + dict: Parameters with REDIS_DB and REDIS_URL populated if applicable. + + Raises: + Exception: If Redis auto-allocation is required but fails. + """ if self.parse_redis: redis_db, redis_host, redis_port = self.find_redis_config( parameters, allocate=True @@ -151,6 +246,15 @@ def parse_params(self, parameters): return parameters def get_params(self): + """ + Retrieve flattened configuration parameters for the application. + + Fetches configuration from S3, merges common config if enabled, + flattens nested structures, and processes special values. + + Returns: + dict: Flattened key-value pairs of configuration parameters. + """ app_params = self.app.get_remote_params() if self.include_common: @@ -159,6 +263,15 @@ def get_params(self): return self.parse_params(app_params) def get_raw_params(self): + """ + Retrieve configuration parameters preserving nested structure. + + Similar to get_params() but keeps the original YAML nesting structure + instead of flattening to underscore-separated keys. + + Returns: + dict: Configuration parameters with original nested structure. + """ app_params = self.app.get_remote_params(flatten=False) if self.include_common: @@ -167,6 +280,24 @@ def get_raw_params(self): return self.parse_params(app_params) def params_to_env(self, export=False): + """ + Convert configuration parameters to shell environment variable format. + + Generates shell-compatible variable assignments that can be sourced + or exported. Special characters in values are escaped. + + Args: + export: If True, prefix each line with "export " for subshell use. + + Returns: + str: Newline-separated environment variable assignments. + + Example: + >>> params.params_to_env() + 'DATABASE_URL="postgres://..."\\nAPI_KEY="secret"' + >>> params.params_to_env(export=True) + 'export DATABASE_URL="postgres://..."\\nexport API_KEY="secret"' + """ strings = [] for parameter, value in self.get_params().items(): os.environ[parameter] = str(value) @@ -185,6 +316,42 @@ def params_to_env(self, export=False): class BootstrapApp: + """ + Represents a single application's configuration within an environment. + + This class manages both local (YAML files) and remote (AWS) configuration + for an application. It handles reading local configuration files, fetching + remote values from S3 and Secrets Manager, and pushing configuration to AWS. + + Configuration File Naming Convention: + - {app_name}.yaml: Non-sensitive configuration values + - {app_name}.secret.yaml: Sensitive values (stored in Secrets Manager) + + Storage Locations: + - S3: s3://{bucket}/{environment}/{app_name}.yaml + - Secrets Manager: {environment}/{app_name}/{secret_key} + - SSM Parameter Store: /{environment}/{app_name}/{parameter_key} + + Value Flattening: + Nested YAML structures are flattened using underscore separators: + ```yaml + database: + host: localhost + port: 5432 + ``` + Becomes: {"database_host": "localhost", "database_port": "5432"} + + Args: + name: The application name (matches filename without .yaml extension). + path: Local filesystem path to the app's YAML configuration file. + environment: Parent BootstrapEnvironment instance. + + Example: + >>> app = BootstrapApp("myapp", "/config/prod/myapp.yaml", environment) + >>> local_config = app.local_values + >>> remote_config = app.remote_values + """ + environment: BootstrapEnvironment def __init__(self, name, path, environment: BootstrapEnvironment): @@ -194,21 +361,26 @@ def __init__(self, name, path, environment: BootstrapEnvironment): @property def s3_client(self) -> S3Client: + """AWS S3 client from parent manager.""" return self.environment.manager.s3_client @property def ssm_client(self) -> SSMClient: + """AWS SSM client from parent manager.""" return self.environment.manager.ssm_client @property def secrets_client(self) -> SecretsManagerClient: + """AWS Secrets Manager client from parent manager.""" return self.environment.manager.secrets_client @property def secret_path(self): + """Local filesystem path to the app's secret YAML file.""" return os.path.join(self.environment.path, f"{self.name}.secret.yaml") def upload_to_s3(self): + """Upload the local YAML configuration file to S3.""" source = self.path bucket = self.environment.manager.bucket_name dest = os.path.join("", self.environment.name, os.path.basename(self.path)) @@ -225,9 +397,19 @@ def upload_to_s3(self): @property def s3_key(self): + """S3 object key for this app's configuration: {environment}/{app}.yaml""" return os.path.join("", self.environment.name, ".".join([self.name, "yaml"])) def fetch_from_s3(self, raw=False) -> typing.Dict[str, Any]: + """ + Fetch configuration values from S3. + + Args: + raw: If True, return values without JSON parsing. + + Returns: + dict: Configuration values from the S3-stored YAML file. + """ if not hasattr(self, "_s3_data"): response = self.s3_client.get_object( Bucket=self.environment.manager.bucket_name, Key=self.s3_key @@ -241,6 +423,15 @@ def fetch_from_s3(self, raw=False) -> typing.Dict[str, Any]: return self._s3_data def parse_value(self, value): + """ + Parse a configuration value, detecting and handling JSON strings. + + Args: + value: The raw value to parse. + + Returns: + str: The parsed value as a string, with JSON properly serialized. + """ try: parsed_value = json.dumps(json.loads(value)) except json.decoder.JSONDecodeError: @@ -250,6 +441,12 @@ def parse_value(self, value): return str(parsed_value).strip() def cleanup_ssm_parameters(self): + """ + Remove SSM parameters that exist remotely but not in local configuration. + + Compares local YAML keys with remote SSM parameters and deletes + any remote parameters that no longer have corresponding local definitions. + """ local_value_keys = set(self.convert_flatten(self.local_values).keys() or []) self.raw = True remote_value_keys = set(self.remote_values or []) @@ -260,19 +457,25 @@ def cleanup_ssm_parameters(self): for parameter in orphaned_ssm_parameters: parameter_id = self.get_parameter_id(parameter) try: - self.ssm_client.delete_parameter( - Name=self.get_parameter_id(parameter) - ) + self.ssm_client.delete_parameter(Name=self.get_parameter_id(parameter)) logger.info(f"Deleted orphaned ssm parameter {parameter}") except Exception: logger.info(f"Parameter: {parameter_id} could not be deleted") def cleanup_secrets(self): + """ + Remove secrets from AWS Secrets Manager that no longer exist locally. + + Compares local secret YAML keys with remote secrets and permanently + deletes any remote secrets without corresponding local definitions. + """ local_secret_keys = self.convert_flatten(self.local_secrets).keys() remote_secret_keys = self.remote_secret_records.keys() orphaned_secrets = [ - item for item in remote_secret_keys if re.sub(r"(-[a-zA-Z]{6})$", "", item) not in local_secret_keys + item + for item in remote_secret_keys + if re.sub(r"(-[a-zA-Z]{6})$", "", item) not in local_secret_keys ] for secret in orphaned_secrets: @@ -284,6 +487,12 @@ def cleanup_secrets(self): @property def local_secrets(self) -> typing.Dict[str, Any]: + """ + Local secrets loaded from {app_name}.secret.yaml file. + + Returns: + dict: Secret key-value pairs, or empty dict if file doesn't exist. + """ if not hasattr(self, "_secrets"): self._secrets = {} if os.path.exists(self.secret_path): @@ -296,6 +505,12 @@ def local_secrets(self) -> typing.Dict[str, Any]: @property def local_values(self) -> typing.Dict[str, Any]: + """ + Local configuration values loaded from {app_name}.yaml file. + + Returns: + dict: Configuration key-value pairs, or empty dict if file doesn't exist. + """ if not hasattr(self, "_values"): self._values = {} if os.path.exists(self.path): @@ -308,6 +523,12 @@ def local_values(self) -> typing.Dict[str, Any]: @property def remote_secrets(self) -> typing.Dict[str, Any]: + """ + Remote secrets fetched from AWS Secrets Manager. + + Returns: + dict: Secret key-value pairs from Secrets Manager. + """ if not hasattr(self, "_remote_secrets"): self._remote_secrets = self.get_remote_secrets() @@ -315,6 +536,12 @@ def remote_secrets(self) -> typing.Dict[str, Any]: @property def remote_secret_records(self) -> typing.Dict[str, Any]: + """ + Remote secret metadata records from AWS Secrets Manager. + + Returns: + dict: Secret metadata keyed by secret name. + """ if not hasattr(self, "_remote_secrets"): self._remote_secrets = self.get_remote_secret_records() @@ -322,6 +549,12 @@ def remote_secret_records(self) -> typing.Dict[str, Any]: @property def remote_ssm_parameters(self) -> typing.Dict[str, Any]: + """ + Remote parameters fetched from AWS SSM Parameter Store. + + Returns: + dict: Parameter metadata keyed by parameter name. + """ if not hasattr(self, "_remote_parameters"): self._remote_parameters = self.get_remote_ssm_parameters() @@ -329,23 +562,42 @@ def remote_ssm_parameters(self) -> typing.Dict[str, Any]: @property def remote_values(self) -> typing.Dict[str, Any]: + """ + Remote configuration values fetched from S3. + + Returns: + dict: Configuration key-value pairs from S3, or empty dict on error. + """ if not hasattr(self, "_remote_values"): try: self._remote_values = self.fetch_from_s3(self.raw) except botocore.exceptions.ClientError as err: - self.environment.manager.click.secho( - err - ) + self.environment.manager.click.secho(err) self._remote_values = {} return self._remote_values def get_local_params(self): + """ + Get merged local values and secrets, flattened. + + Returns: + dict: Combined and flattened local configuration. + """ app_values = self.convert_flatten(self.local_values) app_secrets = self.convert_flatten(self.local_secrets) return {**app_values, **app_secrets} def get_remote_params(self, flatten=True): + """ + Get merged remote values and secrets from AWS. + + Args: + flatten: If True, flatten nested structures using underscore separators. + + Returns: + dict: Combined remote configuration (values + secrets). + """ if flatten: self.raw = False app_values = self.convert_flatten(self.remote_values) @@ -357,18 +609,34 @@ def get_remote_params(self, flatten=True): return {**app_values, **app_secrets} def get_flattened_parameters(self) -> typing.Dict[str, Any]: + """Get local values with nested structures flattened.""" return self.convert_flatten(self.local_values) def get_flattened_secrets(self) -> typing.Dict[str, Any]: + """Get local secrets with nested structures flattened.""" return self.convert_flatten(self.local_secrets) def get_parameter_id(self, parameter): + """Generate SSM parameter path: /{environment}/{app}/{parameter}""" return f"/{self.get_secret_id(parameter)}" def get_secret_id(self, secret): + """Generate secret path: {environment}/{app}/{secret}""" return os.path.join("", self.environment.name, self.name, secret) - def put_parameter(self, parameter_id, parameter_value, tags=None, type="String", overwrite=True): + def put_parameter( + self, parameter_id, parameter_value, tags=None, type="String", overwrite=True + ): + """ + Create or update an SSM parameter. + + Args: + parameter_id: Full parameter path (e.g., /{env}/{app}/{key}). + parameter_value: The value to store. + tags: Optional list of tag dicts with Key and Value. + type: SSM parameter type (default: "String"). + overwrite: Whether to overwrite existing values (default: True). + """ print(f"Creating Parameter {parameter_id}") self.ssm_client.put_parameter( Name=parameter_id, @@ -378,12 +646,22 @@ def put_parameter(self, parameter_id, parameter_value, tags=None, type="String", ) if tags: self.ssm_client.add_tags_to_resource( - ResourceType="Parameter", - ResourceId=parameter_id, - Tags=tags + ResourceType="Parameter", ResourceId=parameter_id, Tags=tags ) def create_secret(self, Name, SecretString, Tags, ForceOverwriteReplicaSecret): + """ + Create or update a secret in AWS Secrets Manager. + + Automatically handles the case where the secret already exists + by updating it instead. + + Args: + Name: Secret name/path (e.g., {env}/{app}/{key}). + SecretString: The secret value to store. + Tags: List of tag dicts (ignored, tags are auto-generated). + ForceOverwriteReplicaSecret: Whether to force overwrite replicas. + """ print(f"Creating Secret {Name}") try: self.secrets_client.create_secret( @@ -401,7 +679,37 @@ def create_secret(self, Name, SecretString, Tags, ForceOverwriteReplicaSecret): SecretString=SecretString, ) - def backoff_with_custom_exception(self, func, exception, message_prefix="", max_attempts=5, base_delay=1, max_delay=10, factor=2, *args, **kwargs): + def backoff_with_custom_exception( + self, + func, + exception, + message_prefix="", + max_attempts=5, + base_delay=1, + max_delay=10, + factor=2, + *args, + **kwargs, + ): + """ + Execute a function with exponential backoff retry on specified exceptions. + + Args: + func: The function to execute. + exception: The exception type to catch and retry on. + message_prefix: Prefix for log messages. + max_attempts: Maximum number of retry attempts (default: 5). + base_delay: Initial delay in seconds (default: 1). + max_delay: Maximum delay in seconds (default: 10). + factor: Exponential backoff factor (default: 2). + *args, **kwargs: Arguments to pass to the function. + + Returns: + The result of the function if successful. + + Raises: + The caught exception if all attempts fail. + """ attempts = 0 delay = base_delay @@ -421,9 +729,18 @@ def backoff_with_custom_exception(self, func, exception, message_prefix="", max_ time.sleep(delay) def push_parameters(self): + """ + Push all local parameters to AWS SSM Parameter Store. + + Iterates through flattened local values and creates/updates + corresponding SSM parameters. Skips values larger than 4096 bytes + (SSM limit) or empty values. + """ for parameter, value in self.get_flattened_parameters().items(): parameter_value = str(value) - if (value_size := sys.getsizeof(parameter_value)) > 4096 or not parameter_value: + if ( + value_size := sys.getsizeof(parameter_value) + ) > 4096 or not parameter_value: self.environment.manager.click.secho( f"Parameter: {parameter} value is too large to store ({value_size})" ) @@ -446,6 +763,13 @@ def push_parameters(self): ) def push_secrets(self): + """ + Push all local secrets to AWS Secrets Manager. + + Iterates through flattened local secrets and creates/updates + corresponding secrets in Secrets Manager. Empty values are stored + as "__EMPTY__" placeholder. + """ for secret, value in self.get_flattened_secrets().items(): sec_val = str(value) if len(sec_val) == 0: @@ -472,11 +796,18 @@ def push_secrets(self): except Exception as err: logger.error(f"Failed to push secret {secret_id}") raise err - self.environment.manager.click.secho( - f"Pushed {secret_id}" - ) + self.environment.manager.click.secho(f"Pushed {secret_id}") def fetch_secret_value(self, secret): + """ + Retrieve a secret's value from AWS Secrets Manager. + + Args: + secret: Secret metadata dict containing 'ARN' key. + + Returns: + str: The secret value, or empty string if stored as "__EMPTY__". + """ response = self.secrets_client.get_secret_value(SecretId=secret["ARN"]) sec_val = response["SecretString"] if sec_val == "__EMPTY__": @@ -485,16 +816,18 @@ def fetch_secret_value(self, secret): @property def remote_ssm_parameter_filters(self): + """SSM DescribeParameters filter for this app's parameters.""" return [ { "Key": "Name", "Option": "Contains", - "Values": [f"/{self.environment.name}/{self.name}"] + "Values": [f"/{self.environment.name}/{self.name}"], } ] @property def remote_secret_filters(self): + """Secrets Manager ListSecrets filter for this app's secrets.""" return [ {"Key": "tag-key", "Values": ["Environment"]}, {"Key": "tag-value", "Values": [self.environment.name]}, @@ -503,7 +836,13 @@ def remote_secret_filters(self): ] def get_remote_ssm_parameters(self): - paginator = self.ssm_client.get_paginator('describe_parameters') + """ + Fetch all SSM parameters for this app from AWS. + + Returns: + dict: Parameter metadata keyed by parameter name (without path prefix). + """ + paginator = self.ssm_client.get_paginator("describe_parameters") parameters = {} filters = self.remote_ssm_parameter_filters response = paginator.paginate( @@ -517,6 +856,12 @@ def get_remote_ssm_parameters(self): return parameters def get_remote_secrets(self) -> typing.Dict[str, str]: + """ + Fetch all secrets for this app from AWS Secrets Manager. + + Returns: + dict: Secret values keyed by secret name (without path prefix). + """ paginator = self.secrets_client.get_paginator("list_secrets") secrets = {} response = paginator.paginate( @@ -530,6 +875,12 @@ def get_remote_secrets(self) -> typing.Dict[str, str]: return secrets def get_remote_secret_records(self): + """ + Fetch secret metadata records from AWS Secrets Manager. + + Returns: + dict: Secret metadata (ARN, name, etc.) keyed by secret name. + """ paginator = self.secrets_client.get_paginator("list_secrets") secrets = {} response = paginator.paginate( @@ -543,6 +894,20 @@ def get_remote_secret_records(self): return secrets def convert_flatten(self, d, parent_key="", sep="_"): + """ + Flatten a nested dictionary into a single-level dictionary. + + Nested keys are joined with the separator to form flat keys. + Example: {"db": {"host": "localhost"}} -> {"db_host": "localhost"} + + Args: + d: The dictionary to flatten. + parent_key: Prefix for keys (used in recursion). + sep: Separator between nested key levels (default: "_"). + + Returns: + dict: Flattened key-value pairs. + """ items = [] if isinstance(d, dict): for k, v in d.items(): @@ -556,6 +921,31 @@ def convert_flatten(self, d, parent_key="", sep="_"): class BootstrapEnvironment: + """ + Represents a deployment environment containing multiple applications. + + An environment is a directory containing YAML configuration files for + multiple applications. Each environment typically represents a deployment + stage (e.g., "production", "staging", "development"). + + The environment scans its directory for YAML files and creates BootstrapApp + instances for each discovered application. + + Directory Structure: + {environment_name}/ + ├── common.yaml # Shared config for all apps + ├── common.secret.yaml # Shared secrets for all apps + ├── app1.yaml # App1 configuration + ├── app1.secret.yaml # App1 secrets + ├── app2.yaml # App2 configuration + └── app2.secret.yaml # App2 secrets + + Args: + name: Environment name (e.g., "production", "staging"). + path: Local filesystem path to the environment directory. + manager: Parent BootstrapManager instance. + """ + manager: BootstrapManager def __init__(self, name, path, manager: BootstrapManager): @@ -567,6 +957,7 @@ def __init__(self, name, path, manager: BootstrapManager): @property def temp_dir(self): + """Temporary directory for this environment's file processing.""" if not hasattr(self, "_temp_dir"): self._temp_dir = os.path.join(self.manager.temp_dir, self.name) os.mkdir(self._temp_dir) @@ -574,6 +965,16 @@ def temp_dir(self): @property def apps(self) -> typing.Dict[str, BootstrapApp]: + """ + Dictionary of BootstrapApp instances for all apps in this environment. + + Scans the environment directory for .yaml/.yml files and creates + BootstrapApp instances. Secret files (.secret.yaml) are excluded + from this list as they're accessed via their parent app. + + Returns: + dict: App name -> BootstrapApp mapping. + """ if not hasattr(self, "_apps"): self._apps = {} for file in os.listdir(self.path): @@ -592,11 +993,49 @@ def apps(self) -> typing.Dict[str, BootstrapApp]: return self._apps def copy_to_temp(self): + """Copy all app configuration files to the temporary directory.""" for _app_name, app in self.apps.items(): shutil.copy(app.path, self.temp_dir) class BootstrapManager: + """ + Central orchestrator for bootstrap configuration operations. + + The BootstrapManager coordinates all bootstrap activities including: + - AWS client management (S3, SSM, Secrets Manager) + - Environment and application discovery + - Configuration push/pull operations + - Cleanup of orphaned resources + + This is the main entry point for programmatic use of the bootstrap system. + It manages lazy-loaded AWS clients and provides access to all environments + and applications within a configuration directory. + + Architecture: + BootstrapManager + └── BootstrapEnvironment (one per environment directory) + └── BootstrapApp (one per YAML file) + + Args: + prefix: SSM parameter path prefix (e.g., "/appconfig"). + region: AWS region for all operations. + click: Click module for CLI output. + values_path: Local directory containing environment subdirectories. + bucket_name: S3 bucket for storing configuration files. + endpoint_url: Custom AWS endpoint URL (for testing with LocalStack). + + Example: + >>> manager = BootstrapManager( + ... prefix="/appconfig", + ... region="eu-west-2", + ... click=click, + ... values_path="./config", + ... bucket_name="app-bootstrap-123456789" + ... ) + >>> manager.put_config(delete_first=True) + """ + _environments: dict[str, BootstrapEnvironment] def __init__( @@ -617,6 +1056,7 @@ def __init__( @property def s3_client(self) -> S3Client: + """Lazily instantiated AWS S3 client.""" if not hasattr(self, "_s3_client"): self._s3_client = boto3.client( "s3", region_name=self.region, endpoint_url=self.endpoint_url @@ -625,16 +1065,16 @@ def s3_client(self) -> S3Client: @property def ssm_client(self): - if not hasattr(self, '_ssm_client'): + """Lazily instantiated AWS SSM client.""" + if not hasattr(self, "_ssm_client"): self._ssm_client = boto3.client( - "ssm", - region_name=self.region, - endpoint_url=self.endpoint_url + "ssm", region_name=self.region, endpoint_url=self.endpoint_url ) return self._ssm_client @property def secrets_client(self) -> SecretsManagerClient: + """Lazily instantiated AWS Secrets Manager client.""" if not hasattr(self, "_secrets_client"): self._secrets_client = boto3.client( "secretsmanager", @@ -645,15 +1085,23 @@ def secrets_client(self) -> SecretsManagerClient: @property def values_path_real(self): + """Resolved absolute path to the values directory.""" return os.path.realpath(self.values_path) @property def temp_dir(self): + """Temporary directory for file processing operations.""" if not hasattr(self, "_temp_dir"): self._temp_dir = tempfile.TemporaryDirectory("app-bootstrap") return self._temp_dir.name def initBootstrap(self): + """ + Initialize the bootstrap infrastructure by creating the S3 bucket. + + Creates a private S3 bucket for storing configuration files. + Handles cases where the bucket already exists. + """ try: self.s3_client.create_bucket( ACL="private", @@ -676,6 +1124,18 @@ def initBootstrap(self): self.click.secho(f"S3 Client Error {err}", bg="red", fg="white") def put_config(self, delete_first): + """ + Push all local configuration to AWS. + + For each environment and application: + 1. Cleans up orphaned SSM parameters and secrets + 2. Uploads YAML files to S3 + 3. Pushes parameters to SSM Parameter Store + 4. Pushes secrets to Secrets Manager + + Args: + delete_first: Whether to delete orphaned resources before pushing. + """ self.cleanup_ssm_parameters() self.cleanup_secrets() for _environment_name, environment in self.environments.items(): @@ -686,17 +1146,28 @@ def put_config(self, delete_first): app.push_secrets() def cleanup_ssm_parameters(self): + """Remove orphaned SSM parameters across all environments and apps.""" for _environment_name, environment in self.environments.items(): for _app_name, app in environment.apps.items(): app.cleanup_ssm_parameters() def cleanup_secrets(self): + """Remove orphaned secrets across all environments and apps.""" for _environment_name, environment in self.environments.items(): for _app_name, app in environment.apps.items(): app.cleanup_secrets() @property def environments(self) -> typing.Dict[str, BootstrapEnvironment]: + """ + Dictionary of all BootstrapEnvironment instances. + + Scans the values_path directory for subdirectories and creates + a BootstrapEnvironment for each. + + Returns: + dict: Environment name -> BootstrapEnvironment mapping. + """ if not hasattr(self, "_environments"): self._environments = {} for item in os.listdir(self.values_path_real): @@ -711,6 +1182,12 @@ def environments(self) -> typing.Dict[str, BootstrapEnvironment]: return self._environments def list_apps(self): + """ + List all applications stored in S3 across all environments. + + Returns: + dict: Environment name -> list of app names mapping. + """ paginator = self.s3_client.get_paginator("list_objects") response_iterator = paginator.paginate( Bucket=self.bucket_name, diff --git a/croudtech_bootstrap_app/cli.py b/croudtech_bootstrap_app/cli.py index 045f25c..73c96df 100644 --- a/croudtech_bootstrap_app/cli.py +++ b/croudtech_bootstrap_app/cli.py @@ -1,3 +1,23 @@ +""" +Command-line interface for croudtech-bootstrap. + +This module provides CLI commands for managing application configuration +across multiple environments using AWS services (S3, SSM, Secrets Manager). + +Commands: + init: Initialize bootstrap infrastructure (create S3 bucket) + get-config: Retrieve configuration for an application + put-config: Push local configuration to AWS + cleanup-secrets: Remove orphaned secrets + list-apps: List all configured applications + manage-redis: Redis database allocation management + +Usage: + croudtech-bootstrap --help + croudtech-bootstrap get-config --environment-name prod --app-name myapp + croudtech-bootstrap put-config ./config +""" + import json import os @@ -16,6 +36,15 @@ def object2table(object): + """ + Format a dictionary as an ASCII table. + + Args: + object: Dictionary to format. + + Returns: + str: ASCII table representation. + """ col1width = len(max(object.keys(), key=len)) col2width = len(str(max(object.values()))) headfoot = "+-%s-+-%s-+" % ("-" * col1width, "-" * col2width) @@ -45,6 +74,12 @@ def object2table(object): ) @click.pass_context def cli(ctx, endpoint_url, put_metrics, bucket_name): + """ + Croudtech Bootstrap - Application configuration management CLI. + + Manage application configuration across multiple environments using + AWS S3, SSM Parameter Store, and Secrets Manager. + """ # ensure that ctx.obj exists and is a dict (in case `cli()` is called # by means other than the `if` block below) ctx.ensure_object(dict) @@ -68,6 +103,7 @@ def cli(ctx, endpoint_url, put_metrics, bucket_name): @click.option("--environment-name", help="The environment name", required=True) @click.option("--region", default="eu-west-2", help="The AWS region") def init(ctx, environment_name, region): + """Initialize bootstrap infrastructure by creating the S3 bucket.""" bootstrap_manager = BootstrapManager( prefix=None, region=region, @@ -89,12 +125,13 @@ def init(ctx, environment_name, region): "--include-common/--ignore-common", default=True, is_flag=True, - help="Include shared variables", + help="Include shared variables from common.yaml", ) @click.option( "--output-format", default="json", type=click.Choice(["json", "yaml", "environment", "environment-export"]), + help="Output format for configuration", ) @click.option( "--parse-redis-param/--ignore-redis-param", @@ -112,6 +149,18 @@ def get_config( output_format, parse_redis_param, ): + """ + Retrieve configuration for a specific application and environment. + + Fetches configuration from AWS (S3 and Secrets Manager), merges with + common configuration if enabled, and outputs in the specified format. + + Output formats: + - json: Pretty-printed JSON + - yaml: YAML format + - environment: Shell variable format (VAR="value") + - environment-export: Shell export format (export VAR="value") + """ bootstrap = BootstrapParameters( environment_name=environment_name, app_name=app_name, @@ -146,10 +195,18 @@ def get_config( "--delete-first", is_flag=True, default=False, - help="Delete the values in this path before pushing (useful for cleanup)", + help="Delete orphaned values before pushing (cleanup mode)", ) @click.argument("values_path") def put_config(ctx, prefix, region, delete_first, values_path): + """ + Push local configuration files to AWS. + + Reads YAML configuration from VALUES_PATH and uploads to: + - S3: Raw YAML files + - SSM Parameter Store: Individual parameters + - Secrets Manager: Sensitive values from .secret.yaml files + """ bootstrap_manager = BootstrapManager( prefix=prefix, region=region, @@ -170,10 +227,16 @@ def put_config(ctx, prefix, region, delete_first, values_path): "--delete-first", is_flag=True, default=False, - help="Delete the values in this path before pushing (useful for cleanup)", + help="Delete orphaned values before cleanup", ) @click.argument("values_path") def cleanup_secrets(ctx, prefix, region, delete_first, values_path): + """ + Remove orphaned secrets from AWS Secrets Manager. + + Compares local configuration files with remote secrets and removes + any secrets that no longer have corresponding local definitions. + """ bootstrap_manager = BootstrapManager( prefix=prefix, region=region, @@ -191,6 +254,7 @@ def cleanup_secrets(ctx, prefix, region, delete_first, values_path): @click.option("--prefix", default="/appconfig", help="The path prefix") @click.option("--region", default="eu-west-2", help="The AWS region") def list_apps(ctx, prefix, region): + """List all applications stored in S3 across all environments.""" bootstrap_manager = BootstrapManager( prefix=prefix, region=region, diff --git a/croudtech_bootstrap_app/redis_config.py b/croudtech_bootstrap_app/redis_config.py index 7a7aaa5..9a52b71 100644 --- a/croudtech_bootstrap_app/redis_config.py +++ b/croudtech_bootstrap_app/redis_config.py @@ -1,3 +1,13 @@ +""" +Redis database allocation management for multi-tenant applications. + +This module provides automatic Redis database allocation for applications +sharing a single Redis instance. Each application is assigned a unique +database number (0-14) to ensure data isolation. + +Database 15 is reserved for storing allocation metadata. +""" + import json from typing import Dict @@ -7,6 +17,45 @@ class RedisConfig: + """ + Manages Redis database allocation for multi-tenant environments. + + In a shared Redis instance, multiple applications need isolated databases. + This class tracks which database numbers are allocated to which applications + and provides automatic allocation of available databases. + + Database Allocation: + - Databases 0-14 are available for applications + - Database 15 is reserved for storing allocation metadata + - Allocations are stored as JSON in the 'allocated_dbs' key + + Allocation Key Format: + Applications are identified by "{environment}_{app_name}" keys. + For example: "production_myapp" -> database 3 + + Usage: + When REDIS_DB is set to "auto" in an application's configuration, + the bootstrap system will automatically allocate a database using + this class. + + Args: + redis_host: Redis server hostname or IP. + redis_port: Redis server port (default: 6379). + app_name: Application name for allocation tracking. + environment: Environment name for allocation tracking. + put_metrics: Whether to publish CloudWatch metrics (default: True). + + Example: + >>> config = RedisConfig( + ... redis_host="localhost", + ... redis_port=6379, + ... app_name="myapp", + ... environment="production" + ... ) + >>> db_number = config.get_redis_database(allocate=True) + >>> print(f"Using Redis database {db_number}") + """ + _redis_dbs: Dict[int, redis.Redis] = {} _config_db = 15 _allocated_dbs_key = "allocated_dbs" @@ -21,6 +70,7 @@ def __init__(self, redis_host, redis_port, app_name, environment, put_metrics=Tr @property def strict_redis(self): + """StrictRedis client for general Redis operations.""" if not hasattr(self, "_strict_redis"): self._strict_redis = redis.StrictRedis( host=self._redis_host, port=self._redis_port @@ -29,6 +79,11 @@ def strict_redis(self): @property def redis_config(self): + """ + Redis client connected to the config database (DB 15). + + Initializes the allocation tracking structure if it doesn't exist. + """ if not hasattr(self, "_redis_config"): self._redis_config = redis.Redis( host=self._redis_host, port=self._redis_port, db=self._config_db @@ -42,13 +97,24 @@ def redis_config(self): @property def redis_db_allocations(self): + """Current database allocation map: {app_key: db_number}.""" return json.loads(self.redis_config.get(self._allocated_dbs_key)) @property def db_key(self): + """Unique key for this app/environment: "{environment}_{app_name}".""" return "%s_%s" % (self._environment, self._app_name) def get_redis_database(self, allocate=False): + """ + Get the allocated database number for this application. + + Args: + allocate: If True, allocate a new database if one doesn't exist. + + Returns: + int or None: The database number, or None if not allocated. + """ allocated_dbs = self.redis_db_allocations if self.db_key not in allocated_dbs and allocate: allocated_db = self.allocate_db() @@ -66,6 +132,15 @@ def get_redis_database(self, allocate=False): return allocated_db def allocate_db(self): + """ + Allocate the next available database to this application. + + Returns: + int: The newly allocated database number. + + Raises: + IndexError: If no databases are available (all 0-14 are allocated). + """ unused_dbs = self.get_unused_dbs() db = unused_dbs[0] db_config = self.redis_db_allocations @@ -75,6 +150,12 @@ def allocate_db(self): return db def deallocate_db(self): + """ + Remove the database allocation for this application. + + Returns: + tuple: (success: bool, db_number: int or None) + """ self.get_unused_dbs() db_config = self.redis_db_allocations if self.db_key in db_config: @@ -86,6 +167,15 @@ def deallocate_db(self): return False, None def get_redis_allocated_db(self, db): + """ + Get a Redis client connected to a specific database. + + Args: + db: Database number to connect to. + + Returns: + redis.Redis: Client connected to the specified database. + """ if db not in self._redis_dbs: self._redis_dbs[db] = redis.Redis( host=self._redis_host, port=self._redis_port, db=db @@ -93,6 +183,12 @@ def get_redis_allocated_db(self, db): return self._redis_dbs[db] def get_unused_dbs(self): + """ + Get list of unallocated database numbers. + + Returns: + list: Available database numbers (from 0-14). + """ possible_dbs = range(0, 15) return list( set(possible_dbs) @@ -100,7 +196,9 @@ def get_unused_dbs(self): ) def get_databases(self): + """Get Redis server database configuration.""" return self.strict_redis.config_get("databases") def get_keyspace(self): + """Get Redis keyspace statistics for all databases.""" return self.strict_redis.info("keyspace")