Backup Storage Bot

Backup Storage Bot is a Python based automated repository backup tool designed for multiple Git repositories. It supports local, S3, and external storage, provides a web UI for browsing backups and logs and enforces optional network restrictions for enhanced security.

Features

• Automatic backups of multiple Git repositories.

• Configurable backup schedule (weeks, days, hours).

• Multiple storage backends:

  • Local filesystem

  • S3 buckets

  • Custom external storage

• Cleanup of old backups and logs based on retention rules.

• Web UI for browsing backups, downloading files, and monitoring logs.

• Manual backup trigger via the web interface.

• Role based access control (Admin, User, Guest).

• Optional network restriction rules for secure environments.

Installation

Clone the repository:

git clone https://github.com/BBlue530/Backup_Storage_Bot.git
cd Backup_Storage_Bot

Install dependencies:

pip install -r requirements.txt

Set environment variables (example):

export GITHUB_PAT="your_github_personal_access_token"
export BUCKET="your_s3_bucket"
export BUCKET_KEY="your_s3_key"
export API_KEY="your_external_api_key"
export AWS_ACCESS_KEY_ID="your_aws_key"
export AWS_SECRET_ACCESS_KEY="your_aws_secret"
export AWS_DEFAULT_REGION="your_aws_region"

Configuration

All settings are stored in app-config.yaml:

backend:
  storage:
    backup:
      backup_schedule:
        weeks: 2
        days: 0
        hours: 0
      connection_ruleset:
        enabled: False
      web_ui:
        enabled: True
      backup_sources:
        backup_storage_bot:
          url: https://github.com/BBlue530/Backup_Storage_Bot.git
          pat: "${GITHUB_PAT}"
      backup_storage:
        local:
          enabled: True
          location: "backup"
        s3_bucket:
          enabled: False
        external_storage:
          enabled: False
      log_storage:
        local:
          enabled: True
          location: "data/log.json"
      cleanup:
        cleanup_old_logs:
          enabled: True
          keep_old_amount: 10
        cleanup_old_backups:
          enabled: True
          keep_old_amount: 5
      auth:
        aws:
          enabled: False

Important Keys:

• backup_schedule: Sets the frequency of backups.

• backup_sources: List all repositories to back up. Use unique keys.

• backup_storage: Choose storage backend(s) and locations.

• log_storage: Configure log storage.

• cleanup: Enable automatic removal of old backups and logs.

• auth: Configure credentials for cloud storage (AWS).

You can disable the web UI if it is not needed by setting web_ui to False.

Pipeline Diagram

    [Read Config & Env] 
        ↓
    [Prepare Backup Directories]
        ↓
    [Mirror Clone] → [Working Clone] → [Bundle Creation]
        ↓
    [External Backup Storage (Optional)]
        ↓
    [Cleanup Old Backups & Logs]
        ↓
    [Logging]
        ↓
      Sleep → Repeat

Manual Backup Trigger

• Admins can manually trigger backups via /backup in the web UI.

Web UI

The web interface provides:

• Browsing backup directories.

• Downloading individual files or entire folders as ZIP.

• Viewing logs and audit trails.

• Role based access (Admin, User, Guest).

Example routes:

• /backup Browse backups, download file/folders, trigger manual backup.

• /home – View logs.

Connection Ruleset and Network Restrictions

To restrict access to your deployment and enhance security enable the connection_ruleset by setting it to True.

When enabled, the bot enforces the following network restrictions:

• Inbound traffic: Restricted to local network LAN only.

• Outbound traffic: Limited to HTTPS port 443 for GitHub and S3. If you use external storage it must also use port 443.

Passwordless Execution

To allow the script to refresh rules without a password (recommended due to GitHub IP changes):

sudo visudo

Add the line:

YOUR_NAME ALL=(root) NOPASSWD: /full/path/to/Backup_Bot/src/connection_ruleset.sh

Then set ownership and permissions:

sudo chown root:YOUR_NAME /full/path/to/Backup_Bot/src/connection_ruleset.sh
sudo chmod 750 /full/path/to/Backup_Bot/src/connection_ruleset.sh

Cleanup

• Old backups and logs are automatically removed according to cleanup_old_backups and cleanup_old_logs settings.

• Keeps only the most recent files as specified in the config.