Stoat CI for GitLab (Alpha)

Stoat provides GitLab CI/CD components for tracking and debugging CI pipelines. It currently has these features:

• Artifact Collection and Report Component: Collect artifacts from your CI jobs and post a summary report as a comment on your merge requests.
• Tailscale Connect Component: Install Tailscale on your CI runners to enable remote SSH access for debugging stuck or flaky jobs.

Live demo:

• Merge request with artifact report
• Job logs with Tailscale

Note

Stoat CI for GitLab is in alpha stage. The overall design and interfaces are subject to change based on user feedback.

Artifact Collection and Report Component

Installation

1. Add Components to Your Pipeline

Include the Stoat collect and report components in .gitlab-ci.yml and add a report stage:

include:
  - component: $CI_SERVER_FQDN/stoat-dev/ci/collect@<VERSION>
    inputs:
      artifacts:
  - component: $CI_SERVER_FQDN/stoat-dev/ci/report@<VERSION>
    inputs:
      # This token is created in the next step
      api_token: $GITLAB_API_TOKEN

stages:
  # ...
  - report

2. Create a GitLab Access Token

• Go to your project's Settings → Access Tokens
• Create a new token with api scope.
• Copy the generated token.
• Go to Settings → CI/CD → Variables
• Add a new variable:
  • Key: GITLAB_API_TOKEN (this variable name must match the one used in the report component input above)
  • Value: [paste your token]
  • Type: Variable
  • Flags: Check "Mask variable" to hide it in logs
  • Flags: DO NOT check "Protect variable" (unless you only want it to run on protected branches)

3. Track Your Artifacts

For each artifact you want to track, configure job artifact according to GitLab docs, and extend the job that generates the artifact with .stoat-collect:

# This is a job that generates a single-file artifact
lint-build:
  stage: build
  # Add this line to extend the Stoat collect job component
  extends: .stoat-collect
  script:
    - npm run lint -- --format json --output-file lint-report.json || true
  # Config job artifact according to GitLab docs
  artifacts:
    when: always
    paths:
      - lint-report.json

# This is a job that generates a directory artifact
test-suite:
  stage: test
  extends: .stoat-collect
  script:
    - npm test
  artifacts:
    paths:
      - coverage/lcov-report

And register the entrypoint of the artifact in the collect component:

include:
  - component: $CI_SERVER_FQDN/stoat-dev/ci/collect@<VERSION>
    inputs:
      artifacts:
        - name: "Lint Report"
          # For single-file artifact, the path is the file itself
          path: "lint-report.json"
        - name: "Test Coverage"
          # For directory artifact, the path is the entrypoint file,
          # i.e. the file you want the link to point to
          path: "coverage/lcov-report/index.html"

That's it! Each job automatically reports any artifacts that match the paths you defined in the component inputs.

Note

The path defined in Stoat collect component points to the entrypoint of the artifact, while the job's artifact path should include the entire directory containing all relevant files. For example, for test coverage report, the job's artifact path is coverage/lcov-report, and the Stoat collect path is coverage/lcov-report/index.html.

Configuration

Report Component Options

Customize the report component behavior with inputs:

include:
  - component: $CI_SERVER_FQDN/stoat-dev/ci/report@<VERSION>
    inputs:
      api_token: $GITLAB_API_TOKEN
      job-name: "artifact-report"          # Custom job name (default: "stoat-report")
      stage: report                        # Stage to run in (default: "report")
      mr_comment: true                     # Post to MR (default: true)
      commit_comment: false                # Post to commit (default: false)

Note

If both mr_comment and commit_comment are enabled, the commit comment will be skipped when running in an MR context to avoid duplicates.

Warning

If both mr_comment and commit_comment are enabled, it is possible that two comments will be posted when an MR is created, one for the MR and one for the commit. The Stoat report component will not run for the commit when an MR is created. However, the commit pipeline may already kick off before an MR is created. To avoid this, you can disable commit_comment and only enable mr_comment.

Example Output

The report component posts a comment like this on your merge request:

Artifacts

Job	Name	Commit	Status
build-frontend	Frontend Build	a1b2c3d	✅
build-backend	Backend Binary	a1b2c3d	✅
test-suite	Test Coverage	a1b2c3d	✅

Powered by Stoat ↗︎

Tailscale Connect Component

The connect component enables remote debugging of CI jobs by installing Tailscale on the runner and providing SSH access.

Prerequisites

1. Create a Tailscale Ephemeral Reusable Auth Key

1. Log in to the Tailscale Admin Console
2. Navigate to Settings → Keys
3. Click Generate auth key
4. Configure the key:

• Check Reusable (allows the key to be used for multiple runners)
• Check Ephemeral (automatically removes devices when they disconnect)
• Set an appropriate expiration (e.g., 90 days)

5. Copy the generated auth key

2. Add the Auth Key to GitLab

1. Go to your project's Settings → CI/CD → Variables
2. Add a new variable:

• Key: TAILSCALE_AUTH_KEY
• Value: [paste your auth key]
• Type: Variable
• Flags: Check "Mask variable" to hide it in logs
• Flags: DO NOT check "Protect variable" (unless you only want it to run on protected branches)

3. Set Up SSH Access on Runners (Optional)

The connect component can automatically set up SSH access in three ways:

Option A: Generate SSH key automatically (Recommended for quick debugging)

No setup required! The component will generate a new SSH key pair for each job and display the private key in the logs.

include:
  - component: $CI_SERVER_FQDN/stoat-dev/ci/connect@<VERSION>
    inputs:
      tailscale_auth_key: $TAILSCALE_AUTH_KEY
      generate_ssh_key: true

When enabled, the component will:

• Generate a new RSA 4096-bit SSH key pair
• Add the public key to the runner's authorized_keys
• Display the private key in job logs (save it to connect)
• Clean up the keys when the job completes

Option B: Provide your own public SSH key

include:
  - component: $CI_SERVER_FQDN/stoat-dev/ci/connect@<VERSION>
    inputs:
      tailscale_auth_key: $TAILSCALE_AUTH_KEY
      public_ssh_key: $MY_SSH_PUBLIC_KEY

To set up your public key:

1. Generate an SSH key locally: ssh-keygen -t rsa -b 4096
2. Copy your public key: cat ~/.ssh/id_rsa.pub
3. Add it as a GitLab CI variable:

• Key: MY_SSH_PUBLIC_KEY
• Value: ssh-rsa AAAAB3NzaC1yc2E... (your full public key)
• Flags: No need to mask (it's a public key)

Option C: Pre-configure SSH manually

If you prefer to set up SSH access manually on each runner:

1. Install SSH server (if not already installed):

   # Ubuntu/Debian
   sudo apt install openssh-server
   
   # RHEL/CentOS/Fedora
   sudo yum install openssh-server

2. Add your public SSH key to the runner user's authorized keys:

   # On your local machine, copy your public key
   cat ~/.ssh/id_rsa.pub
   
   # On the runner machine, add it to authorized_keys
   # Replace 'gitlab-runner' with your actual runner username
   sudo su - gitlab-runner
   mkdir -p ~/.ssh
   echo "your-public-key-here" >> ~/.ssh/authorized_keys
   chmod 600 ~/.ssh/authorized_keys

3. Ensure SSH server is running:

   sudo systemctl enable ssh
   sudo systemctl start ssh

Note

The component will attempt to install and start SSH server automatically if it's not running. This requires the runner to have appropriate permissions (root or sudo access).

Usage

Include the connect component in your pipeline:

include:
  - component: $CI_SERVER_FQDN/stoat-dev/ci/connect@<VERSION>
    inputs:
      tailscale_auth_key: $TAILSCALE_AUTH_KEY

Extend jobs that you need to connect to with .stoat-connect:

test-job:
  stage: test
  extends: .stoat-connect
  script:
    - npm run test

Important

The connect component performs all setup in before_script. Do not override before_script in jobs extending .stoat-connect, or the component will not work.

Configuration Options

include:
  - component: $CI_SERVER_FQDN/stoat-dev/ci/connect@<VERSION>
    inputs:
      tailscale_auth_key: $TAILSCALE_AUTH_KEY
      enabled: true                         # Optional: Enable/disable (default: true)
      install_tailscale: false              # Optional: Install if missing (default: false)
      install_ssh_server: false             # Optional: Install SSH if missing (default: false)
      ssh_user: "gitlab-runner"             # Optional: SSH username (defaults to current user)
      generate_ssh_key: false               # Optional: Generate SSH key pair (default: false)
      public_ssh_key: $MY_SSH_PUBLIC_KEY    # Optional: Your public SSH key to add to runner

Note

install_tailscale and install_ssh_server require explicit opt-in (set to true). Without these flags, the component skips installation if packages are missing, protecting your runner from unexpected modifications.

Per-Job Customization

Disable the connect component for specific jobs:

test-job:
  extends: .stoat-connect
  variables:
    TAILSCALE_ENABLED: "false"
  script:
    - npm run test

Override install flags per job:

test-job:
  extends: .stoat-connect
  variables:
    INSTALL_TAILSCALE: "true"      # Install Tailscale for this job only
    INSTALL_SSH_SERVER: "true"     # Install SSH server for this job only
  script:
    - npm run test

Add custom hostname suffix:

test-job:
  stage: test
  extends: .stoat-connect
  variables:
    TAILSCALE_HOSTNAME_SUFFIX: "fpga-test"
  script:
    - npm run test

Connecting to a Stuck Job

When a job gets stuck:

1. Find the connection info in the job logs:

   With generated SSH key:

   ==========================================
   TAILSCALE DEBUG ACCESS
   ==========================================
   Tailscale IPv4:  100.64.123.45
   Tailscale IPv6:  fd7a:115c:a1e0::1234
   SSH Command:     ssh gitlab-runner@100.64.123.45
   Runner Hostname: runner-abc123
   Job:             flaky-mac-test (ID: 98765)
   Pipeline:        12345
   
   ==========================================
   GENERATED SSH PRIVATE KEY
   ==========================================
   ⚠️  SENSITIVE: Save this private key securely
   
   -----BEGIN OPENSSH PRIVATE KEY-----
   b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAACFwAAAA
   ...
   -----END OPENSSH PRIVATE KEY-----
   
   ==========================================
   
   To use this key:
   1. Save the private key above to a file (e.g., ~/.ssh/stoat_debug_key)
   2. Set permissions: chmod 600 ~/.ssh/stoat_debug_key
   3. Connect: ssh -i ~/.ssh/stoat_debug_key gitlab-runner@100.64.123.45
   

   With existing SSH key or pre-configured SSH:

   ==========================================
   TAILSCALE DEBUG ACCESS
   ==========================================
   Tailscale IPv4:  100.64.123.45
   Tailscale IPv6:  fd7a:115c:a1e0::1234
   SSH Command:     ssh gitlab-runner@100.64.123.45
   Runner Hostname: runner-abc123
   Job:             flaky-mac-test (ID: 98765)
   Pipeline:        12345
   ==========================================
   

2. Connect via SSH:

   # With generated key
   ssh -i ~/.ssh/stoat_debug_key gitlab-runner@100.64.123.45
   
   # With existing key or pre-configured SSH
   ssh gitlab-runner@100.64.123.45

3. Debug the stuck job on the runner

4. Connection cleanup: The Tailscale connection automatically terminates when the job completes or fails

Limitations

• Currently, the connect component supports Linux runners only
• The component performs all the work in before_script. Any job extending this component must not override before_script, or no Tailscale / SSH setup will occur
• MacOS support is planned for future releases
• Multiple jobs running on the same runner simultaneously may cause Tailscale conflicts (this is a known issue and will be addressed in a future update)
• SSH server installation requires appropriate permissions on the runner (root or sudo access)

Tentative Roadmap

• CLI
  • Update the stoat CLI to integrate with GitLab CI/CD with one command
• Metrics
  • Track runtime metrics (e.g., test duration, memory usage)
  • This may not be necessary given GitLab's built-in metrics dashboard
• App
  • Add GitLab app and backend server for smoother integration
  • This should eliminate the necessity to create personal / group access tokens
• Connect
  • Enable users to connect into the runner for real-time debugging
  • Add macOS support for the connect component
  • Improve handling of multiple concurrent jobs on the same runner

Support

• Issues: gitlab.com/stoat-dev/ci/issues
• Website: stoat.dev

CHANGELOG

See CHANGELOG.md

License

MIT