Add scheduled job support and OCI image pull job #147

cmyers-mieweb · 2025-12-10T16:35:23Z

Introduces ScheduledJob model, migrations, and seeder for cron-based job scheduling. Updates job-runner to process scheduled jobs and create pending jobs when schedule conditions are met. Adds support for defaultStorage on nodes, including migration, model, and form update. Implements oci-build-job utility to pull OCI container images to Proxmox nodes using API credentials and storage configuration.

cmyers-mieweb · 2025-12-10T16:50:56Z

Scheduled Jobs / OCI Image Pull Integration

Summary

Add a new ScheduledJobs Sequelize model/migration
Columns: id, schedule (STRING, cron expression), command (STRING), lastRunAt (DATETIME).
Purpose: DB-driven cron-style scheduled tasks.
Add defaultStorage column to Node model
Allows OCI pulls to target the appropriate Proxmox storage.
Seed a ScheduledJob
Runs the OCI image pull job (create-a-container/utils/oci-build-job.js) for Debian 13 and Rocky 9.
Extend the job-runner
Evaluates ScheduledJobs before claiming pending jobs and creates new pending Job records when schedules fire.
oci-build-job implemented
Pulls OCI LXC images directly to Proxmox nodes (no Packer).

Why

Enables recurring automated OCI LXC image pulls for configured Proxmox nodes.
Centralizes scheduling in the database — configurable via API/UI.
Keeps job-runner behavior unchanged by generating standard "pending" Job entries.

Files Changed (High-Level)

Migrations

create-a-container/migrations/XXXXXX-create-scheduled-jobs.js
create-a-container/migrations/XXXXXX-add-default-storage-to-nodes.js

Models

create-a-container/models/scheduledjob.js
node.js (added defaultStorage)

Seeders

create-a-container/seeders/XXXXXX-seed-oci-scheduledjob.js

Job Runner

job-runner.js (added processScheduledJobs, uses cron-parser)

Utils

oci-build-job.js (invoked by ScheduledJob command)

UI

form.ejs (added defaultStorage field)

package.json

Added cron-parser dependency

Architecture Diagram (Mermaid)

graph LR
  A[ScheduledJobs] -->|"evaluated by"| B[Job Runner]
  B -->|"on match"| C[Jobs (pending)]
  C -->|"claimed by"| D[Runner Execution]
  D -->|"executes"| E[OCI Build Job Script]
  E -->|"uses"| F[Node Model (apiUrl, token, defaultStorage)]

Implementation Notes (Concise)

ScheduledJobs Table Columns

id INTEGER PK AUTOINCREMENT
schedule STRING (cron expression, e.g., "0 2 * * *")
command TEXT/STRING
lastRunAt DATETIME NULL
createdAt / updatedAt

Job Runner: `processScheduledJobs()`

Loads all ScheduledJobs
Uses cron-parser to check if now matches
Compares with lastRunAt to avoid duplicate runs
Inserts a record in Jobs table with status "pending"
Updates lastRunAt

Seeded Command

Typically runs:

node oci-build-job.js

Node Model

Added defaultStorage (STRING, NULL)

Step-by-Step Setup (Windows Dev)

Install dependencies

cd c:\Users\cmyers\Documents\GitHub\opensource-server\create-a-container
npm install

Add cron-parser

npm install cron-parser --save

Run migrations

npm run db:migrate
# or:
npx sequelize db:migrate

Seed ScheduledJob

npm run db:seed:all
# or:
npx sequelize db:seed:all

Start job runner

node create-a-container/job-runner.js

Watch logs for scheduled job creation and OCI pull runs.

Testing Instructions

Quick Functional Test

Create a ScheduledJob with:
```
* * * * *
```
Command:
```
node oci-build-job.js
```
Start job-runner → expect new pending job every minute.

Manual run of OCI job

node create-a-container/utils/oci-build-job.js

Verify

DB Jobs table inserts
Per-node pull attempts logged
Pull success/failure messages appear

Suggested Unit / Integration Tests

Model tests
Validate cron string correctness, lastRunAt updating logic.
processScheduledJobs tests
Mock DB + cron-parser, assert inserts.
End-to-end
Disposable DB → runner → assert job gets created.
OCI pull smoke test
Mock Proxmox API endpoint for /pull-image.

DB / Migration Verification Checklist

ScheduledJobs table exists
Nodes.defaultStorage column exists
Seeded OCI ScheduledJob created

Admin UI

Node create/edit form updated
defaultStorage field now selectable/editable

Operational Notes

Job-runner should run at least every 30–60 seconds.
Cron expressions evaluated in server local time unless otherwise configured.
lastRunAt is minute-granular to avoid duplicate runs.

Rollback

Migration down scripts remove both new changes.
Seeder undo removes the OCI ScheduledJob.

Security Notes

Scheduled job commands run with full job-runner privileges.
Restrict ScheduledJob management to trusted admins.
Protect DB tokens & secrets.

Files to Review

create-a-container/migrations/XXXX-create-scheduled-jobs.js
create-a-container/models/scheduledjob.js
create-a-container/migrations/XXXX-add-default-storage-to-nodes.js
node.js
create-a-container/seeders/XXXX-seed-oci-scheduledjob.js
job-runner.js
oci-build-job.js

Acceptance Criteria

ScheduledJobs and defaultStorage exist in schema
Job-runner enqueues Jobs based on schedule
Seeded OCI job runs successfully
Admin UI supports defaultStorage
Automated tests cover scheduling + validation

Notes / TODOs

Define timezone behavior (UTC vs local)
Consider adding timezone field per schedule
Consider adding enabled BOOLEAN
Add audit logs for schedule → job creation

create-a-container/job-runner.js

create-a-container/seeders/20251203000000-seed-oci-build-job.js

create-a-container/utils/oci-build-job.js

runleveldev · 2025-12-11T18:44:08Z

create-a-container/views/nodes/form.ejs


+      <div class="mb-3">
+        <label for="defaultStorage" class="form-label">Default Storage</label>
+        <input 


Has the router been updated to handle this input?

Introduces a script to build and push site-specific OCI images using Docker, a seeder to schedule this job, and a Debian Dockerfile template. Updates oci-build-job.js to prefer LOCAL_REGISTRY for image registry. The README is updated to reference the local registry in example commands.

Replaces separate oci-build-job and build-push-oci scripts with a single oci-build-push-pull job that builds, pushes, and pulls OCI images for all sites and nodes. Updates scheduled job seeders to use the new script and introduces shared Proxmox utilities for task polling and image pulling. Old scripts and their references are removed or deprecated for migration rollback support.

create-a-container/bin/oci-build-job.js

runleveldev · 2025-12-18T13:58:40Z

create-a-container/bin/oci-build-push-pull.js

+ * command path relative to the repository root.
+ */
+
+const { run } = require('../utils/oci-build-push-pull');


Just move the other file here and run it. It should not be in the utils folder because it's not a utility.

runleveldev · 2025-12-18T14:02:49Z

create-a-container/seeders/20251204000000-seed-build-push-oci.js

+  async up(queryInterface, Sequelize) {
+    // This seeder is now superseded by 20251203000000-seed-oci-build-job.js
+    // which uses the combined oci-build-push-pull.js job.
+    // Keeping this file for migration rollback support only.


Delete this file, we don't need it in main since it was never deployed.

runleveldev · 2025-12-18T14:04:24Z

create-a-container/templates/debian.Dockerfile

@@ -0,0 +1,11 @@
+# Debian OCI image template for site-specific builds
+FROM debian:13


We want to derive from Proxmox's base image, not Docker's. See https://github.com/mieweb/opensource-server/blob/main/Dockerfile 2 stage build on how to do that.

excerpt

FROM debian:13 AS builder ARG URL="http://download.proxmox.com/images/system/debian-13-standard_13.1-2_amd64.tar.zst" ARG DOMAIN RUN apt-get update && apt-get install -y --no-install-recommends \ curl \ tar \ zstd \ ca-certificates && \ rm -rf /var/lib/apt/lists/* && \ mkdir -p /rootfs/usr/local/bin && \ curl -fsSL "$URL" | tar --zstd -x -C /rootfs && \ curl -fsSL https://pown.sh/ -o /tmp/pown.sh && \ chmod +x /tmp/pown.sh && \ cp /tmp/pown.sh /rootfs/usr/local/bin/pown.sh && \ chmod +x /rootfs/usr/local/bin/pown.sh && \ chroot /rootfs /usr/local/bin/pown.sh "$DOMAIN"

runleveldev · 2025-12-18T14:05:41Z

create-a-container/utils/oci-build-push-pull.js

+  }
+}
+
+module.exports = { run };


Remove from here down once it's called directly in bin and replace with a single call to run()

runleveldev · 2025-12-18T14:19:47Z

create-a-container/utils/oci-build-push-pull.js

+    // ========== PHASE 2: Prepare all images to pull ==========
+    console.log('\n[oci-build-push-pull] ========== PHASE 2: Prepare Images to Pull ==========');
+
+    const preBuiltImages = getPreBuiltImages();


This Phase 2 can likely be removed (with the following phases refactored accordingly) since we're only concerned with the site-specific images and nothing prebuilt.

runleveldev · 2025-12-18T14:21:24Z

create-a-container/utils/proxmox-api.js

+  async pullImage(nodeName, image, storage) {
+    // Use global fetch (Node 18+). If not available, this will throw and the caller
+    // can fallback or install a fetch polyfill.
+    if (typeof fetch !== 'function') {


Remove this check. We will not be supporting node runtimes <18

runleveldev · 2025-12-18T14:23:17Z

create-a-container/utils/proxmox-api.js

+    headers['Content-Type'] = 'application/json';
+
+    // Determine agent for TLS options if provided
+    let agent = null;


The consumer of this class is responsible for creation the https agent and attaching it to the options object if required. This agent construction here should be removed. And replaced with this.options.httpsAgent

runleveldev · 2025-12-18T14:27:02Z

create-a-container/utils/proxmox-utils.js

If we need these functions (I don't see the need, they're pretty simple wrappers), then they should be methods on the ProxmoxApi class not arbitrary helpers.

runleveldev · 2025-12-18T14:29:02Z

README.md

 ```bash
 # Pull and run the container from GHCR
-pct create <VMID> ghcr.io/mieweb/opensource-server:latest \
+   pct create <VMID> localhost:5000/mieweb/opensource-server:latest \


This is not something that should be updated in this PR. This is part of instructions for bootstrapping the cluster which does use ghcr.io unlike the template builds which won't.

Moved and consolidated the OCI build/push/pull logic into the bin/oci-build-push-pull.js file, removing the old utils/oci-build-push-pull.js and proxmox-utils.js modules. Updated the Dockerfile template to use Proxmox's minimal LXC rootfs for more accurate OCI images. Improved CLI argument parsing and Proxmox API utilities for image pulling and storage selection. Updated README to use ghcr.io for container pulls. Removed obsolete seeder for build-push-oci job.

cmyers-mieweb requested a review from runleveldev December 10, 2025 16:35

runleveldev requested changes Dec 11, 2025

View reviewed changes

Carter Myers added 3 commits December 11, 2025 15:30

Changes requested by Robert

e5a5a05

runleveldev requested changes Dec 18, 2025

View reviewed changes

		@@ -0,0 +1,11 @@
		# Debian OCI image template for site-specific builds
		FROM debian:13

Add scheduled job support and OCI image pull job #147

Are you sure you want to change the base?

Add scheduled job support and OCI image pull job #147

Uh oh!

Conversation

cmyers-mieweb commented Dec 10, 2025

Uh oh!

cmyers-mieweb commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Scheduled Jobs / OCI Image Pull Integration

Summary

Why

Files Changed (High-Level)

Migrations

Models

Seeders

Job Runner

Utils

UI

package.json

Architecture Diagram (Mermaid)

Implementation Notes (Concise)

ScheduledJobs Table Columns

Job Runner: processScheduledJobs()

Seeded Command

Node Model

Step-by-Step Setup (Windows Dev)

Install dependencies

Add cron-parser

Run migrations

Seed ScheduledJob

Start job runner

Testing Instructions

Quick Functional Test

Manual run of OCI job

Verify

Suggested Unit / Integration Tests

DB / Migration Verification Checklist

Admin UI

Operational Notes

Rollback

Security Notes

Files to Review

Acceptance Criteria

Notes / TODOs

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

cmyers-mieweb commented Dec 10, 2025 •

edited

Loading

Job Runner: `processScheduledJobs()`