drop ./devops/run.sh train launch time from 20s -> 2s with web-based callbacks, lazy imports, and default tasks 10k -> 64

[relh]

Summary

This PR reduces training startup latency by overlapping slow preflight work (AWS/SSO checkpoint routing + optional stats client init) with environment/policy construction. It also makes mission lookup lazy/cached, and reduces the default active curriculum task count to avoid unnecessary work on startup.

The only “save/load interface” change in this PR is the introduction of a PolicyStorageDecision object used to decide where checkpoints are saved and how they’re resolved during launch. The on‑disk checkpoint format is unchanged.

Motivation

Startup was spending time in a few places that don’t need to be on the critical path:

• AWS/SSO checks for remote checkpoint routing
• eager loading of mission registries
• very large default curriculum task counts

These changes pull those costs earlier or defer them so that training can begin faster without changing functional behavior.

What Changed (High Level)

• Preflight executor in TrainTool to overlap storage decision + stats client init with env/policy init
• PolicyStorageDecision passed into CheckpointManager to avoid recomputing AWS/SSO routing
• Lazy/cached mission registries and cleaner mission resolution (core + evals)
• Default curriculum active tasks lowered (10000 → 64)
• Lazy imports in heavy recipe entrypoints

---

“Save/Load Interface” (Checkpoint Routing)

PolicyStorageDecision (new data object)

metta/tools/utils/auto_config.py

• Captures where policies should be saved:
  • base_prefix: base remote prefix (e.g., s3://...) or None
  • remote_prefix: run‑specific prefix (base/run) or None
  • reason: why remote is or isn’t used
• using_remote indicates when remote storage is active

CheckpointManager now accepts a decision

metta/rl/checkpoint_manager.py

• CheckpointManager(..., storage_decision=...) uses the precomputed decision to set _remote_prefix
• Avoids recomputing AWS/SSO checks during startup
• Save/load behavior remains the same; only the decision is centralized

TrainTool uses it in preflight

metta/tools/train.py

• auto_policy_storage_decision(run_name) starts in a background thread
• The result is passed to CheckpointManager once available
• Preserves fail‑fast behavior on decision errors

Net effect: the “save/load interface” is still the same file/URI scheme, but the routing decision is now explicit, cached, and overlapped with startup.

---

Launch Sequence (TrainTool)

1. Determine run_name
2. Start preflight executor (if needed):
   • storage decision
   • optional stats client
3. Build vectorized environment + policy
4. Apply storage decision to CheckpointManager
5. Initialize logging + trainer
6. Register components and start training

This keeps behavior identical while shortening the critical path.

---

Mission Lookup (Lazy + Cached)

Lazy registries

• Core missions load immediately; eval missions load only when requested
• Registries are cached (lru_cache) so repeated lookups are cheap

Cleaner resolution

• find_mission(..., include_evals=True) provides a single path for “core + eval” lookup
• If no site matches and mission_name is None, we treat the input as a mission name (fixes easy_hearts in recipe tests)

---

Curriculum Defaults

metta/cogworks/curriculum/curriculum.py

• num_active_tasks default reduced from 10000 → 64
• Tests updated accordingly

---

Recipe + Import Cleanup

recipes/experiment/machina_1.py

• Heavy imports moved inside function bodies to reduce CLI startup time

---

Testing

• CI: Lint, Python, Recipes, C++/Nim
• Updated curriculum default test
• Recipe tests now pass easy_hearts mission lookup via find_mission(..., include_evals=True)

Asana Task

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-official]

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: afea62c | Docs | Was this helpful? Give us feedback!}

[relh]

Startup timing (machina_1 via ./devops/run.sh, sandbox=true, total_timesteps=1): origin/main 22.56s vs richard-launchfixes 2.79s. Both runs used W&B + SSO; main run was a single cold start.

[relh]

• drop ./devops/run.sh train launch time from 20s -> 2s with web-based callbacks, lazy imports, and default tasks 10k -> 64 #4545 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.