Skip to content

Conversation

@relh
Copy link
Contributor

@relh relh commented Dec 28, 2025

Summary

Reduce training/environment stats volume and simplify StatsReporter output. This trims low‑value metrics, removes weight/dormant‑neuron reporting, and narrows rolling averages to a small set of required env metrics.

Why

We’re intentionally cutting noisy or redundant metrics to lower logging overhead and make dashboards easier to read, while keeping core signals intact.

What changed

Environment stats production (mettagrid)

  • Removed label completion tracking/emission from StatsTracker.
  • Reward estimate diffs and env timing stats are still produced in the env but are now filtered out before logging.

Stats filtering (metta/rl/stats.py)

  • filter_movement_metrics now drops:
    • env_attributes/*
    • env_reward_estimates/*
    • env_timing_per_epoch/*
    • env_timing_cumulative/*
  • Removed stale env_label_completions/* filter entry (metric no longer produced in this branch).
  • Core movement metrics kept: only the four direction counters.

StatsReporter (metta/rl/training/stats_reporter.py)

  • Removed weight metrics + dormant‑neuron analysis.
  • Removed parameters/* payload (learning rate, epoch steps, minibatches, schedulefree extras).
  • Hyperparameters now come only from optimizer param groups (lr, schedulefree scheduled_lr, lr_max).
  • Rolling averages computed only for default_zero_metrics instead of all env metrics.

W&B logger (metta/rl/training/wandb_logger.py)

  • Removed direct logging of latest_losses_stats; losses now only flow through StatsReporter.
    • If StatsReporter is disabled, loss metrics will not appear.

System monitor (mettagrid)

  • Dropped static counters: cpu_count, cpu_count_logical/physical, memory_total_mb, gpu_count.

Behavior / compatibility notes

  • W&B/env metric output is substantially reduced. Any dashboards/scripts that reference removed metrics will need updates.
  • Loss metrics are no longer emitted by WandbLogger when StatsReporter is disabled.

Testing

Not run (stats/logging changes only).

Files touched

  • metta/rl/stats.py
  • metta/rl/training/stats_reporter.py
  • metta/rl/training/wandb_logger.py
  • packages/mettagrid/python/src/mettagrid/envs/stats_tracker.py
  • packages/mettagrid/python/src/mettagrid/profiling/system_monitor.py
  • tests/rl/test_stats_reporter_defaults.py

Asana Task

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

def _build_wandb_payload(
self,
*,
losses_stats: dict[str, float],
experience: Any,

P1 Badge Fix optimizer arg regression in _build_wandb_payload

The signature of _build_wandb_payload no longer accepts optimizer (and other hyperparameter inputs), but tests/rl/test_stats_reporter_defaults.py::test_heart_metric_zero_fill_and_preserve still invokes it with optimizer=. That test now raises TypeError: _build_wandb_payload() got an unexpected keyword argument 'optimizer', so the suite fails and any callers using the previous API will break. Please either restore compatibility or update the call sites/tests to match the new signature.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-official
Copy link

datadog-official bot commented Dec 28, 2025

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0f4d8b1 | Docs | Was this helpful? Give us feedback!

@relh relh enabled auto-merge December 28, 2025 16:55
@relh relh added the review wanted: stamp This PR needs a review from any available team member label Dec 28, 2025
@relh relh added this pull request to the merge queue Dec 29, 2025
Merged via the queue into main with commit 69496e9 Dec 29, 2025
16 checks passed
@relh relh deleted the richard-perf1226 branch December 29, 2025 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review wanted: stamp This PR needs a review from any available team member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants