Add audio-gated screen recording for iOS using ReplayKit

[Copilot]

Implements PoC screen recording that captures only when audio exceeds threshold, auto-cutting silence. Uses ReplayKit broadcast extension to analyze audio RMS in real-time and gate video frame writing.

Implementation

Native Module (modules/screen-recorder/)

• Expo module exposing startScreenRecording(), getLastRecordingPath(), isRecording()
• Triggers RPSystemBroadcastPickerView to select broadcast extension
• App group container for data sharing between app and extension

Broadcast Extension (ios/BroadcastExtension/)

• RPBroadcastSampleHandler subclass processes screen + audio samples
• Audio RMS calculation with configurable threshold (default 0.02)
• State machine with hysteresis (300ms to activate, 700ms to deactivate)
• AVAssetWriter outputs H.264/AAC only during active state
• Debug logging of audio levels and state transitions

// Audio gating logic
let rms = sqrt(sum_of_squares / sample_count)
if rms > threshold && silent_for(300ms) → ACTIVE
if rms < threshold && active_for(700ms) → SILENT
write_frames() if state == ACTIVE

UI (app/(tabs)/profile.tsx)

• iOS-only recording controls in Profile tab
• Three buttons: start recording, check last, play last
• Embedded video player with expo-video

Configuration

• Thresholds tunable in SampleHandler.swift
• App groups: group.com.mieweb.pulse.screenrecorder
• Recordings: {container}/Recordings/recording-{timestamp}.mp4

Manual Setup Required

Xcode project modification needed to add Broadcast Extension target:

1. File → New → Target → Broadcast Upload Extension
2. Name: BroadcastExtension, Bundle ID: com.mieweb.pulse.BroadcastExtension
3. Add SampleHandler.swift to target
4. Configure app groups on both targets

See SCREEN_RECORDING_SETUP.md for detailed steps.

Testing

Physical iOS device required (ReplayKit unsupported on simulator). Expected behavior:

• 12s recording with 3s speech, 2s silence, 3s speech, 4s silence → 6s output video
• Console.app shows real-time audio levels and state transitions
• Silent periods completely absent from timeline

Files

• 18 files created (463 lines Swift, 18 lines TS)
• 2 files modified (profile.tsx +186 lines, entitlements +2 lines)
• 5 documentation guides (1,218 lines)

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• cdp.expo.dev
  • Triggering command: /usr/local/bin/node /home/REDACTED/work/pulse/pulse/node_modules/@expo/cli/build/src/utils/telemetry/clients/flushFetchDetached.js /tmp/b527bdf95baf515abf32cd10baf42200/expo-telemetry.json (dns block)
  • Triggering command: /usr/local/bin/node /home/REDACTED/work/pulse/pulse/node_modules/@expo/cli/build/src/utils/telemetry/clients/flushFetchDetached.js /tmp/83dfe08907e57ae4782d1951dd98a567/expo-telemetry.json (dns block)
  • Triggering command: /usr/local/bin/node /home/REDACTED/work/pulse/pulse/node_modules/@expo/cli/build/src/utils/telemetry/clients/flushFetchDetached.js /tmp/83a04291571f28cc5617e268bfc618c1/expo-telemetry.json (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Add “Audio-Gated Screen Recording” PoC to Pulse (iOS)</issue_title>
<issue_description>### Summary
Implement a proof-of-concept screen recording flow in the React Native app Pulse that records the iOS screen via ReplayKit, but only writes video when audio is above a configurable threshold (i.e., auto-cuts silence / low-activity periods). Expose this as a “Record Screen” button in the Pulse UI (iOS-only for now).

---

Background / Motivation

Pulse needs a way to capture short, high-signal screen recordings (e.g. demos, explanations, walkthroughs) without manual editing. The goal is to create something like an “auto short”:

• Capture the screen (ideally any app via iOS screen broadcasting).
• Automatically skip / compress silent or low-audio segments.
• Produce a video that feels edited: mostly “talking/action”, very little dead air.

ReplayKit Broadcast Upload Extensions can access screen + audio samples but not hardware buttons or camera, so audio level is a viable real-time signal for when to include or skip parts of the recording.

This ticket is PoC-only, focused on getting the basic pipeline working end-to-end.

---

Scope

In scope:

• iOS-only PoC.
• A “Record Screen” button in Pulse (React Native).
• ReplayKit-based recording with:
  • Capture of screen + audio.
  • Simple audio-level gating:
    • Only write frames to the output video while audio amplitude is above a threshold.
• Producing and saving a video file that demonstrates:
  • Silence = not recorded / skipped.
  • Speech or loud activity = recorded.

Out of scope (for this ticket):

• Fancy UI for video management (thumbnails, gallery, etc.).
• Timeline editor or segment visualization.
• Android support.
• Tuning ML or complex audio analysis.

---

High-Level Requirements

1. Record Screen Button (React Native UI)

   • Add a “Record Screen” button in Pulse (e.g. on a dev/debug screen or a simple test screen).
   • When tapped:
     • On iOS, trigger the ReplayKit system screen recording prompt (Broadcast).
     • Show some state in Pulse (e.g. “Recording…” vs “Not Recording”) when appropriate.

2. ReplayKit Broadcast Integration (iOS Native)

   • Create a ReplayKit Broadcast Upload Extension for Pulse.
   • The extension should:
     • Receive video + audio sample buffers.
     • Implement a simple audio-level detector:
       • Compute RMS/average power per audio buffer.
       • Compare against a configurable threshold (value can be hard-coded for PoC).
       • Apply small hysteresis & debounce (e.g. require ~200–500 ms above/below threshold to flip state).
     • While in “active” state (audio above threshold):
       • Write incoming video (and audio) samples to an AVAssetWriter output.
     • While in “silent” state (audio below threshold):
       • Skip writing frames (video and audio) to the asset writer.
       • Result: the final video timeline compresses silent segments.

3. Output File Handling

   • When the user stops screen recording (via Control Center or system UI):
     • Finalize the AVAssetWriter.
     • Save the resulting video file to:
       • Either Photos (using Photos framework), or
       • App’s sandbox (Documents/tmp) and expose a simple way to access it (e.g. log path or very simple in-app list).
   • For PoC, logging the file path and optionally presenting a simple “Play Last Recording” button is sufficient.

4. React Native ↔ iOS Bridge

   • Implement a minimal Native Module for iOS to expose:
     • startSystemScreenRecording() → prompts user to start ReplayKit broadcast targeting Pulse’s extension.
     • Optional: openLastRecording() or similar to open the last exported video (for testing).
   • Wire the “Record Screen” button to this native module.

5. Configuration & Debuggability

   • Audio threshold and debounce values should be easy to tweak in code (constants or simple struct).
   • Add basic logging (in debug builds) for:
     • Audio level samples.
     • State transitions: silent → active, active → silent.
     • Start/stop of asset writer.

---

Technical Notes / Implementation Details

• ReplayKit

  • Use Broadcast Upload Extension (RPBroadcastSampleHandler) to receive system-wide screen + audio.
  • Implement processSampleBuffer(_:with:) to handle audio/video:
    • For audio buffers:
      • Extract audio samples, compute power (RMS / average absolute value).
      • Maintain current state: .active or .silent.
    • For video buffers:
      • If state == .active, append to asset writer.
      • If .silent, drop frame.

• Audio Level Detection

  • Use a simple approach for PoC:
    • Convert audio samples to Float.
    • Compute RMS or average absolute value.
    • Compare to a threshold (e.g. 0.01–0.05, to be tuned).
  • Add ...

• Fixes Add “Audio-Gated Screen Recording” PoC to Pulse (iOS) #223

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.