Skip to content

feat: refactor GTFS realtime data storage to use a FeedData struct per feed with granular locking#480

Draft
mann-patwa wants to merge 1 commit intoOneBusAway:mainfrom
mann-patwa:per-feed-lock
Draft

feat: refactor GTFS realtime data storage to use a FeedData struct per feed with granular locking#480
mann-patwa wants to merge 1 commit intoOneBusAway:mainfrom
mann-patwa:per-feed-lock

Conversation

@mann-patwa
Copy link
Contributor

Fixes #479

Description

This PR refactors the GTFS Manager to use fine-grained, per-feed locking for realtime data ingestion.

Previously, a global reader/writer lock (realTimeMutex) was held for the entire duration of a feed update. In environments with multiple realtime feeds updating concurrently, this caused significant lock contention. A slower feed update would block all other feeds from writing and could potentially block readers trying to access the current state.

This change introduces a localized FeedData struct with its own sync.RWMutex. This allows individual feeds to process and store their realtime data independently and concurrently. The global lock is now only acquired briefly during the final aggregation step (buildMergedRealtime), where pointers to the updated feed data are merged into the global system view.


Changes Proposed

1. Introduced FeedData Struct

Replaced the separate:

  • feedTrips
  • feedVehicles
  • feedAlerts
  • feedVehicleLastSeen

With a single:

map[string]*FeedData

This bundles a feed's realtime state into a cohesive structure, improving maintainability and encapsulation.


2. Added Per-Feed Locks

  • Added a sync.RWMutex inside FeedData.
  • Data extraction, parsing, transformation, and assignment for a specific feed now hold only that feed’s lock.
  • Independent feeds no longer block each other during ingestion.

This enables true parallel processing of realtime feeds.


3. Map Synchronization

  • Introduced a feedMapMutex.
  • Ensures safe read/write access to the top-level map[string]*FeedData.
  • Allows new feeds to be registered safely without race conditions.

This guarantees thread-safe feed lifecycle management.


4. Optimized Merging (buildMergedRealtime)

a) Merge Serialization

  • Introduced a mergeMutex to serialize concurrent merge operations.
  • Prevents overlapping global aggregation steps.

b) Feed-Level Locking During Merge

  • Iterates over sorted feeds.
  • Acquires each feed’s RWMutex only long enough to copy its data slices.
  • Releases immediately after copying.

This ensures minimal blocking at the feed level.


5. Updated Tests

  • Refactored existing unit tests and benchmarks to align with the new FeedData architecture.
  • Updated mocks and test setup where necessary to support per-feed locking.

6. Added Test Coverage

New test cases were added to validate:

  • Retaining previous realtime data on HTTP failures.
  • Correctly handling empty payload responses.
  • Proper expiration of stale vehicles based on last-seen timestamps.
  • Safe concurrent execution of feed updates.

@mann-patwa mann-patwa marked this pull request as draft February 25, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Per-Feed Locking for GTFS Realtime Updates

1 participant