feat: refactor GTFS realtime data storage to use a FeedData struct per feed with granular locking#480
Draft
mann-patwa wants to merge 1 commit intoOneBusAway:mainfrom
Draft
feat: refactor GTFS realtime data storage to use a FeedData struct per feed with granular locking#480mann-patwa wants to merge 1 commit intoOneBusAway:mainfrom
FeedData struct per feed with granular locking#480mann-patwa wants to merge 1 commit intoOneBusAway:mainfrom
Conversation
64facf5 to
d7751eb
Compare
…r feed with granular locking
d7751eb to
00d8539
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #479
Description
This PR refactors the GTFS Manager to use fine-grained, per-feed locking for realtime data ingestion.
Previously, a global reader/writer lock (
realTimeMutex) was held for the entire duration of a feed update. In environments with multiple realtime feeds updating concurrently, this caused significant lock contention. A slower feed update would block all other feeds from writing and could potentially block readers trying to access the current state.This change introduces a localized
FeedDatastruct with its ownsync.RWMutex. This allows individual feeds to process and store their realtime data independently and concurrently. The global lock is now only acquired briefly during the final aggregation step (buildMergedRealtime), where pointers to the updated feed data are merged into the global system view.Changes Proposed
1. Introduced
FeedDataStructReplaced the separate:
feedTripsfeedVehiclesfeedAlertsfeedVehicleLastSeenWith a single:
This bundles a feed's realtime state into a cohesive structure, improving maintainability and encapsulation.
2. Added Per-Feed Locks
sync.RWMutexinsideFeedData.This enables true parallel processing of realtime feeds.
3. Map Synchronization
feedMapMutex.map[string]*FeedData.This guarantees thread-safe feed lifecycle management.
4. Optimized Merging (
buildMergedRealtime)a) Merge Serialization
mergeMutexto serialize concurrent merge operations.b) Feed-Level Locking During Merge
RWMutexonly long enough to copy its data slices.This ensures minimal blocking at the feed level.
5. Updated Tests
FeedDataarchitecture.6. Added Test Coverage
New test cases were added to validate: