fco

Inference package for FCO (Floating Car Observer) and TFCO (Temporal Floating Car Observer) vehicle detection in SUMO traffic simulations.

FCO detects surrounding vehicles from V2X-equipped floating cars using geometry-based or neural-network detectors. TFCO enhances FCO detections with temporal neural network fusion, recovering occluded vehicles that individual FCOs cannot see.

---

1. Setup

Requirements

• Python 3.8+
• SUMO traffic simulator (>= 1.25.0)
• CUDA-capable GPU (recommended for TFCO and emulation detector)

Installation

pip install -r requirements.txt

SUMO Setup

Ensure SUMO_HOME is set and SUMO binaries are on your path:

export SUMO_HOME="/usr/share/sumo"

The package uses libsumo (C++ TraCI binding) by default for performance. You can also use the standard traci Python module if needed.

SUMO Route File Requirements

FCO determines entity types (vehicle, bicycle, pedestrian) by parsing the prefix of the SUMO ID -- everything before the first underscore _. Your route files must follow this naming convention:

ID prefix	Detected as	Example IDs
vehicle_*	'vehicle'	vehicle_0, vehicle_1
bike_* / bicycle_* / cyclist_*	'bicycle'	bike_0, bicycle_3
ped_* / pedestrian_* / person_* / walker_*	'pedestrian'	ped_0, person_12

Pedestrians must be defined as <person> elements (not <vehicle>) so SUMO exposes them via traci.person.*.

Example route file:

<routes>
    <vehicle id="vehicle_0" depart="0.00">
        <route edges="edge1 edge2 edge3"/>
    </vehicle>
    <vehicle id="bike_0" type="bicycle" depart="5.00">
        <route edges="edge1 edge4"/>
    </vehicle>
    <person id="ped_0" depart="10.00">
        <walk edges="edge1 edge2"/>
    </person>
</routes>

Converting existing route files: If your route files use different naming (e.g., plain numbers, pv_* prefixes, CARLA-style vehicle.brand.model IDs), use the conversion script:

python -m fco.tools.convert_routes input.rou.xml output.rou.xml

The script infers entity type from the element tag (<person> vs <vehicle>) and SUMO's vClass (either directly on the element or resolved through its referenced <vType>), then renames all IDs to the required vehicle_N / bike_N / ped_N format. All original vType definitions and dimensions are preserved. See --help for options.

Package Structure

fco/
├── fco/                          # FCO detection module
│   ├── fco.py                    # Main FCO class
│   ├── v2x.py                    # V2X penetration tracker
│   ├── detector.py               # Detector factory + implementations
│   ├── emulation_detector.py     # Neural-network emulation detector
│   ├── model_loading.py          # Checkpoint utilities
│   ├── configs/
│   │   ├── config_cv.py          # Camera & CV detection settings
│   │   └── config_simulation.py  # Vehicle dimensions, modal split
│   └── utils/
│       ├── collapser.py          # Multi-FCO detection fusion
│       ├── sumo_utils.py         # SUMO configuration helpers
│       ├── bev_generator.py      # BEV image generation
│       └── ...
├── tfco/                         # TFCO temporal fusion module
│   ├── tfco.py                   # Main TFCO class
│   ├── models.py                 # Model factory & architectures
│   ├── configs/
│   │   └── model_config.yaml     # Architecture templates
│   └── sub_models/               # Neural network components
│       ├── raw_traffic_encoder.py
│       ├── temporal_networks.py
│       ├── traffic_decoder.py
│       ├── image_encoders.py
│       ├── image_decoders.py
│       ├── conv_lstm.py
│       └── video_encoders.py
├── tools/
│   └── convert_routes.py         # Route file ID conversion
├── examples/
│   ├── fco_example.py            # FCO-only inference
│   ├── tfco_example.py           # FCO + TFCO inference
│   └── inference_example.py      # Full TFCO example with metrics
└── requirements.txt

---

2. FCO (Floating Car Observer)

FCO performs vehicle detection from V2X-equipped vehicles in SUMO. A configurable fraction of vehicles (penetration rate) are designated as FCOs. Each FCO detects nearby traffic participants, and detections from multiple FCOs are fused into a unified set.

Basic Usage

from fco.fco.fco import FCO
from fco.fco.utils.sumo_utils import configure_sumo
import libsumo as traci

# Create FCO
fco = FCO(
    detector_mode='2d-raytracing',
    collapser_mode='geometric',
    batch_size=256,
    track_performance=True,
)

# Start SUMO simulation
sumo_cmd = configure_sumo('path/to/simulation.sumocfg', gui=False)
traci.start(sumo_cmd)

# Initialize V2X - 20% of vehicles become FCOs
fco.reset(penetration_rate=0.2)

# Simulation loop
while traci.simulation.getMinExpectedNumber() > 0:
    traci.simulationStep()

    # Fused detections from all FCOs
    detections = fco.detect(collapse=True)

    for det_id, det_info in detections.items():
        x, y = det_info['position']
        angle = det_info['angle']
        confidence = det_info['confidence']
        entity_type = det_info['entity_type']  # 'vehicle', 'bicycle', 'pedestrian'

traci.close()
fco.print_stats()

Constructor Parameters

FCO(
    detector_mode='emulation',       # Detection algorithm (see Detector Modes)
    model_path=None,                 # Path to trained model (emulation mode only)
    building_polygons=None,          # Building polygon file/list (raytracing modes)
    batch_size=256,                  # Batch size for neural network detectors
    collapser_mode='ground_truth',   # Detection fusion mode (see Collapser Modes)
    association_threshold=9.21,      # Chi-squared threshold for association (9.21 = 95% CI for 2D)
    min_confidence=0.3,              # Minimum detection confidence
    gating_distance=5.0,             # Max distance (m) for detection association
    track_performance=True,          # Enable timing statistics
    history_size=1000,               # Rolling window for performance metrics
)

Detector Modes

Mode	Description	Requirements	Speed
'distance'	Detects all entities within a radius. No occlusion.	None	Fastest
'2d-raytracing'	2D ray-based line-of-sight detection with building occlusion.	building_polygons	Fast
'3d-raytracing'	3D computer-vision detection with depth rendering.	open3d, 3D meshes	Slow
'emulation'	Trained neural network that emulates raytracing detection.	model_path, building_polygons	Fast (GPU)

Distance detector -- simplest baseline, detects everything within max_distance meters. No occlusion modeling.

2D raytracing -- casts rays from each FCO and checks intersection with vehicle and building polygons. The num_rays parameter (default 360) controls angular resolution. Use fast=True (default) for vectorized numpy operations. Requires building polygons as a list of Shapely Polygon objects.

Emulation detector -- a trained CNN that predicts detection results from a pre-rendered BEV scene. Supports three target types: binary (detected/not), iou (intersection-over-union score), and bbx (bounding box with position, IoU, and angle predictions). Requires a trained model directory.

Collapser Modes

When multiple FCOs detect the same vehicle, the collapser fuses these detections:

Mode	Description	Use case
'ground_truth'	Groups by SUMO vehicle ID	Simulation benchmarks (fast, exact)
'geometric'	Mahalanobis distance clustering + Covariance Intersection fusion	Realistic deployment scenarios

The geometric collapser uses:

• Association: Mahalanobis distance with gating
• Position fusion: Covariance Intersection (CI)
• Angle fusion: Circular mean with 360-degree wrapping
• Confidence: Geometric mean with multi-detection boost

Detection Output Format

{
    'det_0': {
        'position': (x, y),             # World coordinates (meters)
        'angle': 90.0,                  # Heading in degrees (0=North, clockwise)
        'vtype': 'vehicle',             # Type string from SUMO ID prefix
        'entity_type': 'vehicle',       # Normalized: 'vehicle', 'bicycle', or 'pedestrian'
        'width': 1.8,                   # Meters
        'length': 4.5,                  # Meters
        'distance': 35.2,              # Distance from detecting FCO (meters)
        'confidence': 0.95,            # Detection confidence (0.0-1.0)
        'num_detections': 3,           # Number of FCOs that detected this entity (collapsed only)
        'position_uncertainty': 0.5,   # Position uncertainty (collapsed only)
    },
}

V2X Penetration Rate

The penetration_rate parameter in fco.reset() controls what fraction of vehicles are FCOs:

• 0.1 = 10% of vehicles are V2X-equipped
• 0.2 = 20% (default)
• 1.0 = all vehicles are FCOs

New vehicles entering the simulation are randomly assigned V2X status according to this rate.

Visualization

FCO provides three BEV visualization methods:

# All FCOs (yellow) + collapsed detections (green)
fco.visualize(save_path='bev.png', image_size=1024, radius=200.0)

# Single FCO centered with its individual detections
fco.visualize_single(fco='vehicle_42', image_size=1024, radius=100.0)

# All FCOs with per-FCO detections (color-coded, uncollapsed)
fco.visualize_uncollapsed(save_path='bev_uncollapsed.png')

Running the Example

python -m fco.examples.fco_example \
    --sumo_file path/to/simulation.sumocfg \
    --detector_mode 2d-raytracing \
    --fco_penetration 0.2 \
    --max_steps 3600

---

3. TFCO (Temporal Floating Car Observer)

TFCO adds a temporal neural network on top of FCO detections. It maintains a sliding window of past FCO detections and uses a trained model to predict the full set of vehicles -- including those currently occluded.

Basic Usage

from fco.tfco.tfco import TFCO
from fco.fco.utils.sumo_utils import configure_sumo
import libsumo as traci

# Create TFCO (FCO is auto-created from the model's stored config)
tfco = TFCO(
    model_path='path/to/trained_model/model_epoch_50.pth',
    fco_kwargs={'batch_size': 256, 'track_performance': True},
    device='cuda',
    track_performance=True,
)

# Start SUMO
sumo_cmd = configure_sumo('path/to/simulation.sumocfg', gui=False)
traci.start(sumo_cmd)
tfco.reset(penetration_rate=0.2)

# Simulation loop
while traci.simulation.getMinExpectedNumber() > 0:
    traci.simulationStep()

    fco_detections, tfco_predictions = tfco.detect()

    # TFCO returns None until it has enough temporal history
    if tfco_predictions is not None:
        for pred in tfco_predictions:
            x, y = pred['position']
            confidence = pred['confidence']
            is_visible = pred['is_visible']  # True = seen by FCO, False = recovered by TFCO

traci.close()
tfco.print_stats()

Constructor Parameters

TFCO(
    model_path,                   # Path to .pth checkpoint (required)
    fco=None,                     # Pass an existing FCO instance...
    fco_kwargs=None,              # ...or pass kwargs to create one automatically
    device='cuda',                # 'cuda' or 'cpu'
    track_performance=True,       # Enable timing statistics
    history_size=1000,            # Rolling window for performance metrics
)

You must provide either fco (an existing FCO instance) or fco_kwargs (a dict of arguments to create one). When using fco_kwargs, TFCO reads the model's stored dataset metadata to automatically configure the detector mode, building polygons, and collapser -- ensuring training/inference consistency.

How It Works

1. FCO detection -- runs fco.detect(collapse=True) to get fused detections
2. History accumulation -- appends detections to a temporal buffer (deque of length sequence_len)
3. Neural network prediction -- once the buffer is full, feeds the detection sequence through the trained model
4. Visible vehicle merging -- if predict_retained=False, merges currently-visible FCO detections with recovered predictions

The model won't produce predictions until it has accumulated sequence_len frames of history. Check tfco.can_predict to know when predictions are available.

Auto-Loaded Configuration

TFCO automatically loads the following from the model directory:

File	Contents
config.yaml	Training config: sequence length, target type, feature flags, center point, radius, etc.
network_configs.yaml	Architecture-specific hyperparameters (embed_dim, num_heads, layers, etc.)
dataset_metas.yaml	Dataset info: detector mode, building polygons, emulation model path

This means you typically only need to provide the model_path -- everything else is configured automatically.

Model Architectures

Architecture	Input types	Best for
SpatioTemporalTransformer	detections, BEV	General purpose (default)
MaskedSequenceTransformer	detections	Matched/tracked datasets
ConvLSTMModel	BEV	BEV sequences
ThreeDCNNModel	BEV	BEV sequences
TwoDCNNModel	BEV	BEV sequences
ViTModel	BEV	BEV with Vision Transformer encoder

The architecture is selected automatically from the model's config.yaml.

Target Types

Target	Output	Description
'positions'	List[Dict]	Per-vehicle position predictions with confidence
'bev'	torch.Tensor	Bird's-eye view occupancy map

Feature Flags

These are set during training and loaded automatically. They control what information is included in the input to the neural network:

Flag	Features added
add_angle	sin/cos encoded heading
add_dimensions	Normalized length and width
add_type	One-hot vehicle type (car, truck, other)
add_confidence	Detection confidence score
add_uncertainty	Position uncertainty from collapser
add_num_fcos	Number of FCOs that detected this vehicle

TFCO Prediction Output Format

For target='positions':

{
    'position': (x, y),               # World coordinates (meters)
    'confidence': 0.87,               # Prediction confidence (0.0-1.0)
    'is_visible': False,              # True = from FCO, False = recovered by model
    'type': 'car',                    # 'car', 'bike', or 'pedestrian' (if predict_type=True)
    'type_confidence': 0.92,          # Type prediction confidence
    'type_probabilities': {           # Full probability distribution
        'car': 0.92,
        'bike': 0.05,
        'pedestrian': 0.03
    }
}

When predict_retained=False, the output contains two kinds of predictions:

• Recovered vehicles (is_visible=False): predicted by the model, not currently seen by any FCO
• Visible vehicles (is_visible=True): directly observed by FCO, merged in with confidence 1.0

Key Properties

Property	Type	Description
tfco.can_predict	bool	Whether enough history has been accumulated
tfco.center_point	(float, float)	Center of observation area
tfco.radius	float	Observation radius in meters
tfco.sequence_len	int	Required temporal history length
tfco.target_type	str	'positions' or 'bev'
tfco.num_fcos	int	Number of active FCOs
tfco.detection_history	deque	Current temporal buffer

Trained Model Directory Structure

Each trained model directory should contain:

trained_model/
├── config.yaml              # Training configuration
├── network_configs.yaml     # Architecture hyperparameters
└── model_epoch_50.pth       # Model weights checkpoint

Running the Example

python -m fco.examples.tfco_example \
    --model_path path/to/trained_model/model_epoch_50.pth \
    --sumo_file path/to/simulation.sumocfg \
    --fco_penetration 0.2 \
    --device cuda