README.md

---
library_name: pytorch
pipeline_tag: robotics
tags:
  - autonomous-driving
  - vision-language-action
  - cooperative-driving
  - CARLA
---

# Coop-SimLingo

Coop-SimLingo is the hint-enabled SimLingo baseline released with
[CoDrive](https://github.com/YongshuoLiu/CoDrive), a closed-loop benchmark for
temporally grounded cooperative vision-language driving.

## Files

- `coop_simlingo_fp32.pt`: model-only FP32 PyTorch state dictionary.
- `coop_simlingo_inference.yaml`: frozen inference architecture.
- `model_metadata.json`: format and SHA256 metadata.
- `SHA256SUMS`: integrity checksums.

## Usage

Clone CoDrive, create the pinned environment, and download this model through
its helper:

```bash
git clone https://github.com/YongshuoLiu/CoDrive.git
cd CoDrive
conda env create -f environment-simlingo.yaml
conda activate simlingo
python -m pip install -e . --no-deps
bash codrive.sh setup
```

For one-image inference without CARLA:

```bash
bash codrive.sh predict \
  --image front_1024x512.jpg \
  --speed 6.2 \
  --target 15.0,0.2 \
  --next-target 30.0,0.4 \
  --scenario S1 \
  --front-car-pos 18.0,-1.2
```

A complete, runnable S1 record with the front image, numeric inputs, inference
script, and `best-005` reference output is available in the
[CoDrive example directory](https://github.com/YongshuoLiu/CoDrive/tree/main/examples/s1_pedestrian_emergence):

```bash
bash codrive.sh example --device cuda:0
```

The public Python API is `codrive.CoopSimLingoRuntime`. S1-S5 use the
`<FRONT_CAR_POS>` token and its numeric ego-frame position. The canonical S6
cross-traffic warning is non-positional.

## Inputs and cooperative hints

The model requires a rectified `1024x512` front image with `110-degree`
horizontal FoV, current speed in m/s, and two ego-frame navigation target
points. The optional canonical hints are:

| ID | Exact hint |
| --- | --- |
| S1 | `The pedestrian at position <FRONT_CAR_POS> is about to enter the ego lane.` |
| S2 | `The vehicle at position <FRONT_CAR_POS> will cut into the ego lane soon.` |
| S3 | `The static object at position <FRONT_CAR_POS> is blocking the ego lane.` |
| S4 | `A wrong-way vehicle at position <FRONT_CAR_POS> will cross the ego left-turn path. Yield before turning left.` |
| S5 | `A wrong-way vehicle at position <FRONT_CAR_POS> will cross the ego right-turn path. Yield before turning right.` |
| S6 | `A vehicle approaching from the left will cross the ego right-turn path. Yield before turning right.` |

S1-S5 additionally require the referenced `[forward, right]` ego-frame
position in meters. Positive values mean forward/right; negative lateral
values mean left. For example, `[18.0, -3.5]` is 18 m ahead and 3.5 m left.
The value is not a pixel, GPS/world coordinate, or distance-bearing pair.
`<FRONT_CAR_POS>` is a hazard-anchor token and may refer to an actor or
expected conflict point, not necessarily a front car. The special token must
not be replaced with coordinate text; the numeric value is supplied separately
through the CoDrive API.

Current speed is a non-negative scalar in meters per second, not km/h. The
camera, speed, navigation targets, warning, and hazard position must describe
the same timestamp. Full details and command examples are provided in
`docs/inference_interface.md` in the CoDrive repository.

## Scope

The model predicts future waypoints for recorded data, vehicle integration,
and CARLA evaluation. It is not a physical-vehicle controller. Deployment
requires independent localization, camera calibration, V2X message handling,
waypoint control, collision monitoring, watchdogs, and emergency stopping.
Follow `docs/real_vehicle.md` in the CoDrive repository.

## Base model

The architecture uses SimLingo with `OpenGVLab/InternVL2-1B`. Users must comply
with the licenses and terms of the upstream SimLingo and InternVL2 components.