--- language: - en license: mit library_name: stable-baselines3 pipeline_tag: reinforcement-learning tags: - robotics - control - reinforcement-learning - sac - ur7e - simulation - cpu - numpy model-index: - name: UR7e SAC Line Following — Kinematic Simulation results: [] --- # UR7e SAC Line Following — Kinematic Simulation A frozen Soft Actor-Critic policy by **Dev Kumar** for a six-axis UR7e-labelled kinematic environment. This release makes an existing trained checkpoint reusable: original weights, a portable NumPy actor, complete environment files, seeded evaluation against a classical controller, and an interactive browser lab. - **Source:** [UR7e Reinforcement Learning Suite](https://github.com/devkumar-projects/UR7e-Reinforcement-Learning-Suite), commit `fbba2e4ef768219b7f925c091ff027d025883500`. - **Interactive lab:** [UR7e Policy Lab](https://huggingface.co/spaces/devkumarhf/UR7e-Policy-Lab). - **Experimental analysis:** [Kaggle benchmark](https://www.kaggle.com/code/devkumarkaggle/ur7e-sac-vs-dls-reproducible-policy-benchmark). - **Original training:** 1,900,000 steps in checkpoint metadata; SAC, two 512-unit ReLU layers. No training was performed for this release. - **Inputs / outputs:** 29 float32 observations → 6 normalized joint increments; ±1 maps to ±0.05 radians per environment step. ## Measured evaluation 600 CPU episodes: **100 prespecified trajectories** (20 for each of five shapes), two controllers, three sensor-noise conditions. Conditions reuse the same trajectories; they are not 600 independent trajectories. Both controllers use the same sensor observations, joint limits, starting pose and termination rules. DLS uses fixed damping 0.02 and gain 0.8; it was not optimized against the test results. | Sensor noise σ | Controller | Strict completion | Mean tracking RMS (mm)* | |---|---|---:|---:| | 0 mm / mrad | SAC | 93/100 | 5.21 | | 0 mm / mrad | DLS | 90/100 | 2.38 | | 2 mm / mrad | SAC | 92/100 | 5.75 | | 2 mm / mrad | DLS | 89/100 | 3.55 | | 5 mm / mrad | SAC | 89/100 | 7.56 | | 5 mm / mrad | DLS | 87/100 | 6.50 | *Mean of per-episode RMS values over acquired episodes, including failures. Position noise is in metres and joint-angle noise in radians: the 2 condition uses 0.002 m / 0.002 rad; the 5 condition uses 0.005 m / 0.005 rad. Known target points are not perturbed. Noise is independent Gaussian per observed step. This is an illustrative sensor perturbation, not a calibrated hardware noise model. ![Measured comparison](evaluation/comparison.png) The observed SAC completion rate is slightly higher for this fixed suite; DLS has lower tracking error. The experiment does not establish statistical superiority, optimality of either method or physical-robot performance. Wilson intervals are illustrative per-condition binomial intervals, not training-seed uncertainty or a formal stratified-sampling analysis. ## Completion audit The upstream `completed` flag is insufficient for evaluation: a stagnation stop can set it true, and cursor advancement can terminate before the final point is within tolerance. This release preserves the original transition dynamics, but reports **strict completion** only when the episode terminates in tracking phase, reaches the final cursor and the actual endpoint is within 25 mm, with no tracking error above 60 mm. Raw flags, stop reasons, steps and endpoint errors are retained. Inspect `evaluation/episodes.csv`, `summary.json`, `test-manifest.json` and `test_release.py`. The original repository's historical 97% figure uses a different evaluation. It is not substituted for the measured results above. The original training trajectory seed history is unavailable; exact disjointness from training cannot be certified. New evaluation trajectories and controller settings were fixed before the benchmark. ## Use on CPU without PyTorch Download this repository's `policy.py` and `actor.npz`, then: ```python import numpy as np from policy import Policy policy = Policy() # Supply a real 29-element observation constructed by the supplied environment. action = policy.predict(observation) ``` Observation order: normalized joint angles (6), end-effector position in metres (3), cursor/60 progress (1), approach/tracking phase (1), current target (3), five lookahead target points (15). Angles are divided by the respective positive joint limit. The actor does not normalize Cartesian coordinates. `actor.npz` is loaded with `allow_pickle=False`; export parity against the original deterministic policy is recorded in `evaluation/actor-parity.json` (343 observations, maximum absolute action difference below 0.000004). ## Reproduce the full evaluation ```bash python -m venv .venv # Activate the virtual environment for your operating system. pip install -r requirements.txt OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 python export_actor.py OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 python test_release.py OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 python evaluate.py ``` Install the CPU build of PyTorch if required by your platform. The measured run used CPU throughout; no paid inference service is involved. The original SB3 archive contains serialized Python training metadata; use `checkpoint.load_model()` for the original trusted artifact, or the NumPy actor for inference without that deserialization. `checkpoint.py` reconstructs documented spaces and discards unused saved training observations for NumPy-version portability. ## Scope and limitations This is a **position-only kinematic simulation policy**. The default URDF is not calibrated to an individual robot. There is no dynamic, torque, contact-force, tool-orientation, collision or real-time safety model, and no validated sim-to-real transfer. Inference timing in `summary.json` describes this CPU run, not a robot control-rate guarantee. The exported actor is the deterministic mean action; it does not reproduce stochastic SAC sampling or resume training by itself. A physical controller requires separate integration and validation. ## Attribution and licences Original project, policy and release: Dev Kumar; original MIT notice is retained in `LICENSE`. The generated robot description identifies upstream Universal Robots `ur_description`; its comments, licence and provenance are retained in `THIRD_PARTY_NOTICES.md` and `UR-DESCRIPTION-LICENSE`. No meshes are redistributed. The benchmark and inference code remain inspectable, and no paper or peer-review status is claimed by this model release. [Portfolio](https://dev-kumar.com/) · [GitHub](https://github.com/devkumar-projects) · [Kaggle](https://www.kaggle.com/devkumarkaggle) · [ORCID](https://orcid.org/0009-0008-2279-9163) · [HAL research record](https://cv.hal.science/dev-kumar) ## Full research report [Reinforcement Learning for UR7e Trajectory Following and UR5e Manipulation (PDF)](https://dev-kumar.com/reports/ur7e-ur5e-reinforcement-learning.pdf). Authors: Hugo Simon, Dev Kumar, Basile Kulik and Kevin Belin. The manuscript is separate from the model/data release and retains its own distribution terms. Evaluation scope and limitations for this resource remain as documented above.