---
license: cc-by-4.0
tags:
  - reinforcement-learning
  - multi-objective
  - production-scheduling
  - attention
  - ppo
  - morl
language:
  - en
pipeline_tag: reinforcement-learning
---

# SmartFlexPack MORL: Sequential Attention Policy for Packaging Scheduling

Trained model weights for the paper:

> **Multi-Objective Reinforcement Learning for Adaptive Packaging Scheduling via Sequential Attention Policy**  
> L. Venturoso, G. Egger, F. Visentin, M. Todescato  
> Fraunhofer Italia Research EFRE1054 Smartflexpack  
> *Manuscript in preparation*

## Model description

`SeqAttentionActorCritic` (12,355 parameters) decides, for each order, between
inline printing and labelling. At each planning event a one-layer multi-head
self-attention encoder (4 heads, d=32) encodes the pending orders; for each
decision a 54→64→64 MLP combines that embedding with ten live decision features
and twelve global features, including the preference vector
`(tardiness, cost, CO2)`, and outputs two action logits and a value.

A single trained policy covers the whole trade-off: the operator sets the
preference at inference time, with no retraining.

## Results

In the reference plant configuration (3 label lines, inline 0.45 s/pack) the
policy exceeds the hypervolume of the full family of 32 realisable reference
controllers by **+0.058** (95% CI [+0.025, +0.095], about 7%), replicated on an
independent seed set (+0.050). The advantage grows with a slower inline line
(12–15% at 0.60 s/pack); with a single label line, and with the fastest inline
line, the policy ties the reference family or falls slightly below it.

## Repository structure

```
ppo/seed_{1..5}/policy_200.pt   # final PPO checkpoints (5 independent seeds)
ppo/seed_{1..5}/metrics.csv     # training metrics
bc/bc_seq_morl.pt               # behavioural-cloning warm start
configs/paper.json              # training configuration
normalisation/                  # observation and reward normalisers
inference/seq_morl/             # minimal code to load the policy
manifest.json                   # source revision and SHA-256 of every file
```

## Inference

```python
import sys, torch
from pathlib import Path
from huggingface_hub import snapshot_download

release = Path(snapshot_download("fhi/smartflexpack-morl"))
sys.path.insert(0, str(release / "inference"))
from seq_morl.policies.attention import SeqAttentionActorCritic

model = SeqAttentionActorCritic()
model.load_state_dict(torch.load(release / "ppo/seed_1/policy_200.pt",
                                 map_location="cpu", weights_only=True))
model.eval()

with torch.no_grad():
    ctx = model.encode_sequence(order_feats)              # [N, 32], pending orders
    logits, value = model.forward_single(ctx[i], extra)   # extra: 10 decision + 12 global
    action = logits.argmax(dim=-1).item()                 # 0 = inline, 1 = label
```

Simulation and evaluation: `experiments/evaluate_policy.py` in the project
repository; it also needs the separate DataGenerator.

## Training

Behavioural-cloning warm start, then 5 independent seeds of 200 PPO updates,
10 parallel episodes of 600 orders, lr=3e-4 (Adam). About 3.8 h CPU per seed.

Training code: `experiments/train_bc_warm_start.py`, `experiments/train_policy.py`

## Citation

```bibtex
@unpublished{venturoso2026morl,
  title  = {Multi-Objective Reinforcement Learning for Adaptive Packaging
            Scheduling via Sequential Attention Policy},
  author = {Venturoso, Leonardo and Egger, Georg and
            Visentin, Francesco and Todescato, Marco},
  year   = {2026},
  note   = {Manuscript in preparation}
}
```

## License

[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)
Fraunhofer Italia Research, EFRE1054 Smartflexpack project
(funded under FESR 2021–27, Priority 1 "Smart" R&I programme).