Keep each thread's adapter gate to itself

#2
by subin - opened
vLLM Semantic Router org

The adapter gate in vela3_adapters.py was one object for the whole process. active() set its adapter name and row mask, and every MultiLoRA.forward read them, so when two threads called system_one on the same model, one thread's passes ran with the other's adapter. On vela3-edge on CPU, with one thread asking span questions in a loop and another asking the same refund question 25 times, all 25 answers came back wrong: 0.736 to 0.820 against 0.956 when asked alone.

The gate is now a threading.local, so each thread sees only the adapter it switched on, and a new thread starts with every adapter off. Nothing else changes: active(), MultiLoRA.forward and the graph check in modeling_vela3.py read the gate as before, now on the calling thread.

Tested on vela3-edge on CPU. In the same two-thread run, 0 of 25 answers now differ from the answer asked alone. One request each over three states, with the nine router questions, a hazard question, an ordinary choice and yes/no question and three span questions (PII, relation, entities), gives byte-identical JSON before and after. The change is the same in all six sizes.

vLLM Semantic Router org

Thanks, merged.

adaamko changed pull request status to merged

Sign up or log in to comment