I tried it compared to my flash next it seems like a better model so far 32gb vram + 64 gb ram

#7
by saifvj - opened

the vision model from flash next strata engine is working perfectly on the edge fork.

one thing is the speed its slower than flash next unsloth iq3xxs the new model - strata i normally get 90 t/s decode cold and 120 t/s decode warm with flash next on strata
on this one i am getting 40 t/s cold and 60 t/s warm decode

prefill on flash next strata is 8192 in 4s
prefill on edge is 8192 in 6s

even with more experts on vram compared to flash next its a bit slower am i doing something wrong here?

although i felt like its better quality in a few areas for example this response below - flash next would have just gone ahead with whatever it thought work but edge replied as below

prompt - can you generate a fresh character for me

Response - Happy to generate a fresh character! I just need a bit of direction since "fresh character" is pretty open — is this for the Second Cup project (the chai stall story), or something entirely new? And what kind of character are you envisioning?

What kind of character do you want? Give me a rough idea — age, gender, role, vibe, setting — and whether you want a full multi-panel character sheet or just a single portrait.

Full character sheet (multi-panel) for a new character — I'll give you details
Single portrait of a new character — I'll describe them
A new character for the Second Cup story
Surprise me — something fresh and interesting

saifvj changed discussion title from I tried it compared to my flash next it seems like a better model so far 5090 + 64 gb ram to I tried it compared to my flash next it seems like a better model so far 32gb vram + 64 gb ram

Also huihui refusal removal vectors are working on this model too so if you just apply them model becomes uncensored straight away

Edge0 org

Thanks — and yes, the vision encoder works on the Edge0 fork because it is the same encoder; the repos now ship it as mmproj-BF16.gguf so you no longer need to borrow it from the base repo.

On speed: two flags in the shipped model.json are tuned for our test box and you should drop them: --pool-workers 24 (default 0 = every physical core minus one; 24 oversubscribes a smaller CPU) and --pcie-frac 0.1 (the share of expert work on the CPU lane; leave it out and a single-GPU serve refines it in the first decode windows). Also try the Q3_K folder rather than NVFP4: on a 32 GB card part of the expert pool is read from host RAM either way, and NVFP4's CPU-side expert decode is slower than the Q3_K/IQ4_NL path (about 3.3 vs 5.8 GiB/s in our measurements). Our reference on a 24 GB 5090 D is 121.2 agent / 99.7 chat for Q3_K and 98.3 / 79.6 for NVFP4.

Could you paste the line expert cache auto: ... GiB free -> N slots and the spec acceptance from your log, plus your CPU and how many cores? That decides whether the rest is the machine or something we should fix.

my cpu is Intel Core i7-11700K, 8 cores and 16 threads.
i upped the --spec from 4 to 6 so i started getting between 50 - 70 t/s
I don’t set --pool-workers or --pcie-frac in my config, so nothing to drop there.

image

and below is the same run on flash next on my pc

image

the flash next model i used is from unsloth and below is its size

image

Edge0 org

That is expected rather than something you set up wrong. The IQ3_XXS build you compare against is a heavier quantisation — it trades precision for speed. Edge0's experts are FP4, which keeps more precision per weight, so the same number of experts moves more bytes per token and decode is slower. The two builds are not the same trade-off.

I am downloading the q3k model and test it along to see how much the speed difference is - according to you what is the quality difference between nvfp4 and q3k

i started with nvfp4 assuming nvfp4 might be more suited for my 5090 32 gb card since it fits and it does work honestly 60-70 t/s is not a bad speed if its better quality but that depends on how much better it is compared to q3k

q3k definitely made it faster on the prefill and some improvements on the decode that went up to 85 t/s it starts fast but stabilizes at around 85 t/s
image

btw one small issue when i put the expert cache to a manual number it gives me an oom error even if the manual number is less than the auto number. for example below i changed expert cache to 9500 while auto picks 9852 and it works but 9500 manually doesnt work and crashes

Traceback (most recent call last):
File "I:\Strata-main\Strata-edge0\serve\server.py", line 9711, in
sys.exit(main())
^^^^^^
File "I:\Strata-main\Strata-edge0\serve\server.py", line 9396, in main
engine = StrataEngine(exe, engine_args(cfg) + (effort_end or []), cwd=cfg.get("cwd"), log=cfg.get("log"),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "I:\Strata-main\Strata-edge0\serve\server.py", line 1152, in init
raise EngineDied(why + (f" (see {log})" if log else "") + note +
EngineDied: the engine exited before it was ready (see I:/Strata-main/Strata-edge0/edge0-145b-q3k.log)
the engine process ended: exit code 1
the engine log's last lines:
FileExpertSource: allocating 28.49 GiB page-locked cache complement
FileExpertSource: before complement allocation: requested 28.49 GiB, available RAM 47.03 GiB, available commit 73.13 GiB (limit 117.84 GiB), available virtual address space 130894.11 GiB
FileExpertSource: copied cache complement through layer 8/48 (6.54 GiB, 2 s)
FileExpertSource: copied cache complement through layer 16/48 (10.90 GiB, 4 s)
FileExpertSource: copied cache complement through layer 24/48 (14.96 GiB, 5 s)
FileExpertSource: copied cache complement through layer 32/48 (21.18 GiB, 7 s)
FileExpertSource: copied cache complement through layer 40/48 (25.24 GiB, 8 s)
FileExpertSource: copied cache complement through layer 48/48 (28.49 GiB, 10 s)
FileExpertSource: mapped pinned cache complement ready: resident 28.49 GiB, pinned 28.49 GiB in 13 s; page-locked and mapped
FileExpertSource: 3684 of the prompt path's 3684 lendable slots keep their experts in RAM too
strata generate: resident RAM mode: 28.49 GiB of experts in RAM (page-locked), 9500 in the GPU cache; adaptive swaps exchange them with the GPU cache (no file reads)
strata generate: the file tier reads unbuffered (STRATA_UNBUFFERED_LOAD=1)
strata generate: the mapped view of the experts is closed (every expert outside RAM and VRAM is read unbuffered)
strata serve: Windows commit: 44.4 GiB left of a 117.8 GiB limit (RAM + page file)
strata: prefill plan: chunk 8192, ring 96 kept: file set 0.3 GiB does not fit the page cache 0.0 GiB -> about 1.5 GB read from the drive per 32768-token prompt (4 chunks); no option does better (the set cannot fit: fewest passes win)
strata: prefill plan inputs: 37.3 GiB experts, 14.4 GiB in the GPU caches, 28.5 GiB RAM copy (3684 lend slots kept), file tier 0.0 GiB; 0.0 GiB available for the page cache (the file tier reads unbuffered: none); PCIe 25.4 GB/s; drive 1.50 GB/s (assumed); a 32768-token prompt
strata serve: prompt chunk auto: 8192 tokens, a 96-slot ring
strata serve: the prompt path borrows 3684 CUDA0 cache slots (5.59 GiB)
strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata serve: verify: mapped staging allocation failed [pinned-memory-exhausted] the OS gave no more page-locked memory
Press any key to continue . . .

no worries i got opus to create a fork of your engine that fixed the manual experts issue and here is the problem that existed

“With --resident-budget-gib set, every expert slot taken away from the GPU cache is added to the page-locked RAM copy, so a manual --expert-cache below what auto picks pinned too much memory and the start failed with pinned-memory-exhausted. Auto avoided this only by luck, since its slot count shifts from start to start with the free-VRAM reading, and it could fail the same way on a bad start. We made three changes. The engine now takes its pin reserve around the resident copy (it never did on this path), measures how much page-locked memory the driver will actually give and caps the pinned copy a little under that, and treats a manual --expert-cache as a ceiling. A manual setting now gets the same reserve terms and the same post-write check and shrink that auto uses, and a model load that previously ran only on the auto path now runs on both. With the same slot count, manual and auto now end up with the same cache size, pinned RAM and free VRAM. Values that used to crash (9,500 slots) now start, and manual runs no longer end with 0 MiB of VRAM free.”

Sign up or log in to comment