Not Lain PRO
not-lain
AI & ML interests
custom AI models with HF integration, HuggingFace fellow š¤
Recent Activity
liked
a Space
about 17 hours ago
abidlabs/daggr-3d
reacted
to
alvarobartt's
post
with š
2 days ago
š„ `hf-mem` v0.4.1 now also estimates KV cache memory requirements for any context length and batch size with the `--experimental` flag!
`uvx hf-mem --model-id ... --experimental` will automatically pull the required information from the Hugging Face Hub to include the KV cache estimation, when applicable.
š” Alternatively, you can also set the `--max-model-len`, `--batch-size` and `--kv-cache-dtype` arguments (Ć la vLLM) manually if preferred.
reacted
to
alvarobartt's
post
with š„
2 days ago
š„ `hf-mem` v0.4.1 now also estimates KV cache memory requirements for any context length and batch size with the `--experimental` flag!
`uvx hf-mem --model-id ... --experimental` will automatically pull the required information from the Hugging Face Hub to include the KV cache estimation, when applicable.
š” Alternatively, you can also set the `--max-model-len`, `--batch-size` and `--kv-cache-dtype` arguments (Ć la vLLM) manually if preferred.
Organizations
