kutomanovhennadii/iec-gpt-4o-mini-2024-07-18
Viewer • Updated • 2.96k • 183 • 1
Hennadii Kutomanov PRO
kutomanovhennadii
AI & ML interests
Independent researcher. IEC studies the behavior and degradation of language models under controlled load. Code: github.com/kutomanovhennadii/instruction-capacity
Recent Activity
new activity about 2 hours ago
kutomanovhennadii/iec-gpt-4o-mini-2024-07-18:[bot] Conversion to Parquet repliedto their post about 2 hours ago
A model’s context window can hold hundreds of thousands of tokens. But how many instructions can it actually follow at once?
Developers working with AI agents run into this problem all the time. You add rules to CLAUDE.md, and the agent starts ignoring some of the earlier ones. As the conversation grows longer, the model forgets the original requirements. You connect dozens of tools, and the quality of its work drops, even if many of those tools are never used.
In the IFScale study, models were given up to 500 instructions. Even the top-performing models followed only about 68% of them.
We have learned to work around these limitations: shortening rule files, clearing the context, loading only the tools we need. But these are still largely rules of thumb.
Yet there is a fundamental gap.
We measure model speed, token cost, reasoning performance, and context window size. But we have no generally accepted way to measure how reliably a model follows instructions as the load increases.
We cannot say how much additional load a particular model can handle. We cannot reliably compare two models in terms of how their performance degrades under load. We cannot estimate in advance whether a prompt will be too demanding.
What if we treated degradation not as an annoying quirk of LLMs, but as a measurable property? What if we could express load in defined units, record the outcome of each instruction, identify the laws governing performance degradation, and determine a model’s limit with a measurement uncertainty?
That is what my new article is about: why existing benchmarks are not enough, what it takes to measure these limits, and how we approached the problem.
At the end, I introduce IEC, an open-source experimental instrument we built to study model behavior under controlled load.
https://huggingface.co/blog/kutomanovhennadii/why-measure-a-models-behavior-under-load
Originally written in Russian. English translation assisted by AI.
repliedto their post about 14 hours ago
A model’s context window can hold hundreds of thousands of tokens. But how many instructions can it actually follow at once?
Developers working with AI agents run into this problem all the time. You add rules to CLAUDE.md, and the agent starts ignoring some of the earlier ones. As the conversation grows longer, the model forgets the original requirements. You connect dozens of tools, and the quality of its work drops, even if many of those tools are never used.
In the IFScale study, models were given up to 500 instructions. Even the top-performing models followed only about 68% of them.
We have learned to work around these limitations: shortening rule files, clearing the context, loading only the tools we need. But these are still largely rules of thumb.
Yet there is a fundamental gap.
We measure model speed, token cost, reasoning performance, and context window size. But we have no generally accepted way to measure how reliably a model follows instructions as the load increases.
We cannot say how much additional load a particular model can handle. We cannot reliably compare two models in terms of how their performance degrades under load. We cannot estimate in advance whether a prompt will be too demanding.
What if we treated degradation not as an annoying quirk of LLMs, but as a measurable property? What if we could express load in defined units, record the outcome of each instruction, identify the laws governing performance degradation, and determine a model’s limit with a measurement uncertainty?
That is what my new article is about: why existing benchmarks are not enough, what it takes to measure these limits, and how we approached the problem.
At the end, I introduce IEC, an open-source experimental instrument we built to study model behavior under controlled load.
https://huggingface.co/blog/kutomanovhennadii/why-measure-a-models-behavior-under-load
Originally written in Russian. English translation assisted by AI.
Organizations
None yet