Update README.md
#6
by Myric - opened
README.md
CHANGED
|
@@ -77,3 +77,27 @@ cd llama.cpp && cmake -B build && cmake --build build -j
|
|
| 77 |
> contract this draft model needs, and upstream PR
|
| 78 |
> [ggml-org/llama.cpp#25165](https://github.com/ggml-org/llama.cpp/pull/25165)
|
| 79 |
> covers the target architecture only.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 77 |
> contract this draft model needs, and upstream PR
|
| 78 |
> [ggml-org/llama.cpp#25165](https://github.com/ggml-org/llama.cpp/pull/25165)
|
| 79 |
> covers the target architecture only.
|
| 80 |
+
|
| 81 |
+
### Production experience
|
| 82 |
+
I ran the i-quality version of this on my dgx-spark with a llama.cpp backend and a nice large context and
|
| 83 |
+
hooked it up to Opencode. I had it read the source of a moderately complex (19M of text) golang project I've
|
| 84 |
+
been working on. It took a while to read and summarize the entire project into context. I'd say it meandered
|
| 85 |
+
a little bit as I watched the reasoning process, but it inferred some pretty subtle details.
|
| 86 |
+
|
| 87 |
+
I then gave it a task to create a pair of systemd --user unit files for a pair of new models I had generated.
|
| 88 |
+
It needed to create the files in the proper format to be discovered by my parser, which was buried in the code.
|
| 89 |
+
It had to decide proper context sizes and model tuning parameters based on the model size and my available
|
| 90 |
+
system parameters. It figured out my system architecture autonomously, including the unified memory and cuda
|
| 91 |
+
architecture. It made educated guesses about the drafter settings (and got them right).
|
| 92 |
+
|
| 93 |
+
It determined that naming conventions and picked sane defaults for both models. It crafted and placed the
|
| 94 |
+
two units, ran the proper system tools to reread the units, ran the systemd verify command to check that the
|
| 95 |
+
format was correctly parsed by systemd.
|
| 96 |
+
|
| 97 |
+
It then wrote a little parser to double-check that the systemd units it had written parsed correctly and fixed
|
| 98 |
+
a bug it found in its parsing string. Once all that was done, it summarized all this for me and reported success.
|
| 99 |
+
If anything I would have preferred it asked for more clarification, but it completed the entire task autonomously.
|
| 100 |
+
Everything worked on the first try.
|
| 101 |
+
|
| 102 |
+
This is a capable large-context model natively trained to 1M tokens of context. I think this is the model I'll
|
| 103 |
+
use when I want to do something hard on my local machine that the fast little models get stuck.
|