spow12/ChatWaifu_2.0_vision

README.md

@@ -5,8 +5,6 @@ language:
 
 # To request a quant, open an new discussion in the Community tab (if possible with the full url somewhere in the title *AND* body)
 
-**We are currently nearing our (humongously large) storage limit on huggingface, and need to decide how to approach this. Most likely, by deleting lots of repositories**
-
 **You can search models, compare and download quants at https://hf.tst.eu/**
 
 **You can see the current quant status at https://hf.tst.eu/status.html**
@@ -118,9 +116,22 @@ uploads are small.
 
 ## How do you create imatrix files for really big models?
 
-By using llama.cpp's RPC mode and distriobuting imatrix computations over multiple hosts. For some older models,
-or in some special cases, I am not above doing the imatrix computation on a quant (mostly Q8_0, but I have been using Q4_K_S
-and lower for some models in the past). We've also streamed some models from nvme.
+Through a combination of these ingenuous tricks:
+
+1. I am not above using a low quant (e.g. Q4_K_S, IQ3_XS or even Q2_K), reducing the size of the model.
+2. An nvme drive is "only" 25-50 times slower than RAM. I lock the first 80GB of the model in RAM, and
+   then stream the remaining data from disk for every iteration.
+3. Patience.
+
+The few evaluations I have suggests that this gives good quality, and my current set-up allows me to
+generate imatrix data for most models in fp16, 70B in Q8_0 and almost everything else in Q4_K_S.
+
+The trick to 3 is not actually having patience, the trick is to automate things to the point where you
+don't have to wait for things normally. For example, if all goes well, quantizing a model requires just
+a single command (or less) for static quants, and for imatrix quants I need to select the source gguf
+and then run another command which handles download/computation/upload. Most of the time, I only have
+to do stuff when things go wrong (which, with llama.cpp being so buggy and hard to use,
+is unfortunately very frequent).
 
 ## What do I need to do to compute imatrix files for large models?
 
@@ -176,11 +187,10 @@ Update 2024-09: just looked at implementing it, and no, the problems that keep m
 ## So who is mradermacher?
 
 Nobody has asked this, but since there are people who really deserve mention, I'll put this here. "mradermacher" is just a
-pseudonymous throwaway account I created to goof around, but then started to quant models in early 2024. A few months later, @nicoboss joined
-and contributed hardware, power and general support - practically all imatrix generations are done on his computer(s).
+pseudonymous throwaway account I created to goof around, but then started to quant models. A few months later, @nicoboss joined
+and contributed hardware, power and general support - practically all imatrix computatuions are done on his computer(s).
 Then @Guilherme34 started to help getting access to models, and @RichardErkhov first gave us the wondrous
-FATLLAMA-1.7T, followed by access to his server to quant more models, likely to atone for his sins. Finally, in 2026, @simonko912 started
-handling model requests and model queuing, greatly reducing workload for the rest iof the team.
+FATLLAMA-1.7T, followed by access to his server to quant more models, likely to atone for his sins.
 
 So you should consider "mradermacher" to be the team name for a fictional character called Michael Radermacher.
-There are no connections to anything else (i.e. other Radermachers) on the internet, other than an mradermacher_hf account on reddit.
+There are no connections ot anything else on the internet, other than an mradermacher_hf account on reddit.