Update README.md

README.md

@@ -28,6 +28,16 @@ Run them in your choice of tools:
 
 Note: if it's a newly supported model, you may need to wait for an update from the developers.
 
+## Warning for CUDA users
+
+If running with CUDA on a model that has `Q4_K` or `Q5_K` for `61.blk.ffn_down_exps` you should compile with the following flag:
+
+`-DGGML_CUDA_FORCE_CUBLAS=ON`
+
+to avoid `NaN` activations. This is most likely an issue in `llama.cpp` itself and will hopefully be resolved, but in the meantime this fixes the issue.
+
+You can check in the info on huggingface which models are affected, but it seems to be `Q5_K_S`, `Q4_K_S`, `Q3_K_XL`, `Q3_K_L`, `Q3_K_M`, `IQ3_M`, `IQ3_XS`, and `IQ3_XXS`
+
 ## Prompt format
 
 ```