How do I run it using Oobabooga? I'm getting the following error
llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'step35'
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model 'user_data/models/Step-3.5_Flash/step3p5_flash_Q4_K_S-00001-of-00012.gguf'
srv load_model: failed to load model, 'user_data/models/Step-3.5_Flash/step3p5_flash_Q4_K_S-00001-of-00012.gguf'
srv operator(): operator(): cleaning up before exit...
main: exiting due to model loading error
12:40:32-033723 ERROR Error loading the model with llama.cpp: Server process terminated
unexpectedly with exit code: 1
I downloaded the 12 parts and put them into the models directory, when trying to load it on Oobabooga (I have 128GB of RAM + 24GB VRAM) I'm shown the previous error prompt. Hope someone can help me figure this one out
Currently only their custom fork of llama.cpp has support for the model. It will probably be about a month before oobabooga supports this model.
https://github.com/stepfun-ai/Step-3.5-Flash/tree/main/llama.cpp