running SQLCoder on M4 Max
Hi,
I am part way through my MSc project where I want to test some different approaches to prompt creation and knowledge retrieval in text-to-SQL systems.
I plan to test a few open source models and potentially have a Agentic setup to build concise prompt, so for running locally i was going to upgrade my m3 pro to an m4 max. I will have a dodgy internet conneciton for some of the time i will be researching so dont realy want to rely on cloud compute.
I know you say that this model does run on apple silicon but how is the performance on token generation?
Apologies if this is a noddy quesiton
Thanks in advance.
Hi @theglover , you can consider starting with the MLX code we have in our evaluation harness: https://github.com/defog-ai/sql-eval/?tab=readme-ov-file#mlx
We haven't used that code in awhile - you might need to update that based on the current state of the various libraries / dependencies.
Thank you - i will take a look