mlx-community

/

Meta-Llama-3-8B-Instruct-4bit

like 81

Follow

MLX Community 13k

Instruct models don't have this issue.

It's only the base models.

I have the same issue.
when setting the max_tokens to 1024 the model doesn't stop generating.

Please give me a reproducible example :)

hi,
As suggested in the model card, but with max_tokens=1024-

'''
from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Meta-Llama-3-8B-Instruct-4bit")
response = generate(model, tokenizer, prompt="hello", verbose=True, max_tokens=1024)
'''

The model does not stop generating text, as shown in the next discussion: huggingface.co/mlx-community/Meta-Llama-3-8B-Instruct-4bit/discussions/3

Hey @prince-canuma

Here's the sample code:

from mlx_lm import load, generate
from markdown import markdown
from IPython.display import Markdown, display

model, tokenizer = load("mlx-community/Meta-Llama-3-8B-Instruct-4bit")

response = generate(model, tokenizer, 
                prompt="What is 5 plus 5?",
                verbose=True,
                max_tokens=400)

And attached is the response it's giving me.

Can you run the same command using the terminal and share the results?

python -m mlx_lm.generate --model ... --prompt "..."

Should setting eos_token to "<|end_of_text|>" in configuration files do the trick? It doesn't work for me unfortunately.