Hi!

When the return_full_text parameter is set to False, the full text is still returned in the output.

Yeah this is somewhat expected, BLOOM being so large has to undergo a special deployment mechanism where we essentially recoded some generate functions. Right now that deployment has very limited features.

Issue 2: Incorrect handling of punctuation in generated text

Given the code snipped you shared, can you try running inference("Hello, world") I',m suspecting that you're generated_text doesn't correspond to the comment you shared.

Given the code snipped you shared, can you try running inference("Hello, world") I',m suspecting that you're generated_text doesn't correspond to the comment you shared.

I don't think I understand what you are trying to point out about the second issue.
The inputs and outputs I wrote in the example are just to let understand what I'm trying to convey more clearly and in a shorter form, and are not actual inputs and outputs.
Here you can see an actual input and output:

>>> generated_text = inference("  ..  ?  !  ,")[0]["generated_text"]
>>> generated_text
' .. ? ! ,! ! ! ! ! ! ! ! ! !  '
>>> inference(generated_text)[0]["generated_text"]
'..?!,!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!'

I couldn't reproduce the first example. Ran the following:

from huggingface_hub import InferenceApi

inference = InferenceApi("bigscience/bloom", token =True)
generated_text = inference("Hello")[0]["generated_text"]

print(generated_text) # prints "Hello, I am a young woman of 28 years old who has just arrived in New Braunfels for"

new_generated_text = inference(generated_text)[0]["generated_text"]

print(new_generated_text) # prints "Hello, I am a young woman of 28 years old who has just arrived in New Braunfels for the first time. I like everything but above all a perfect hygiene. I take my time because quality"

But the second one seems to do that:

from huggingface_hub import InferenceApi

inference = InferenceApi("bigscience/bloom", token =True)
generated_text = inference("  ..  ?  !  ,")[0]["generated_text"]

print(generated_text) # prints " .. ? ! ,! ! ! ! ! ! ! ! ! !  "

new_generated_text = inference(generated_text)[0]["generated_text"]

print(new_generated_text) # prints "..?!,!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!"

Thanks for reporting. I'm guessing this has something to do with the inference API somehow. cc @Narsil

Okay just to let you know we've found the issue. It's linked to our tokenization mechanism.

>>> tok = AutoTokenizer.from_pretrained("bigscience/bloom", use_fast = True)
>>> tok.decode(tok.encode("Hello , there"))
'Hello, there'
>>> tok.decode(tok.encode("Hello , there"), clean_up_tokenization_spaces=False)
'Hello , there'

We'll work towards fixing this ASAP!

How is this issue moving forward? I have the same problem, returning full text although specified not to :(

Sorry we're coming back from holidays + our HF offsite. https://github.com/huggingface/text-generation-inference/pull/13 should patch BLOOM. It just requires to be deployed which should happen ASAP
We still want to improve the fix, typically handling return_full_text option. If you feel like contributing, please feel free to do so

cc @olivierdehaene

Closing as @olivierdehaene has deployed newer version. It fixes the bug from https://huggingface.co/bigscience/bloom/discussions/153#63a07cd82fabbbb8998f60de

Feel free to re-open if you still see an issue.