Poor Translation Help
#24
by madscientist1224 - opened
When translating from English to French, I find that it leaves off quite a bit of what I am trying to translate. For example, when prompted with "Hello, my name is John. What is your name? Unfortunately I don't speak much French." The returned translation is "Je m'appelle John, et vous, comment vous appelez-vous ?"
Is this due to the model size not being able to effectively pick up on the sentiment, temperature, or some other issue?
it is tokenizer problem and the model will pick form where it feels to be a new line , use NLTK for splitting the sentence and add to the bag that can contain around 200 tokens , then it will work