Copy of danielrosehill/hebrew-latin-token-classifier @ 01f092f1 (DOI 10.57967/hf/10997; DOI repos cannot be moved)

e303bc1 verified 1 day ago

5.63 kB

So the purpose of this voice note is to give a little bit of context about this particular project, the need for it, as I saw the need for it, the need for it was quite simple. I couldn't find that it already existed and thought it would be helpful for a project which I'll explain. This is a classifier for identifying Romanized Hebrew. So Hebrew words that are written in the Latin Alphabet, the English language Alphabet that appear in text. And I'll explain what might seem like a very obscure use case or a very niche model, but it wasn't something that I could find and there have been some really useful small models developed for the other direction.
So to do that, I'll tell the story of the project that I built this for. So for about the past year and a half, I've been developing, producing a podcast called My Weird Prompts. And the basic idea was I sent in prompts and often I'd send in specific prompts, often about AI engineering and never got around to reading the lengthy responses and thought, I'd love to have that in audio format. So this podcast is a, is a pipeline, Python pipeline running in modal. And I send in my questions as text.
And we generate using chatterboxtts, which is an English model, or at least we have it with English. And because many of my prompts touched upon subjects related to living in Israel, the generated text from the script writing agent would just contain Hebrew words written in English. And you can predict what went wrong or what goes wrong, which is that it's quite specific if the, if the words, the Hebrew words like Shabbat kosher are sufficiently well populated in the, in the training set for an English language model that the phonetics are known, at least the phonetics is they're pronounced by English speakers, they'll be pronounced in at least an intelligible way. But there is a huge amount of words that are just slightly too far down the long tail, but which are kind of essential to having conversations about Israel. We might talk about the, the National Insurance body, we might talk about the Misraad Ta Kalkala.
Now, of course we can have a heuristic, always use a translation, but sometimes there isn't really a good English translation. So I was, you know, thinking about how I can actually go about improving upon this. And there's been different approaches that are used, like language detection, of course, but it's a very tricky case in when you have a language, a script generation agent that's emitting text in Latin Alphabet, that's what it does. And because it does that. The fact that the text is written in that Alphabet is strongly suggestive even to these models, that it's English text, especially when the surrounding text and it's not expecting code switching, which is, I guess, what's happening in this case.
It's essentially the written script itself contains code switching. So I was thinking and thinking about different ways that this could be tackled and I started breaking it down into what's the problem here? And the first problem I could think about was, well, it's seeing the script and it's assuming everything is English. So it's a first step. Let's flag what's actually Hebrew that's been written in English characters.
And that's what this model is intended to do. And I'll explain the intended use case, because it doesn't stop here, is that this pass would flag all the Hebrew words. And the next flag, as I envision it in the pipeline, would then, and this is, thankfully there's an existing model for this. Take that word that's been flagged. Let's take, for argument's sake, and render that in Hebrew characters.
Right? And then the final step would be attaching the language tag for the span that it's Hebrew. So now a model that's expecting Hebrew receives text that is actually in Hebrew. And so that's the intended pipeline. It's those two accessory models and in my case, using concatenation so that the sentence, it would involve actually for code switching, concatenation within sentence boundaries, which is quite challenging, of course, but actually would be, I think, useful in doing this.
So that's the purpose of this model. And in terms of validation, some initial runs mostly actually works good initial, good initial benchmark. And the failure modes are subtle to think about because you get to a question of Shabbat, how do you handle Shabbat? Because, you know, we can. There's.
There's two approaches to it, I guess. One is the more one is the approach that we're going to say, well, Shabbat is a Hebrew word, it should be classified as Hebrew, but the implication is that it's going to be then in this use case at least, rendered by a Hebrew TTS engine. The other way to look at it is that while Shabbat has entered the English lexicon and shouldn't be classified, or maybe the classifier in a future iteration has a third type of Hebrew words that are predictably in the English corpus, so that a TTS model that is just running in English mode is not going to have trouble and leave it as a user setting as to how you want those words to be handled by the classifier. But for the proof of concept that this represents, I think it's probably a moot point, because hopefully it should work reliably either way. But the intention was to catch those words like I need to go to the misraad ibrute today I went to the etc.
All these words to have that word level classification built into the pipeline and get proper phonetic rendering in code switching examples. That's the purpose of the Latin classifier. Comments, feedbacks and ideas for the pipeline more than welcome public at danielrosehill. Com.

	So the purpose of this voice note is to give a little bit of context about this particular project, the need for it, as I saw the need for it, the need for it was quite simple. I couldn't find that it already existed and thought it would be helpful for a project which I'll explain. This is a classifier for identifying Romanized Hebrew. So Hebrew words that are written in the Latin Alphabet, the English language Alphabet that appear in text. And I'll explain what might seem like a very obscure use case or a very niche model, but it wasn't something that I could find and there have been some really useful small models developed for the other direction.
	So to do that, I'll tell the story of the project that I built this for. So for about the past year and a half, I've been developing, producing a podcast called My Weird Prompts. And the basic idea was I sent in prompts and often I'd send in specific prompts, often about AI engineering and never got around to reading the lengthy responses and thought, I'd love to have that in audio format. So this podcast is a, is a pipeline, Python pipeline running in modal. And I send in my questions as text.
	And we generate using chatterboxtts, which is an English model, or at least we have it with English. And because many of my prompts touched upon subjects related to living in Israel, the generated text from the script writing agent would just contain Hebrew words written in English. And you can predict what went wrong or what goes wrong, which is that it's quite specific if the, if the words, the Hebrew words like Shabbat kosher are sufficiently well populated in the, in the training set for an English language model that the phonetics are known, at least the phonetics is they're pronounced by English speakers, they'll be pronounced in at least an intelligible way. But there is a huge amount of words that are just slightly too far down the long tail, but which are kind of essential to having conversations about Israel. We might talk about the, the National Insurance body, we might talk about the Misraad Ta Kalkala.
	Now, of course we can have a heuristic, always use a translation, but sometimes there isn't really a good English translation. So I was, you know, thinking about how I can actually go about improving upon this. And there's been different approaches that are used, like language detection, of course, but it's a very tricky case in when you have a language, a script generation agent that's emitting text in Latin Alphabet, that's what it does. And because it does that. The fact that the text is written in that Alphabet is strongly suggestive even to these models, that it's English text, especially when the surrounding text and it's not expecting code switching, which is, I guess, what's happening in this case.
	It's essentially the written script itself contains code switching. So I was thinking and thinking about different ways that this could be tackled and I started breaking it down into what's the problem here? And the first problem I could think about was, well, it's seeing the script and it's assuming everything is English. So it's a first step. Let's flag what's actually Hebrew that's been written in English characters.
	And that's what this model is intended to do. And I'll explain the intended use case, because it doesn't stop here, is that this pass would flag all the Hebrew words. And the next flag, as I envision it in the pipeline, would then, and this is, thankfully there's an existing model for this. Take that word that's been flagged. Let's take, for argument's sake, and render that in Hebrew characters.
	Right? And then the final step would be attaching the language tag for the span that it's Hebrew. So now a model that's expecting Hebrew receives text that is actually in Hebrew. And so that's the intended pipeline. It's those two accessory models and in my case, using concatenation so that the sentence, it would involve actually for code switching, concatenation within sentence boundaries, which is quite challenging, of course, but actually would be, I think, useful in doing this.
	So that's the purpose of this model. And in terms of validation, some initial runs mostly actually works good initial, good initial benchmark. And the failure modes are subtle to think about because you get to a question of Shabbat, how do you handle Shabbat? Because, you know, we can. There's.
	There's two approaches to it, I guess. One is the more one is the approach that we're going to say, well, Shabbat is a Hebrew word, it should be classified as Hebrew, but the implication is that it's going to be then in this use case at least, rendered by a Hebrew TTS engine. The other way to look at it is that while Shabbat has entered the English lexicon and shouldn't be classified, or maybe the classifier in a future iteration has a third type of Hebrew words that are predictably in the English corpus, so that a TTS model that is just running in English mode is not going to have trouble and leave it as a user setting as to how you want those words to be handled by the classifier. But for the proof of concept that this represents, I think it's probably a moot point, because hopefully it should work reliably either way. But the intention was to catch those words like I need to go to the misraad ibrute today I went to the etc.
	All these words to have that word level classification built into the pipeline and get proper phonetic rendering in code switching examples. That's the purpose of the Latin classifier. Comments, feedbacks and ideas for the pipeline more than welcome public at danielrosehill. Com.