doesn't have to go down.


i really wasn't even going to play with that part of the system yet.


i was just going to change the the


uhhuh okay.


yeah just work with the models?


yeah just look at the length of the models and just see what happens.


yeah.


so


cool.


okay.


so uh what's uh


i guess your plan for you you you guys' plan for the next next week is just continue on these these same things we've been talking about for aurora and


i guess we can try to have some kind of new baseline for next week perhaps.


with all these minor things modified.


and then do other things.


play with the spectral subtraction.


and retry the m s g and things like that.


yeah we we have a big list.


big list?


you have a big list of of things to do.


well that's good.


i think that after all of this uh um confusion settles down in another


some point a little later next year there will be some sort of standard and it'll get out there.


and hopefully it'll have some effect from something that that has uh been done by our group of people.


but uh


even if it doesn't there's there's there'll be standards after that.


so


does anybody know how to um run matlab?


sort of in batch mode like you send it a bunch of commands to run and it gives you the output.


is it possible to do that?


i i think uh mike tried it.


yeah?


and he says it's impossible so he went to octave.


octave is the um unix clone of of matlab which you can batch.


octave.


uh.


okay.


great.


thanks.


yeah.


i was going crazy trying to do that.


huh.


what is octave so?


what's that?


it's a free software?


uh octave?


yeah it's it's it's free.


i think we have it here running somewhere.


um it's a little behind.


it's the same syntax but it's a little behind in that matlab went to these like um


you can have cells and you can you can uh implement object oriented type things with matlab.


uh octave doesn't do that yet.


so i think you


octave is kind of like matlab.


um four point something or


if it'll do like a lot of the basic matrix and vector stuff.


the basic stuff right.


that's perfect.


yeah.


great!


okay guess we're done.


okay.


well although by the way


so uh he's not here.


so


so you get to


yeah i will try to explain the thing that i did this this week during this week.


well uh you know that i work i begin to work with a new feature to detect voice unvoice.


uhhuh.


what i trying two m l p to to the with this new feature and the fifteen feature uh from the uh base system.


the the mel cepstrum?


no satly the the mel cepstrum


the new base system the new base system.


oh the


yeah we


the aurora system.


yeah the aurora system with the new filter v a d or something like that.


and i'm trying two m l p one one that only have three output.


voice unvoice and silence.


and other one that have fifty six output.


the probabilities of the allophone.


and i tried to do some experiment of recognition with that.


and only have result with with the m l p with the three output.


and i put together the fifteen features and the three m l p output.


and well the result are a little bit better but more or less similar.


uh i i'm i'm slightly confused.


huh.


what what feeds the uh the three output net?


voice unvoice and


no no what feeds it?


what features does it see?


the feature the input?


the inputs are the fifteen the fifteen uh bases feature.


uhhuh.


the with the new code.


and the other three features are r the variance of the difference between the two spectrum.


the variance of the auto correlation function except the the first point because half the height value is r zero.


and also r zero.


the first coefficient of the auto correlation function.


that is like the energy with these three feature


right.


also these three feature.


you wouldn't do like r one over r zero or something like that?


i mean usually for voiced unvoiced you'd do yeah you'd do something you'd do energy.


yeah.


but then you have something like spectral slope which is you get like r one over r zero or something like that.


uh yeah.


what are the r s?


i'm sorry i missed it.


r correlations.


no.


oh.


r no.


auto correlation?


yes yes the variance of the auto correlation function that uses that.


well that's the variance.


but if you just say what is


i mean to first order um


yeah one of the differences between voiced unvoiced and silence is energy.


another one is but the other one is the spectral shape.


i'll


the spectral shape.


yeah.


and so r one over r zero is what you typically use for that.


i don't use that.


i can't use


i'm saying that's what people typically use.


huh.


see because it because this is this is just like a single number to tell you um does the spectrum look like that or does it look like that.


uhhuh.


oh.


r r zero.


right?


so if it's if it's um if it's low energy uh but the but the spectrum looks like that or like that it's probably silence.


uh but if it's low energy and the spectrum looks like that it's probably unvoiced.


so if you just if you just had to pick two features to determine voiced unvoiced you'd pick something about the spectrum like uh r one over r zero um and r zero.


okay.


or you know you'd have some other energy measure.


and


like in the old days people did like uh zero crossing counts.


yeah yeah.


well i can also use this.


yeah.


um


because the result are a little bit better.


but we have in a point that everything is more or less the similar more or less similar.


but um


it's not quite better.


right but it seemed to me that what you were what you were getting at before was that there is something about the difference between the original signal or the original f f t and with the filter which is what


and the variance was one take uh on it.


i used this too.


right.


but it it could be something else.


suppose you didn't have anything like that.


then in that case if you have two nets


all right and this one has three outputs and this one has


uhhuh.


whatever.


fifty six or something.


if you were to sum up the probabilities for the voiced and for the unvoiced and for the silence here we've found in the past you'll do better at voiced-unvoiced-silence than you do with this one.


so just having the three output thing doesn't doesn't really buy you anything.


yeah.


the issue is what you feed it.


i have


so uh


so you're saying take the features that go into the voiced-unvoiced-silence net and feed those into the other one as additional inputs rather than having a separate


no


well that's another way.


that wasn't what i was saying.


but yeah that's certainly another thing to do.


no i was just trying to say if you if you bring this into the picture over this what more does it buy you?


huh.


and what i was saying is that the only thing i think that it buys you is um based on whether you feed it something different.


and something different in some fundamental way.


and so the kind of thing that that she was talking about before was looking at something uh um something uh about the difference between the the uh um log f f t uh log power uh and the log magnitude uh f spectrum uh and the um uh filter bank.


and so the filter bank is chosen in fact to sort of integrate out the effects of pitch.


and she's saying


you know trying


so the particular measure that she chose was the variance of this of this difference.


uhhuh.


but that might not be the right number.


maybe.


right?


i mean maybe there's something about the variance that's that's not enough.


or maybe there's something else that that one could use.


but i think that for me the thing that that struck me was that uh you want to get something back here.


so here's here's an idea.


uh what about it you skip all the all the really clever things and just fed the log magnitude spectrum into this?


uh i'm sorry.


this is


you have the log magnitude spectrum and you were looking at that and the difference between the filter bank and and computing the variance.


yeah.


that's a clever thing to do.


what if you stopped being clever?


ha!


and you just took this thing in here because it's a neural net and neural nets are wonderful.


and figure out what they can what they most need from things and i mean that's what they're good at.


so i mean you're you're you're trying to be clever and say what's the statistic that should we should get about this difference.


but uh in fact you know maybe just feeding this in or or feeding both of them in.


huh.


you know another way saying let it figure out what's the what is the interaction.


especially if you do this over multiple frames.


uhhuh.


then you have this over time and and both kinds of measures.


and uh you might get uh something better.


um


so so don't uh don't do the division.


but let the net have everything.


that's another thing you could do.


yeah.


i mean it seems to me if you have exactly the right thing then it's better to do it without the net.


because otherwise you're asking the net to learn this.


you know say if you wanted to learn how to do multiplication.


i mean you could feed it a bunch of you could feed two numbers that you wanted to multiply into a net.


and have a bunch of non linearities in the middle.


and train it to get the product of the output and it would work.


but it's kind of crazy.


because we know how to multiply.


and you you'd be you know much lower error usually if you just multiplied it out.


but suppose you don't really know what the right thing is.


and that's what these sort of dumb machine learning methods are good at.


so


um


anyway.


it's just a thought.


how long does it take carmen to train up one of these nets?


oh not too much.


yeah.


huh one day or less.


huh.


it's probably worth it.


what are what are your uh frame error rates for for this?


uh fifty six.


uh no the frame error rate?


fifty six i think.


is that maybe that's accuracy?


percent.


the accuracy.


fifty six percent accurate for voice unvoice


uhhuh.


no for


yes


i don't remember for voice unvoice.


oh okay.


maybe for the other one.


yeah voiced unvoiced hopefully would be a lot better.


okay.


for voiced.


i don't


should be in nineties somewhere.


better.


maybe for voice unvoice.


right.


this is for the other one.


i should i can't show that.


but i think that fifty five was for the when the output are the fifty six phone.


uhhuh.


that i look in the with the other nnn the other m l p that we have are more or less the same number.


silence will be better but more or less the same.


i think at the frame level for fifty six that was the kind of number we were getting for for uh um reduced band width uh stuff.


i think that i i i think that for the other one for the three output is sixty two three more or less.


that's all?


it's


yeah.


that's pretty bad.


yeah because it's noise also.


aha!


oh yeah.


and we have


i know.


okay.


but even in


oh yeah in training.


still


uh


well actually so this is a test that you should do then.


um if you're getting fifty six percent over here


uh that's in noise also.


right?


yeah yeah yeah.


oh okay.


if you're getting fifty six here try adding together the probabilities of all of the voiced phones here and all of the unvoiced phones.


will be


and see what you get then.


yeah.


i bet you get better than sixty three.


well i don't know.


but i i i think that we i have the result more or less maybe.


i don't know.


i don't i'm not sure but i remember that i can't show that.


okay but that's a that is a a good check point.


you should do that anyway.


okay?


given this this uh regular old net that's just for choosing for other purposes uh add up the probabilities of the different subclasses and see see how well you do.


uh and that you know anything that you do over here should be at least as good as that.


uhhuh.


okay.


i will do that.


the targets for the neural net uh they come from forced alignments?


but


uh no.


timit canonical mappings.


timit.


uh.


oh so this is trained on timit.


yeah.


yeah noisy timit.


yeah this for timit.


but noisy timit?


noisy timit.


right.


we have noisy timit with the noise of the the t i digits.


and now we have another noisy timit also with the noise of uh italian data base.


i see.


well there's going to be it looks like there's going to be a noisy uh some large vocabulary noisy stuff too.


somebody's preparing.


really?


i forget what it'll be.


resource management.


wall street journal.


something.


some some read task actually that they're preparing.


huh!


for what


for aurora?


yeah.


oh!


yeah so the uh


uh the issue is whether people make a decision now based on what they've already seen or they make it later.


and one of the arguments for making it later is let's make sure that whatever techniques that we're using work for something more than than connected digits.


huh.


so


when are they planning when would they do that?


huh i think late uh i think in the summer sometime.


okay thanks.


this is the work that i did during this date.


uhhuh.


and also huh i hynek last week say that if i have time i can to begin to to study.


well seriously the france telecom proposal


to look at the code


and something like that.


to know exactly what they are doing


because maybe that we can have some ideas.


but not only to read the proposal.


look look carefully what they are doing with the program and something like that.


and i begin to to work also in that.


but the first thing that i don't understand is that they are using r the uh log energy that this quite


i don't know why they have some constant in the expression of the lower energy.


i don't know what that means.


they have a constant in there you said?


yeah.


oh at the front it says uh log energy is equal to the rounded version of sixteen over the log of two.


this


uh


then maybe i can understand.


uh times the


well this is natural log.


and maybe it has something to do with the fact that this is


is that some kind of base conversion?


i i have no idea.


or


yeah that's what i was thinking.


but but um then there's the sixty four.


uh i don't know.


because maybe they're the threshold that they are using on the basis of this value


experimental results.


mcdonald's constant.


i don't know exactly.


because well i thought maybe they have a meaning.


but i don't know what is the meaning of take exactly this value.


yeah it's pretty funny looking.


so they're taking the number inside the log and raising it to sixteen over log base two.


i don't know.


yeah i


um right.


sixteen over two.


does it have to do with those sixty fours?


or


um


if we ignore the sixteen the natural log of one over the natural log of two times the


uhhuh.


well maybe somebody will think of something.


but this is uh


it may just be that they they want to have for very small energies


they want to have some kind of a


yeah the the effect i don't i can understand the effect of this.


no?


because it's to to do something like that.


well it says since you're taking a natural log it says that when when you get down to essentially zero energy this is going to be the natural log of one which is zero.


so it'll go down to uh to the natural log being


so the lowest value for this would be zero.


so you're restricted to being positive.


and this sort of smooths it for very small energies.


uh why they chose sixty four and something else that was probably just experimental.


yeah.


and the the the constant in front of it i have no idea.


um


well i i will look to try if i move this parameter in their code what happens?


maybe everything is maybe they tres hole are on basis of this.


uh i mean it they they probably have some particular fixed point arithmetic that they're using.


i don't know.


yeah i was just going to say maybe it has something to do with hardware.


and then it just


something they were doing.


i mean that they're probably working with fixed point or integer or something.


i think you're supposed to on this stuff anyway.


and and so maybe that puts it in the right realm somewhere.


well it just yeah puts it in the right range.


or


i think given at the level you're doing things in floating point on the computer i don't think it matters would be my guess.


uhhuh.


but


i this more or less anything


yeah.


okay and when did stephane take off?


he took off


i think that stephane will arrive today or tomorrow.


oh he was gone these first few days.


and then he's here for a couple days before he goes to salt lake city.


okay.


he's i think that he is in las vegas or something like that.


so he's he's going to i. cassp which is good.


i i don't know if there are many people who are going to i. cassp.


so so i thought make sure somebody go.


do have have people sort of stopped going to i. cassp in recent years?


um people are less consistent about going to i. cassp.


and i think it's still it's still a reasonable forum for students to to present things.


uh it's i think for engineering students of any kind i think it's it's if you haven't been there much it's good to go to.


uh to get a feel for things a range of things.


not just speech.


uh


but i think for for sort of dyed in the wool speech people um i think that i c s l p and eurospeech are much more targeted.


uhhuh.


and then there's these other meetings like h l t and and uh a s r u.


so there's there's actually plenty of meetings that are really relevant to to uh computational uh speech processing of one sort or another.


um so


i mean i mostly just ignored it because i was too busy and didn't get to it.


so uh


want to talk a little bit about what we were talking about this morning?


oh!


just briefly or or anything else?


um uh yeah.


so i i guess some of the progress i i've been getting a getting my committee members for the quals.


and um so far i have morgan and hynek mike jordan.


and i asked john ohala and he agreed.


cool.


yeah yeah.


so i'm i i just need to ask um malek.


one more.


um tsk


then uh i talked a little bit about um continuing with these dynamic um acoustic events.


and um we're we're we're thinking about a way to test the completeness of a a set of um dynamic uh events.


uh completeness in the in the sense that um if we if we pick these x number of acoustic events do they provide sufficient coverage for the phones that we're trying to recognize or or the the words that we're going to try to recognize later on?


and so morgan and i were uh discussing um uh a form of a cheating experiment where we get um we have uh um a chosen set of features or acoustic events.


and we train up a hybrid um system to do phone recognition on timit.


so the idea is if we get good phone recognition results using um these set of acoustic events then um that that says that these acoustic events are sufficient to cover a set of phones.


at least found in timit.


um so it would be a a measure of are we on the right track with with the the choices of our acoustic events?


um so that's going on.


and also just uh working on my uh final project for jordan's class.


uh which is


actually let me


hold that thought.


yeah.


let me back up while we're still on it.


okay sure.


the the other thing i was suggesting though is that given that you're talking about binary features uh maybe the first thing to do is just to count.


and uh count co occurrences and get probabilities for a discrete h m m.


because that'd be pretty simple.


because it's just say if you had ten ten events uh that you were counting uh each frame would only have a thousand possible values for these ten bits.


and uh so you could make a table that would say if you had thirty nine phone categories that would be a thousand by thirty nine.


and just count the co occurrences and divide them by the the uh uh uh uh count the co occurrences between the event and the phone and divide them by the number of occurrences of the phone.


and that would give you the likelihood of the of the event given the phone.


and um then just use that in a very simple h m m.


and uh you could uh do phone recognition then and uh wouldn't have any of the issues of the uh training of the net or


i mean it'd be on the simple side.


but


uhhuh.


uh um


you know if uh uh the example i was giving was that if if you had um onset of voicing and and end of voicing as being two kinds of events then if you had those all marked correctly and you counted co occurrences you should get it completely right.


so


um but you'd get all the other distinctions you know randomly wrong.


i mean there'd be nothing to tell you that.


so um uh


if you just do this by counting then you should be able to find out in a pretty straightforward way whether you have a sufficient uh set of events to to do the kind of level of of uh classification of phones that you'd like.


so that was that was the idea.


and then the other thing that we were discussing was was um okay how do you get the your training data?


because uh the switchboard transcription project uh uh you know was half a dozen people or so working off and on over a couple years.


and


uh similar similar amount of data to what you're talking about with timit training.


so it seems to me that the only reasonable starting point is uh to automatically translate the uh current timit markings into the markings you want.


and uh it won't have the kind of characteristic that you'd like of catching funny kind of things that maybe aren't there from these automatic markings.


but but


uh it's uh


it's probably a good place to start.


yeah.


yeah and a short short amount of time.


just to again just to see if that information is sufficient to uh determine the phones.


uhhuh.


huh.


so


yeah you could even then to to get an idea about how different it is you could maybe take some subset and you know go through a few sentences mark them by hand.


and then see how different it is from you know the canonical ones.


right.


just to get an idea a rough idea of if it really even makes a difference.


you can get a little feeling for it that way.


yeah that is probably right.


i mean uh my my guess would be that this is since timit's read speech that this would be less of a big deal.


if you went and looked at spontaneous speech it'd be more more of one.


and the other thing would be say if you had these ten events you'd want to see well what if you took two events or four events or ten events or


and you know and


and hopefully there should be some point at which having more information doesn't tell you really all that much more about what the phones are.


you could define other events as being sequences of these events too.


uh you could.


but the thing is what he's talking about here is a uh a translation to a per frame feature vector.


so there's no sequence in that.


i think.


i think it's just a


unless you did like a second pass over it or something after you've got your


yeah but we're just talking about something simple here yeah to see if


yeah.


yeah yeah.


i'm adding complexity.


just


you know the idea is with a with a very simple statistical structure could you could you uh at least verify that you've chosen features that are sufficient?


okay and you were saying something starting to say something else about your your class project or


oh.


yeah um


so for my class project i'm um i'm tinkering with uh support vector machines.


something that we learned in class.


and uh um basically just another method for doing classification.


and so i'm going to apply that to um compare it with the results by um king and taylor who did um these.


um


using recurrent neural nets they recognized um a set of phonological features.


and made a mapping from the m f c c's to these phonological features.


so i'm going to do a similar thing with with support vector machines.


and see if


so what's the advantage of support vector machines?


what


um so support vector machines are are good with dealing with a less amount of data.


huh.


and um so if you if you give it less data it still does a reasonable job in learning the the patterns.


um and um


i guess it


yeah.


they're sort of succinct.


and and they uh


does there some kind of a distance metric that they use?


or how do they for what do they do for classification?


um.


right.


so the the simple idea behind a support vector machine is um you have you have this feature space.


right?


uhhuh.


and then it finds the optimal separating plane um between these two different um classes.


and um and so um


what it at the end of the day what it actually does is it picks those examples of the features that are closest to the separating boundary.


and remembers those.


and and uses them to recreate the boundary for the test set.


so given these um these features or or these these examples um critical examples which they call support support vectors then um given a new example if the new example falls um away from the boundary in one direction then it's classified as being a part of this particular class.


oh.


and otherwise it's the other class.


so why save the examples?


why not just save what the boundary itself is?


um huh


let's see.


uh


yeah that's a good question.


i


that's another way of doing it.


yeah.


huh.


right so so it i mean i i guess it's


sort of an equivalent.


you know it it goes back to nearest neighbor sort of thing.


uhhuh.


right.


um if is it uh


when is nearest neighbor good?


well nearest neighbor good is good if you have lots and lots of examples.


um but of course if you have lots and lots of examples then it can take a while to to use nearest neighbor.


there's lots of look ups.


so a long time ago people talked about things where you would have uh a condensed nearest neighbor.


where you would you would you would pick out uh some representative examples which would uh be sufficient to represent to to correctly classify everything that came in.


oh.


i i think i think support vector stuff sort of goes back to to that kind of thing.


i see.


um


so rather than doing nearest neighbor where you compare to every single one you just pick a few critical ones.


and


yeah.


huh.


and the


you know um neural net approach uh or gaussian mixtures for that matter are sort of fairly brute force kinds of things where you sort of you predefine that there is this big bunch of parameters.


and then you you place them as you best can to define the boundaries.


and in fact as you know these things do take a lot of parameters


and and uh if you have uh only a modest amount of data you have trouble uh learning them.


um so i i guess the idea to this is that it it is reputed to uh be somewhat better in that regard.


uhhuh.


right.


it can be a a reduced um parameterization of of the the model by just keeping certain selected examples.


so


but i don't know if people have done sort of careful comparisons of this on large tasks or anything.


maybe maybe they have.


i don't know.


yeah i don't know either.


do you get some kind of number between zero and one at the output?


actually you don't get a you don't get a nice number between zero and one.


you get you get either a zero or a one.


um uh


there are there are


well basically it's it's um you you get a distance measure at the end of the day.


and then that distance measure is is um is translated to a zero or one.


um


but that's looking at it for for classification for binary classification.


that's for classification.


and you get that for each class you get a zero or a one.


right?


right.


but you have the distances to work with.


you have the distances to work with.


because actually mississippi state people did use support vector machines for uh uh speech recognition and they were using it to estimate probabilities.


yeah.


yeah they they had a had a way to translate the distances into into probabilities with the with the simple um uh sigmoidal function.


and did they use sigmoid or a softmax type thing?


and didn't they like exponentiate or something?


um yeah.


and then divide by the sum of them?


there's some there's like one over one plus the exponential or something like that.


or


oh it


oh so it is a sigmoidal.


okay.


all right.


did the did they get good results with that?


i mean they're okay.


i i don't i don't think they were earth earth shattering.


but i think that uh this was a couple years ago.


huh.


i remember them doing it at some meeting.


and and um i don't think people were very critical because it was interesting just to to try this.


and you know it was the first time they tried it.


so so the you know the numbers were not incredibly good.


but there's you know it was reasonable.


uhhuh.


i i don't remember anymore.


i don't even remember what the task was.


it was broadcast news or something.


i don't know.


right.


uh so barry if you just have zero and ones how are you doing the speech recognition?


oh!


i'm not i'm not planning on doing speech recognition with it.


i'm just doing detection of phonological features.


oh okay.


so uh for example this this uh feature set called the uh sound patterns of english um is just a bunch of um binary valued features.


let's say is this voicing or is this not voicing?


is this sonorants not sonorants?


and stuff like that.


okay.


so


did you find any more mistakes in their tables?


uh i haven't gone through the entire table yet.


yeah.


yesterday i brought chuck the table.


and i was like wait this is is the mapping from n to to this phonological feature called um coronal.


is is should it be shouldn't it be a one.


or should it should it be you know coronal instead of not coronal as it was labelled in the paper.


so i haven't hunted down all the all the mistakes yet.


uhhuh.


but


but as i was saying people do get probabilities from these things.


and and uh we were just trying to remember how they do.


but people have used it for speech recognition and they have gotten probabilities.


so they have some conversion from these distances to probabilities.


right yeah.


there's you have you have the paper.


right?


the mississippi state paper?


yeah if you're interested you could look.


and


okay okay.


yeah.


i can i can show you i


so in your in in the thing that you're doing uh you have a vector of ones and zeros for each phone?


our


uhhuh.


uh is this the class project?


or


okay.


um


is that what you're


right right right.


so for every phone there is there is a um a vector of ones and zeros uh corresponding to whether it exhibits a particular phonological feature or not.


and so when you do your


i'm what is the task for the class project?


to come up with the phones?


or to come up with these vectors to see how closely they match the phones?


oh.


right.


um to come up with a mapping from um m f c c's or some feature set um to uh to whether there's existence of a particular phonological feature.


and um


yeah.


basically it's to learn a mapping from from the m f c c's to uh phonological features.


is it did that answer your question?


i think so.


okay.


i guess i mean uh i'm not sure what you what you're what you get out of your system.


do you get out a uh a vector of these ones and zeros and then try to find the closest matching phoneme to that vector?


uhhuh.


or


no no.


i'm not i'm not planning to do any any phoneme mapping yet.


just it's it's basically it's it's really simple basically a detection of phonological features.


i see.


and um because the uh


so king and and taylor um did this with uh recurrent neural nets.


and this their their idea was to first find a mapping from m f c c's to uh phonological features.


and then later on once you have these phonological features then uh map that to phones.


so i'm i'm sort of reproducing phase one of their stuff.


huh.


so they had one recurrent net for each particular feature?


right.


right right right.


i did they compare that i mean what if you just did phone recognition and did the reverse lookup?


uh


so you recognize a phone and which ever phone was recognized you spit out it's vector of ones and zeros.


uhhuh.


i expect you could do that.


i mean uh


uh.


that's probably not what he's going to do on his class project.


yeah.


no.


so um have you had a chance to do this um thing we talked about yet with the uh


um


insertion penalty?


uh no actually i was going a different


that's a good question too.


but i was going to ask about the the um changes to the data in comparing p l p and mel cepstrum for the s r i system.


well what i've been


changes to the data.


i'm not sure i


right.


so we talked on the phone about this that that there was still a difference of a of a few percent.


and you told me that there was a difference in how the normalization was done.


and i was asking if you were going to do redo it uh for p l p with the normalization done as it had been done for the mel cepstrum.


uhhuh.


uh right.


no i haven't had a chance to do that.


what i've been doing is uh trying to figure out


okay.


it just seems to me like there's a um


well it seems like there's a bug.


because the difference in performance is it's not gigantic.


but it's big enough that it it seems wrong.


yeah.


i agree.


and


but i thought that the normalization difference was one of the possibilities.


yeah but i don't i'm not


right?


i guess i don't think that the normalization difference is going to account for everything.


so what i was working on is um just going through and checking the headers of the wavefiles.


to see if maybe there was a um a certain type of compression or something that was done that my script wasn't catching.


so that for some subset of the training data uh the the the features i was computing were junk.


which would you know cause it to perform okay.


but uh you know the the models would be all messed up.


so i was going through and just double checking that kind of think first to see if there was just some kind of obvious bug in the way that i was computing the features.


uhhuh.


i see.


okay.


looking at all the sampling rates to make sure all the sampling rates were what eight k what i was assuming they were.


yeah.


um


yeah that makes sense to check all that.


so i was doing that first before i did these other things just to make sure there wasn't something


although really uh uh a couple three percent uh difference in word error rate uh could easily come from some difference in normalization i would think.


but


and i think i'm trying to remember but i think i recall that andreas was saying that he was going to run sort of the reverse experiment.


uh which is to try to emulate the normalization that we did.


but with the mel cepstral features.


sort of you know back up from the system that he had.


i thought he said he was going to


i have to look back through my my email from him.


yeah he's probably off at at uh his meeting now.


he's gone now.


the i think they should be roughly equivalent.


i mean again the cambridge folk found the p l p actually to be a little better.


right.


uh so it's um


i mean the other thing i wonder about was whether there was something just in the the bootstrapping of their system which was based on


but maybe not since they


yeah see one thing that's a little bit um


i was looking i've been studying and going through the logs for the system that um andreas created.


and um his uh the way that the s r i system looks like it works is that it reads the wavefiles directly.


uh and does all of the cepstral computation stuff on the fly.


and so there's no place where these where the cepstral files are stored anywhere that i can go look at and compare to the p l p ones.


so whereas with our features he's actually storing the cepstrum on disk and he reads those in.


but it looked like he had to give it uh


even though the cepstrum is already computed he has to give it uh a front end parameter file.


which talks about the kind of uh computation that his mel cepstrum thing does.


uhhuh.


so i i don't know if that it probably doesn't mess it up.


it probably just ignores it if it determines that it's already in the right format or something.


but the the the two processes that happen are a little different.


so


yeah.


so anyway there's stuff there to sort out.


okay.


let's go back to what you thought i was asking you.


no and i didn't have a chance to do that.


ha!


oh!


you had the same answer anyway.


yeah i've been um i've been working with um jeremy on his project.


and then i've been trying to track down this bug in uh the icsi front end features.


so one thing that i did notice yesterday i was studying the um the uh rasta code.


and it looks like we don't have any way to um control the frequency range that we use in our analysis.


we basically it looks to me like we do the f f t um and then we just take all the bins.


and we use everything.


we don't have any set of parameters where we can say you know only process from you know a hundred and ten hertz to thirty seven fifty.


um


at least i couldn't see any kind of control for that.


yeah i don't think it's in there.


i think it's in the uh uh uh the filters.


so the f f t is on everything.


but the filters


for instance ignore the the lowest bins and the highest bins.


and what it does is it it copies


the the filters?


which filters?


the filter bank which is created by integrating over f f t bins.


uhhuh.


when you get the mel when you go to the mel scale?


right.


yeah.


it's bark scale.


and it's it it um it actually copies the uh um the second filters over to the first.


so the first filters are always


and you can you can specify a different number of uh features different number of filters.


i think.


as i recall.


so you can specify a different number of filters and whatever um uh you specify the last ones are going to be ignored.


so that that's a way that you sort of change what the what the bandwidth is.


you can't do it without i think changing the number of filters.


i saw something about uh that looked like it was doing something like that but i didn't quite understand it.


but


so maybe


yeah so the idea is that the very lowest frequencies and and typically the veriest highest frequencies are kind of junk.


uhhuh.


and so um


you just for continuity you just approximate them by by the second to highest and second to lowest.


it's just a simple thing we put in.


and and so if you


but so the but that's a fixed uh thing?


there's nothing that lets you


yeah i think that's a fixed thing.


but see see my point?


if you had if you had ten filters then you would be throwing away a lot at the two ends.


and if you had if you had fifty filters you'd be throwing away hardly anything.


um i don't remember there being an independent way of saying we're just going to make them from here to here.


use this analysis bandwidth or something.


but i i i don't know.


it's actually been awhile since i've looked at it.


yeah i went through the feacalc code and then looked at you know just calling the rasta libs and thing like that.


and i didn't i couldn't see any place where that kind of thing was done.


but um i didn't quite understand everything that i saw.


yeah see i don't know feacalc at all.


so


uhhuh.


but it calls rasta with some options.


and um


right.


but i i think in


i don't know.


i guess for some particular database you might find that you could tune that and tweak that to get that a little better.


but i think that in general it's not that critical.


i mean there's


yeah.


you can you can throw away stuff below a hundred hertz or so.


and it's just not going to affect phonetic classification at all.


another thing i was thinking about was um is there a


i was wondering if there's maybe um certain settings of the parameters when you compute p l p which would basically cause it to output mel cepstrum.


so that in effect what i could do is use our code.


but produce mel cepstrum.


and compare that directly to


well it's not precisely.


huh.


i mean


um um


what you can do is um you can definitely change the the filter bank from being uh a uh trapezoidal integration to a a a triangular one.


uhhuh.


which is what the typical mel mel cepstral uh filter bank does.


and some people have claimed that they got some better performance doing that.


so you certainly could do that easily.


but the fundamental difference


i mean there's other small differences


there's a cubic root that happens.


right?


yeah.


but you know as opposed to the log in the other case.


i mean the fundamental difference that we've seen any kind of difference from before which is actually an advantage for the p l p uh i think is that the the smoothing at the end is auto regressive instead of being cepstral uh from cepstral truncation.


so um it's a little more noise robust.


um and that's that's why when people started getting databases that had a little more noise in it like like uh um broadcast news and so on.


that's why cambridge switched to p l p i think.


so um


that's a difference that i don't think we put any way to get around.


since it was an advantage.


uhhuh.


um uh


but we did uh we did hear this comment from people at some point that um it uh they got some better results with the triangular filters rather than the trapezoidal.


so that is an option in rasta.


huh.


uh and you can certainly play with that.


but i think you're probably doing the right thing to look for bugs first.


yeah just it just seems like this kind of behavior could be caused by you know some of the training data being messed up.


i don't know.


could be.


you know you're sort of getting most of the way there but there's a


so i started going through and looking


one of the things that i did notice was that the um log likelihoods coming out of the log recognizer from the p l p data were much lower much smaller.


than for the mel cepstral stuff and that the average amount of pruning that was happening was therefore a little bit higher for the p l p features.


uhhuh!


so since he used the same exact pruning thresholds for both i was wondering if it could be that we're getting more pruning.


oh!


he he he used the identical pruning thresholds even though the the range of of the


yeah.


oh well that's that's a pretty good point right there.


right.


so


i would think that you might want to do something like uh you know look at a few points to see where you are starting to get significant search errors.


that's


well what i was going to do is i was going to take um a couple of the utterances that he had run through.


then run them through again.


but modify the pruning threshold and see if it you know affects the score.


but i mean you could uh if if if that looks promising you could you know uh run the overall test set with a with a few different uh pruning thresholds for both.


uhhuh.


and presumably he's running at some pruning threshold that's that's uh you know gets very few search errors.


but is is relatively fast.


right i mean yeah generally in these things you you turn back pruning really far.


and


so i i didn't think it would be that big a deal.


because i was figuring well you have it turned back so far that you know it


but you may be in the wrong range for the p l p features for some reason.


yeah.


and the uh the the run time of the recognizer on the p l p features is longer.


which sort of implies that the networks are bushier.


you know there's more things it's considering.


which goes along with the fact that the matches aren't as good.


so uh you know it could be that we're just pruning too much.


so


yeah maybe just be different kind of distributions.


and and


uhhuh.


so that's another possible thing.


they they should really shouldn't.


there's no particular reason why they would be exactly behave exactly the same.


uhhuh right.


right.


there's lots of little differences.


trying to track it down.


yeah.


i guess this was a little bit off topic i guess.


because i was i was thinking in terms of this as being a a a a core item that once we once we had it going we would use for a number of the front end things also.


um


want to


that's as far as my stuff goes.


what's what's on


well i tried this mean subtraction method.


due to avendano i'm taking um six seconds of speech.


um i'm using two second f f t analysis frames stepped by a half second.


so it's a quarter length step.


and i i take that frame and four the four i take


sorry.


i take the current frame and the four past frames and the four future frames.


and that adds up to six seconds of speech.


and i calculate um the spectral mean of the log magnitude spectrum over that n.


i use that to normalize the the current center frame by mean subtraction.


and i then then i move to the next frame and i i do it again.


well actually i calculate all the means first.


and then i do the subtraction.


and um the i tried that with h d k.


the aurora setup of h d k training on clean t i digits.


and um it it helped.


um in a phony reverberation case


um where i just used the simulated impulse response um the error rate went from something like eighty it was from something like eighteen percent to um four percent.


and on meeting recorder far mike digits mike on channel f it went from um forty one percent error to eight percent error.


on on the real data not with artificial reverb?


right.


uhhuh.


and that that was um trained on clean speech only.


which i'm guessing is the reason why the baseline was so bad.


and


that's actually a little side point is i think that's the first results that we have uh uh uh of any sort on the far field uh on on the far field data uh for recorded in in meetings.


oh um actually um adam ran the s r i recognizer.


did he?


on the near field on the


on the far field also.


he did one p z m channel and one p d a channel.


oh did he?


oh.


i didn't recall that.


what kind of numbers was he getting with that?


i i'm not sure.


i think it was about five percent error for the p z m channel.


five.


i think.


yeah.


so why were you getting forty one here?


is this


um


i i'm i'm guessing it was the the training data.


uh clean t i digits is like pretty pristine training data.


and if they trained the s r i system on this t v broadcast type stuff


i think it's a much wider range of channels.


and it


no.


but wait a minute.


i i i i think he


what am i saying here?


so that was the s r i system.


maybe you're right.


because it was getting like one percent so it's still this kind of ratio.


it was it was getting one percent or something on the near field.


wasn't it?


uhhuh.


or it wa it was around one.


yeah.


yeah i think it was getting around one percent for the near for the for the close mike.


huh.


so it was like one to five?


okay.


so it's still this kind of ratio.


it's just


it's a lot more training data.


so


so probably it should be something we should try then is to is to see if is at some point just to take to transform the data.


and then and then uh use use it for the s r i system.


you you mean um


so you're so you have a system which for one reason or another is relatively poor.


and and uh you have something like forty one percent error.


uh and then you transform it to eight by doing doing this this work.


um


so here's this other system which is a lot better.


but there's still this kind of ratio.


it's something like five percent error with the the distant mike.


and one percent with the close mike.


so the question is how close to that one can you get if you transform the data using that system?


right so so i guess this s r i system is trained on a lot of broadcast news or switchboard data.


yeah.


is that right?


do you know which one it is?


it's trained on a lot of different things.


it's trained on uh a lot of switchboard call home.


uhhuh.


um a bunch of different sources.


some digits.


there's some digits training in there.


okay.


one thing i'm wondering about is what this mean subtraction method um will do if it's faced with additive noise.


huh.


because i i it's because i don't know what log magnitude spectral subtraction is going to do to additive noise.


well it's it's not exactly the right thing.


that's that's the


but uh but you've already seen that because there is added noise here.


that's that's yeah that's true.


yeah.


that's a good point.


so um


okay so it's then then it's it's it's reasonable to expect it would be helpful if we used it with the s r i system.


and


i mean as helpful


i mean


so that's the question.


yeah we're often asked this when we work with a system that that isn't isn't sort of industry industry standard great.


uhhuh.


uh and we see some reduction in error using some clever method.


then you know will it work on a on a on a good system?


so uh


you know this other one's it was a pretty good system.


i think you know one one percent word error rate on digits is uh digit strings is not uh you know stellar.


but but given that this is real digits as opposed to uh sort of laboratory


well


and it wasn't trained on this task either.


and it wasn't trained on this task.


actually one percent is sort of you know sort of in a reasonable range.


people would say yeah i could i can imagine getting that.


and uh so the the four or five percent or something is is is quite poor.


uh you know if you're doing a uh a sixteen digit uh credit card number you'll basically get it wrong almost all the time.


so so uh um


huh.


a significant reduction in the error for that would be great.


huh okay.


and and then uh


yeah.


so


cool.


sounds good.


all right um i actually have to run.


so i don't think i can do the digits.


but um i guess i'll leave my microphone on?


uh yeah.


thank you.


yep.


so uh


the suggestion was to have these guys start to


okay.


why don't you go ahead dave?


um


so yeah the this past week i've been mainly occupied with um getting some results from the s.r.i. system trained on this short hub five training set for the mean subtraction method.


and um


i ran some tests last night.


but um


the results are suspicious.


um it's um because they're the baseline results are worse than um andreas than results andreas got previously.


and it could have something to do with um


that's on digits?


that's on digits.


it it it could it could have something to do with um downsampling.


huh.


that's that's worth looking into.


apart from that i guess the the main thing i have i have to talk is um where i'm planning to go over the next week.


so i've been working on integrating this mean subtraction approach into the smartkom system.


and there's this question of well so um in my tests before with h.t.k.i. found it worked it worked the best with about twelve seconds of data used to estimate the mean.


but we'll often have less in the smartkom system.


um


so i think we'll use as much data as we have at a particular time.


and we'll we'll concatenate utterances together um to get as much data as we possibly can from the user.


but um there's a question of how to set up the models so um we could train the models.


if we think twelve seconds is ideal we could train the models using twelve seconds to calculate the mean to mean subtract the training data.


or we could um use some other amount.


so like i did an experiment where i um was using six seconds in test.


um but for i tried twelve seconds in train.


and i tried um um the same in train.


i'm i tried six seconds in train.


and six seconds in train was about point three percent better.


um and um it's not clear to me yet whether that's something significant.


so i want to do some tests and um actually make some plots of um for a particular amount of data and test what happens if you vary the amount of data in train.


uhhuh.


uh guenter i don't know if you followed this stuff but this is uh a uh uh long-term long-term window f.f.t.'s.


yeah.


yeah he you talked about it.


we we spoke about it already.


oh okay.


so you know what he's doing.


all right.


so i was i actually ran the experiments mostly.


and i i was i was hoping to have the plots with me today.


i just didn't get to it.


but um


yeah i i would be curious about people's feedback on this.


because i'm i i think there are some i think it's it's kind of like a a bit of a tricky engineering problem.


i'm trying to figure out what's the optimal way to set this up.


so um i'll try to make the plots and then put some postscript up on my on my web page.


and i'll mention it in my status report if people want to take a look.


you could clarify something for me.


you're saying point three percent.


you take a point three percent hit when the training and testing links are don't match or something?


is that what it is?


or


well it


i i don't think it it's just for any mismatch you take a hit.


yeah.


in some cases it might be better to have a mismatch.


like i think i saw something like like if you only have two seconds in test or um maybe it was something like four seconds you actually do a little better if you um train on six seconds than if you train on four seconds.


right.


um


but the case uh with the point three percent hit was using six seconds in test um comparing train on twelve seconds versus train on six seconds.


and which was worse?


the train on twelve seconds.


okay.


but point three percent uh from what to what?


that's point three percent.


on the the the accuracies went from it was something vaguely like ninety five point six accuracy um improved to ninety five point nine when i


so four point four to four point one.


so yeah.


so about a about an eight percent uh seven or eight percent relative.


uh


well i think in a you know if if you were going for an evaluation system you'd care.


but if you were doing a live system that people were actually using nobody would notice.


it's uh i think the thing is to get something that's practical that that you could really use.


huh.


that's that's interesting.


all right the uh i see your point.


i guess i was thinking of it as um an interesting research problem.


yeah.


the how to i was thinking that for the a.s.r.u. paper we could have a section saying for smartkom we we in we tried this approach in uh interactive system which i don't think has been done before.


uhhuh.


and and then there was two research questions from that.


and one is the does it still work if you just use the past history.


all right.


and the other was this question of um what i was just talking about now.


so i guess that's why i thought it was interesting.


i mean a short time f.f.t. short time cepstrum calculation uh mean mean calculation work that people have in commercial systems they do this all the time.


they the they calculate it from previous utterances and then use it you know.


um


but but uh


as you say there hasn't been that much with this long longtime uh spectra work.


uh


oh oh okay.


so that's that's that's standard.


pretty common.


yeah.


okay.


but uh


yes.


no it is interesting.


and the other thing is i mean there's two sides to these really small uh gradations in performance.


um i mean on the one hand in a practical system if something is uh four point four percent error four point one percent error people won't really tell be able to tell the difference.


on the other hand when you're doing uh research you may uh you might find that the way that you build up a change from a ninety five percent accurate system to a ninety eight percent accurate system is through ten or twelve little things that you do that each are point three percent.


so so the they they it's i don't mean to say that they're they're irrelevant.


uh they are relevant.


but um for a demo you won't see it.


uhhuh.


right.


and um


let's let's see.


um


and then there's um another thing i want to start looking at um is um the choice of the analysis window length.


so i've just been using two seconds.


just because that's what carlos did before.


uh i wrote to him asking about he chose the two seconds.


and it seemed like he chose it a bit informally.


so um


with the with the h.t.k. set-up i should be able to do some experiments on just varying that length.


say between one and three seconds in a few different reverberation conditions.


um say this room and also a few of the artificial impulse responses we have for reverberation.


just um making some plots and seeing how they look.


so


with the the sampling rate i was using one second or two seconds or four seconds is at a power of two um number of samples.


and um i'll i'll for the ones in between i guess i'll just zero pad.


uhhuh.


i guess one thing that might also be an issue uh because part of what you're doing is you're getting a a spectrum over a bunch of different kinds of speech sounds.


and so it might matter how fast someone was talking for instance.


oh.


you know if you if if if there's a lot of phones in one second maybe you'll get a a really good sampling of all these different things.


and and uh on the other hand if someone's talking slowly maybe you'd need more.


i don't know if you have some samples of faster or slower speech.


huh.


but it might make a difference.


i don't know.


uh yeah.


i don't i don't think the t.i. digits data that i have um is would be appropriate for that.


yeah probably not.


yeah.


but what do you what about if i i fed it through some kind of um speech processing algorithm that changed the speech rate?


yeah but then you'll have the degradation of of uh whatever you do uh added onto that.


but maybe.


maybe if you get something that sounds that that's does a pretty job at that.


well uh just if you think it's worth looking into.


you could imagine that.


i mean it it is getting a little away from reverberation.


um yeah.


it's just that you're making a choice.


uh i was thinking more from the system aspect if you're making a choice for smartkom that that that it might be that it's it the optimal number could be different depending on


right.


could be.


i don't know.


and and the third thing um uh is um barry explained l.d.a. filtering to me yesterday.


and so um mike shire in his thesis um did a a series of experiments um training l.d.a. filters in on different conditions.


and you were interested in having me repeat this for


for this mean subtraction approach?


is is that right?


or for these long analysis windows i guess is the right way to put it.


i guess the the the issue i was the general issue i was bringing up was that if you're have a moving moving window uh a a a set of weights times things that uh move along shift along in time that you have in fact a linear time invariant filter.


and you just happened to have picked a particular one by setting all the weights to be equal.


and so the issue is what are some other filters that you could use uh in that sense of filter.


uhhuh.


and um


as i was saying i think the simplest thing to do is not to train anything but just to do some sort of uh uh hamming or hanning uh kind of window kind of thing.


just sort of to de emphasize the jarring.


so i think that would sort of be the first thing to do.


but then yeah the l.d.a. uh is interesting because it would sort of say well suppose you actually trained this up to do the best you could by some criterion.


what would the filter look like then?


uh


that's sort of what we're doing in this aurora stuff.


and uh


it's still not clear to me in the long run whether the best thing to do would be to do that or to have some stylized version of the filter that looks like these things you've trained up because you always have the problem that it's trained up for one condition and it isn't quite right for another.


so


uh that's that's why that's why rasta filter has actually ended up lasting a long time.


people still using it quite a bit because you don't change it.


doesn't get any worse.


huh.


anyway.


okay.


so um


actually i was just thinking about what i was asking about earlier which is about having less than say twelve seconds in the smartkom system to do the mean subtraction.


you said in systems where you use cepstral mean subtraction they concatenate utterances.


and do you know how they address this issue of um testing versus training?


can


i think what they do is they do it always online.


go ahead.


i mean that you just take what you have from the past.


that you calculate the mean of this and subtract the mean.


um


and then you can yeah you you can increase your window while you get while you are getting more samples.


and um so so in in that case what do they do when they're um performing the cepstral mean subtraction on the training data?


so because you'd have hours and hours of training data.


so do they cut it off and start over?


at intervals.


or


so do you have uh you you mean you have files which are hours of hours long?


oh well no.


i guess not.


yeah.


but


i mean usually you have in the training set you have similar conditions.


i mean file lengths are i guess the same order or in the same size as for test data or


aren't they?


okay.


but it's


so if someone's interacting with the system though uh morgan uh morgan said that you would tend to um chain utterances together.


um


well i think what i was i thought what i was saying was that um at any given point you are going to start off with what you had from before.


oh.


from


and so if you're splitting things up into utterances.


so for instance in a dialogue system where you're going to be asking uh you know for some information there's some initial something.


and you know the first time out you you might have some general average.


but you you you don't have very much information yet.


but at after they've given one utterance you've got something.


you can compute your mean cepstra from that.


uhhuh.


and then can use it for the next thing that they say.


uh


so that you know the performance should be better that second time.


and i think the heuristics of exactly how people handle that and how they handle their training i'm sure vary from place to place.


but i think the ideally it seems to me anyway that you you would want to do the same thing in training as you do in test.


but that's that's just uh a prejudice.


and i think anybody working on this with some particular task would experiment.


right.


i i guess the question i had was um amount of data was the amount of data that you'd give it to um update this estimate.


because say you if you have say five thousand utterances in your training set um and you you keep the mean from the last utterance.


by the time it gets to the five thousandth utterance


no but those are all different people with different i mean in


so for instance in in the in a telephone task these are different phone calls.


so you don't want to chain it together from a from a different phone call.


okay.


so so so they would


so it's within speaker.


yeah.


within phone call.


if it's a dialogue system it's within whatever this characteristic you're trying to get rid of is expected to be consistent over.


huh.


and it


right?


so you'd you and so in training you would start over at at every new phone call or at every new speaker.


now you know maybe you'd use something from the others.


just because at the beginning of a call you don't know anything.


and so you might have some kind of general thing that's your best guess to start with.


but


so i i


you know a lot of these things are proprietary.


so we're doing a little bit of guesswork here.


i mean what do what do people do who really face these problems in the field?


well they have companies.


and they don't tell other people exactly what they do.


but but i mean when you the the hints that you get from what they when they talk about it are that they do they all do something like this.


right.


okay.


i see.


because i so this smartkom task first off it's this t.v. and movie information system.


yeah but you might have somebody who's using it.


and


and then later you might have somebody else who's using it.


yeah.


and so you'd want to set some


i i see.


i was i was about to say so if if you ask it what what movies are on t.v. tonight.


if i look at my wristwatch when i say that it's about two seconds.


the way i currently have the mean subtraction um set up the the analysis window is two seconds.


so what you just said about what do you start with raises a question of what do i start with then.


uhhuh.


i guess it because


well.


so in that situation though maybe what's a little different there is i think you're talking about there's only one


it it it also depends


we're getting a little off track here.


oh.


but but but


right.


uh there's been some discussion about whether the work we're doing in that project is going to be for the kiosk or for the mobile or for both.


and i think for this kind of discussion it matters.


if it's in the kiosk then the physical situation is the same.


it's going to you know the exact interaction of the microphone's going to differ depending on the person and so forth.


but at least the basic acoustics are going to be the same.


so if it's really in one kiosk then i think that you could just chain together and and you know as much as much speech as possible to


because what you're really trying to get at is the is the reverberation characteristic.


yeah.


but in in the case of the mobile uh presumably the acoustic's changing all over the place.


and in that case you probably don't want to have it be endless because you want to have some sort of it's it's not a question of how long do you think it's you can get an approximation to a stationary something given that it's not really stationary.


huh.


so


and i i i guess i just started thinking of another question.


which is for for the very first frame what what do i do.


if i'm if i take if i use that frame to calculate the mean then i'm just going to get nothing.


uhhuh.


right.


um


so i should probably have some kind of default mean for the first couple of frames.


okay.


or subtract nothing.


i mean it's


and and that's that's i guess that's something that's people have figured out how to deal with in cepstral mean subtraction as well?


yeah yeah.


yeah people do something.


they they uh they have some um


uh


in in cepstral mean subtraction for short-term window analysis windows as is usually done you're trying to get rid of some very general characteristic.


and so uh if you have any other information about what a general kind of characteristic would be then you you can do it there.


you can also


you can also reflect the data.


so you take uh


you know.


i'm not sure how many frames you need.


but you take that many from the front and flip it around to as the negative value.


uhhuh.


yeah.


that's


so you can always


the other thing is that and and i i remember b.b.n. doing this is that if you have a multi pass system um if the first pass it takes most of the computation the second and the third pass could be very very quick.


just looking at a relatively small small uh space of hypotheses.


huh.


then you can do your first pass without any subtraction at all.


oh.


and then your second pass uh uh eliminates those most of those hypotheses by uh by having an improved improved version of the analysis.


okay.


so


so that was all i had for now.


do you want to go barry?


um so for the past uh week or two i've been just writing my uh formal thesis proposal.


um so i'm taking this qualifier exam that's coming up in two weeks.


and i i finish writing a proposal and submit it to the committee.


um


and uh should i should i explain uh more about what what i'm proposing to do and and stuff?


yes briefly.


yeah briefly.


so briefly i'm proposing to do a a new approach to speech recognition using um a combination of uh multi band ideas and ideas um about the uh acoustic phonetic approach to speech recognition.


um so i will be using these graphical models that um that implement the multi band approach to recognize a set of intermediate categories that might involve uh things like phonetic features or other other feature things that are more closely related to the acoustic signal itself.


um and the hope in all of this is that by going multi band and by going into these um intermediate classifications that we can get a system that's more robust to to unseen noises and situations like that.


and so


some of the research issues involved in this are um one what kind of intermediate categories do we need to classify.


um another one is um what what other types of structures in these multi band graphical models should we consider in order to um combine evidence from the sub bands.


and uh the third one is how do we how do we merge all the uh information from the individual uh multi band classifiers to come up with word word recognition or or phone recognition things.


so basically that's that's what i've been doing.


so you've got two weeks huh?


and


i got two weeks to brush up on um presentation stuff.


and um


oh i thought you were finishing your thesis in two weeks.


but


oh that too.


yeah.


are you going to do any dry runs for your thing?


or are you just going to


yes.


i um i'm i'm going to do some.


would you be interested?


sure.


to help out.


okay.


thanks.


is that it?


that's it.


uh hhh.


let's see.


so we've got forty minutes left.


and it seems like there's a lot of material.


any suggestions about where we where we should go next?


huh.


uh


do you want to go sunil?


maybe we'll just start with you.


yeah.


but i actually stuck most of this in our last meeting with guenter.


um


but i'll just


so the last week uh i showed some results with only speechdat-car.


which was like some fifty six percent.


and uh i didn't


i mean i i found that the results


i mean i wasn't getting that results on the t.i. digit.


so i was like looking into why what is wrong with the t.i. digits.


why why i was not getting it.


and i found that the noise estimation is a reason for the t.i. digits to perform worse than the baseline.


so uh i actually picked


i mean the first thing i did was i just scaled the noise estimate by a factor which is less than one to see if that because i found there are a lot of zeros in the spectrogram for the t.i. digits when i used this approach.


so the first thing i did was i just scaled the noise estimate.


and i found


so the the results that i've shown here are the complete results using the new


well the the new technique is nothing but the noise estimate scaled by a factor of point five.


so it's just an ad hoc.


i mean some intermediate result because it's not optimized for anything.


so the results the trend the only trend i could see from those results was like the the the current noise estimation or the uh noise composition scheme is working good for like the car noise type of thing.


because i've the only only very good result in the t.i. digits is the noise car noise condition for their test a..


which is like the best i could see that


uh for any non stationary noise like babble or subway or any street some restaurant noise it's like it's not performing very well.


so


the so that that's the first thing i uh i could make out from this stuff.


and


yeah i think what is important to see is that there is a big difference between the training modes.


uhhuh.


yeah.


if you have clean training you get also a fifty percent improvement.


but if you have muddy condition training you get only twenty percent.


uh and in that twenty percent it's very inconsistent across different noise conditions.


huh.


so i have like a forty five percent for car noise.


and then there's a minus five percent for the babble.


and there's this thirty three for the station.


and so it's it's not it's not actually very consistent across.


the only correlation between the speechdat-car and this performance is the stationarity of the noise that is there in these conditions and the speechdat-car.


and uh


so the overall result is like in the last page.


which is like forty seven.


which is still very imbalanced because there are like fifty six percent on the speechdat-car and thirty five percent on the t.i. digits.


and


uh


the fifty six percent is like comparable to what the french telecom gets.


but the thirty five percent is way off.


i'm sort of confused.


but


this


i'm looking on the second page.


oh.


yep.


and it says fifty percent.


looking in the lower right-hand corner.


fifty percent relative performance.


for the clean training.


is that


and if you if you look


is that fifty percent improvement?


yeah.


for that's for the clean training and the noisy testing for the t.i. digits.


so it's improvement over the baseline mel cepstrum.


but the baseline mel cepstrum under those training doesn't do as well.


i i'm i'm trying to understand why it's it's eighty percent.


that's an accuracy number i guess.


yeah yeah yeah.


right?


so that's not as good as the one up above.


no.


but the fifty is better than the one up above.


so i'm confused.


uh actually the noise compensation whatever uh we are put in it works very well for the high mismatch condition.


i mean it's consistent in the speechdat-car.


and in the clean training also it gives it


but this fifty percent is is that the the high mismatch performance equivalent to the high mismatch performance in the speech.


so so since the high mismatch performance is much worse to begin with it's easier to get a better relative improvement.


yeah.


i do.


yeah yeah.


so by putting this noise.


yeah if we look at the figures on the right we see that the reference system is very bad.


oh.


the reference drops like a very fast.


oh oh oh oh oh oh.


like for clean clean training condition.


i see.


nnn.


this is this is t.i. digits we're looking at?


this whole page is t.i. digits?


or this is


oh yeah.


it's not written anywhere.


yeah it's t.i. digits.


the first spreadsheet is t.i. digits.


huh.


how does clean training do for the uh car?


stuff?


the car?


oh.


still it still uh that that's still consistent.


i mean i get the best performance in the case of car which is the third column in the a. condition.


no.


i mean this is added noise.


i mean this is t.i. digits.


i'm sorry.


i meant in in the in the uh multi language uh uh finnish and


uh


this is next next page.


that's the next next spreadsheet is.


so that is the performance for italian finnish and spanish.


training condition.


oh right.


so clean corresponds to high mismatch.


yeah.


and increase


that's increase


improvement.


that's percentage increase is the percentage improvement over the baseline.


it's it's a


so that's


which means decrease in word error rate?


okay.


so percentage increase means decrease.


yeah yeah.


the the there was a very long discussion about this on on the on the uh amsterdam meeting.


how to how to calculate it then.


there's there's a


i i i guess you are using finally this the scheme which they


which is there in the spreadsheet.


i'm not changing anything in there.


huh.


all right.


so


uh


yeah.


so all the h.m. numbers are very good.


in the sense they are better than what the french telecom gets.


but the the only number that's still i mean which stephane also got in his result was that medium mismatch of the finnish.


which is very which is a very strange situation where we used the we changed the proto for initializing the h.m.m.


i mean this this is basically because it gets stuck in some local minimum in the training.


that seventy five point seven nine in the finnish mismatch which is that the eleven point nine six what we see.


uhhuh.


so we have to jiggle it somehow?


so we start with that different proto and it becomes eighty eight.


which is like some fifty percent improvement.


wait a minute.


start with a different what?


different prototype.


which is like a different initialization for the uh transition probabilities.


it's just that right now the initialization is to stay more in the current state.


which is point four point six right?


yeah.


and if it changes to point five point five which is equal for transition and self loop where it becomes eighty eight percent.


well but that involves mucking with the back end.


which is not allowed.


we can't do it.


huh.


so


i mean it uh like


it is well known this this medium match condition of the finnish data has some strange effects.


very


it has a very few at uh actually uh i mean words also.


i mean that is


that too.


it's a very very small set actually.


uhhuh.


there is a a there is a lot of uh there are a lot of utterances with music in with music in the background.


so there is


yeah.


yeah yeah yeah.


it has some music also.


i mean very horrible music like like 4x.


i know.


so maybe for that one you need a much smarter v.a. d?


huh


if it's music.


uh


so


that that's the that's about the results.


and uh


the summary is like


okay.


so there are the other thing what i tried was which i explained in the last meeting is using the channel zero for uh for both dropping and estimating the noise.


and that's like just to get a feel of how good it is.


i guess the fifty six percent improvement in the speechdat-car becomes like sixty seven percent.


like ten percent better.


but that's that's not a that's a cheating experiment.


that's just


but the but the uh forty seven point nine percent which you have now that's already a remarkable improvement in comparison to the first proposal.


yeah.


so we had forty four percent in the first proposal.


uhhuh.


we have a big


so the major improvement that we got was in all the high mismatch cases.


because all those numbers were in sixties and seventies.


because we never had any noise compensations.


huh.


so that's where the biggest improvement came up.


not much in the well match and the medium match and t.i. digits also right now.


so this is still at three or four percent improvement over the first proposal.


yeah so that's good.


so


then if we can improve the noise estimation then it should get better.


i i started thinking about also


i mean yeah uh i discovered the same problem when i started working on uh on this aurora task almost two years ago.


that you have the problem with this


at the beginning we had only this condition training of the t.i. digits.


yeah.


and uh i i found the same problem.


just taking um what we were used to use i mean uh some type of spectral subtraction you get even worse results than the basis.


and uh


i i tried to find an explanation for it.


yes.


stephane also has the same experience of using the spectral subtraction right?


huh.


uhhuh.


so here here i mean i found that it's if i changed the noise estimate i could get an improvement.


so that's so it's something which i can actually pursue is the noise estimate.


and


yeah i think what you do is in when when you have the the this multi condition training mode um then you have then you can train models for the speech for the words as well as for the pauses where you really have all information about the noise available.


it was surprising.


at the beginning it was not surprising to me that you get really the best results on doing it this way.


i mean in comparison to any type of training on clean data and any type of processing.


but it was


so


it it seems to be the best what what what we can do in this moment is multi condition training.


and when we now start introducing some some noise reduction technique we we introduce also somehow artificial distortions.


yeah.


and these artificial distortions uh i have the feeling that they are the reason why why we have the problems in this multi condition training.


that means the h.m.m.'s we trained they are they are based on gaussians.


and on modeling gaussians.


and if you


can i move a little bit with this?


and if we introduce now this this spectral subtraction or wiener filtering stuff


so usually what you have is maybe um


i'm i'm showing now an envelope.


um


maybe you'll for this time.


so usually you have maybe in clean condition you have something which looks like this.


and if it is noisy it is somewhere here.


and then you try to subtract it or wiener filter or whatever.


and what you get is you have always these problems that you have this these these these zeros in there.


and you have to do something if you get these negative values.


i mean this is your noise estimate and you somehow subtract it or do whatever.


uh and then you have


and then i think what you do is you introduce some some artificial distribution in this.


uh


in in the models.


i mean you you train it also this way.


but somehow there is there is no longer a a gaussian distribution.


it is somehow a strange distribution which we introduce with these artificial distortions.


and and i was thinking that that might be the reason why you get these problems in the especially in the multi condition training mode.


uhhuh.


yeah yeah.


yeah.


the the models are not complex enough to absorb that additional variability that you're introducing.


thanks adam.


yes.


well that's


so


i also have the feeling that um the reason why it doesn't work is yeah that the models are much are um not complex enough.


because i actually i always had a good experience with spectral subtraction.


just a straight spectral subtraction algorithm when i was using neural networks big neural networks which maybe are more able to model strange distributions.


and


but


then i tried the same exactly the same spectral subtraction algorithm on these aurora tasks.


and it simply doesn't work.


huh.


it's even it uh hurts even.


we probably should at some point here try the tandem the the the system two kind of stuff with this with the spectral subtraction for that reason.


uhhuh.


because again it should do a transformation to a domain where it maybe looks more gaussian.


yeah.


i i was


just yesterday when i was thinking about it um what what we could try to do or do about it


i mean if you if you get at this in this situation that you get this this negative values and you simply set it to zero or to a constant or whatever if we if we would use there a somehow um a random generator which which has a certain distribution not a certain yeah a special distribution we should see we we have to think about it.


it's


and that we so introduce again some natural behavior in this trajectory.


very different from speech.


still i mean it shouldn't confuse the


yeah i mean similar to what what you see really in in the real um noisy situation.


okay.


or in the clean situation.


but but somehow a a natural distribution.


but isn't that again sort of the idea of the additive thing?


if it as as we had in the j. stuff.


i mean basically if if you have random data um in in the time domain then when you look at the spectrum it's going to be pretty flat.


uhhuh.


and and


uh


so just add something everywhere rather than just in those places.


it's just a constant right?


yeah.


i think


it's it's just especially in these segments.


i mean you introduce um very artificial behavior.


and


well see if you add something everywhere it has almost no effect up up up on on top.


and it and it and it has significant effect down there.


that was sort of the idea.


huh.


the that's true.


that those those regions are the cause for this those negative values or whatever you get.


so


i mean we we could uh we we could think how what what we could try.


i mean it it was just an idea.


yeah yeah.


uhhuh.


i mean we


i think when it's noisy people should just speak up.


to


if we look at the france telecom proposal they use some kind of noise addition.


they have a random number generator right?


and they add noise on the trajectory of uh the log energy only right?


oh they do.


oh.


yep.


c.c. zero and log energy also.


yeah.


um


but i don't know how much effect it this have.


but they do that.


now?


huh.


so it it it it it is somehow similar to what


i think because they have log energy.


and then just generate random number.


they have some kind of mean and variance.


and they add this number to to the log energy simply.


to the


the the log energy the after the clean cleaning up.


uhhuh.


so they add a random random noise to it.


to the just the energy or to the mel uh to the mel filter?


no.


only to the log energy.


only


yeah.


oh.


so it because i mean i think this is most interesting for the mel filters.


right?


or or f.f.t.'s.


one or the other.


but but they do not apply filtering of the log energy or what


like uh i mean


like like a spectral subtraction.


or


no their filter is not m. domain.


so they did filter their time signal.


i


and then what


and then they calculate from this the log energy.


then after that it is almost the same as the baseline system.


uhhuh.


and then the final log energy that they that they get that to the to that they add some random noise.


yeah but again that's just log energy as opposed to filter bank energy.


yeah.


huh.


so it's not the mel.


you know it's not the mel filter bank output.


these are log energy computed from the time domain signal.


not from the mel filter banks.


so


maybe it's just a way to decrease the importance of this particular parameter in the in the world feature vector.


did


if you add noise to one of the parameters you widen the distributions.


becomes flat.


the variance yeah reduces.


and


huh yeah.


eee-sss-uh.


so it could reduce the dependence on the amplitude and so on.


yeah?


yeah.


although


maybe.


so is uh is that about it?


uhhuh.


uh


or


so the


okay.


so the other thing is the i'm just looking at a little bit on the delay issue where the delay of the system is like a hundred and eighty millisecond.


so i just just tried another system i mean another filter which i've like shown at the end.


which is very similar to the existing uh filter.


only uh only thing is that the phase is is like a totally nonlinear phase.


because it's a it's not a symmetric filter anymore.


this is for the l.d. a?


so so this this is like so this makes the delay like zero for l.d.a.


because it's completely causal.


oh.


so


so i got actually just the results for the italian for that.


and that's like


so the fifty one point o. nine has become forty eight point o. six.


which is like three percent relative degradation.


uhhuh.


so i have like the fifty one point o. nine.


and


i don't know it fares for the other conditions.


so it's just like it's like a three percent relative degradation.


with the


but but is there is there a problem with the one hundred eighty milliseconds?


or


well this is


uh


yeah i mean i talked to to uh i uh i talked uh about it with with hynek.


i mean there is


this is


so so basically our our position is that um we shouldn't be unduly constraining the latency at this point.


because we're all still experimenting with trying to make the performance better in the presence of noise.


uh there is a minority in that group who is arguing who are arguing for um uh having a further constraining of the latency.


so we're just continuing to keep aware of what the tradeoffs are and you know what what do we gain from having longer or shorter latencies.


huh.


but since we always seem to at least get something out of longer latencies not being so constrained we're tending to go with that if we're not told we can't do it.


uhhuh.


what where was the um the smallest latency of all the systems last time?


the french telecom.


well france telecom was was was very short latency.


it's


and they had a very good result.


what what was it?


it was thirty five.


it was in the order of thirty milliseconds.


or


yeah.


thirteen?


thirty.


thirty four.


yeah so it's possible to get very short latency.


but again we're the the approaches that we're using are ones that take advantage of


i was just curious about where we are compared to you know the shortest that people have done.


but but i think this thirty milliseconds they they did it did not include the the delta calculation.


and this is included now.


you know?


so if they include the delta it will be an additional forty millisecond.


uhhuh.


i i don't remember the


they were not using the h.t.k. delta?


no they're using a nine point window.


which is like a four on either side.


nine point.


which is like


okay.


so.


huh.


they didn't include that.


yeah.


so


where does the compression in decoding delay comes from?


that's the way the the the frames are packed.


like you have to wait for one more frame to pack.


because it's the c.r.c. is computed for two frames always.


uhhuh.


well that they would need that forty milliseconds also.


right?


no.


they actually changed the compression scheme altogether.


so they have their own compression and decoding scheme.


and they i don't know what they have.


oh.


but they have coded zero delay for that.


because they i know they changed it.


their compression.


they have their own c.r.c.


their their own error correction mechanism.


so they don't have to wait more than one more frame to know whether the current frame is in error.


oh okay.


so they changed the whole thing so that there's no delay for that compression and part also.


huh.


uhhuh.


even you have reported actually zero delay for the compression.


i thought maybe you also have some different


no i think i i used this scheme as it was before.


okay.


uh.


we've got twenty minutes.


so we should probably try to move along.


uh did you want to go next stephane?


i can go next.


yeah.


oh.


it's


wait a minute.


we have to take


i think i'm confused.


well.


all right.


so you have one sheet.


this one is you don't need it.


uh


so you have to take the whole the five.


there should be five sheets.


okay.


i have four now.


because i left one with dave because i thought i was dropping one off and passing the others on.


so no.


we're not.


thanks.


please give me one.


uh we need one more over here.


okay maybe there's not enough for everybody.


i can share with barry.


but


yeah.


oh okay.


can we look at this?


so


yeah there are two figures showing actually the huh um performance of the current v.a.d.


so it's a neural network based on p.l.p. parameters.


uh which estimate silence probabilities.


and then i just put a median filtering on this.


to smooth the probabilities right?


um


i didn't use the the scheme that's currently in the proposal.


because i don't want to.


in the proposal


well in in the system we want to add like speech frame before every word and a little bit of of uh a couple of frames after also.


uh but to estimate the performance of the v.a.d. we don't want to do that.


because it would artificially increase the um the false alarm rate of speech detection.


right?


there is normally a figure for the finnish and one for italian.


and maybe someone has two for the italian.


because i'm missing one figure here.


no.


well.


well whatever.


uh yeah so one surprising thing that we can notice first is that apparently the speech miss rate is uh higher than the false alarm rate.


so


so so what is the lower curve and the upper curve?


it means


uhhuh.


yeah there are two curves.


yeah.


one curve's for the close talking microphone which is the lower curve.


uh okay.


and the other one is for the distant microphone.


which has more noise.


it's logical that it performs worse.


so as i was saying the miss rate is quite important.


uh which means that we tend to label speech as as a silence.


and


uh i didn't analyze further yet.


but i think it's it may be due to the fricative sounds.


which may be in noisy condition maybe label labelled as silence.


and it may also be due to the alignment.


because


well the reference alignment.


because right now i just use an alignment obtained from from a system trained on channel zero.


i checked it a little bit.


but there might be alignment errors.


um yeah.


like the fact that the the models tend to align their first state on silence and their last state on silence also.


so the reference reference alignment would label as speech some silence frame before speech and after speech.


this is something that we already noticed before.


when


huh


so this this could also explain uh the high miss rate maybe.


and and this this curves are the average over the whole database.


uh


so


yeah.


right.


huh.


um


and the different points of the curves are for five uh thresholds on the probability uh from point three to point seven.


so that threshold


uhhuh.


okay.


so the detection threshold is very


so the


the v.a. d?


yeah yeah.


there first a threshold on the probability that puts all the values to zero or one.


and then the median filtering.


so the median filtering is fixed.


you just change the threshold.


it's fixed.


yeah.


so going from channel zero to channel one uh almost double the error rate.


well so it's a reference performance that we can you know if we want to to work on the v.a.d. we can work on this basis.


and


is this is this v.a.d.a.m.l. p?


how how big is it?


it's a very big one.


i don't remember.


so three three hundred and fifty inputs.


uh


six thousand hidden nodes and two outputs.


okay.


uhhuh.


middle sized one.


uh ppp


i don't know you have questions about that or suggestions?


huh.


so.


it seems the performance seems worse in finnish.


which


well it's not trained on finnish.


it's worse.


it's not trained on finnish.


yeah.


what's it trained on?


i mean the m.l.p.'s not trained on finnish.


right.


oh oh.


sorry.


uh it's italian t.i. digits.


oh it's trained on italian?


uhhuh.


okay.


and


that's right.


and also there are like funny noises on finnish.


more than on italian.


i mean like music.


and um


yeah the yeah it's true.


so yeah.


we were looking at this.


but for most of the noises noises are


um


i don't know if we want to talk about that.


but


well the the car noises are below like five hundred hertz.


and we were looking at the music utterances.


and in this case the noise is more about two thousand hertz.


well the music energy's very low apparently.


yeah.


uh


uh from zero to two two thousand hertz.


so maybe just looking at this frequency range for from five hundred to two thousand would improve somewhat the v.a.d.


huh.


and


so there are like some some some parameters you wanted to use or something?


yes.


or yeah.


uhhuh.


uh the next


um


so is the is the is the training is the training based on these labels files which you take as reference here?


oh it's there.


when you train the neural net you


no.


it's not.


it's it was trained on some alignment obtained


uh for the italian data i think we trained the neural network on with embedded training.


so re estimation of the alignment using the neural network i guess.


that's right?


yeah.


we actually trained uh the on the italian training part.


we we had another system with


so it was a a phonetic classification system for the italian aurora data.


it must be somewhere.


for the aurora data that it was trained on it was different.


like for t.i. digits you used a a previous system that you had i guess.


what


no it


yeah yeah.


that's true.


so the alignments from the different database that are used for training came from different system.


then we put them together.


well you put them together and trained the v.a.d. on them.


huh.


uh


but did you use channel did you align channel one also?


or


i just took their entire italian training part.


yeah.


so it was both channel zero plus channel one.


so


so the alignments might be wrong then on channel one right?


on one.


possible.


so we might


we can do a realignment.


that's true.


at least want to retrain on these alignments.


which should be better because they come from close talking microphone.


the that was my idea.


i mean if if it if it is not the same labeling which is taking the spaces.


okay.


yeah possible.


huh.


i mean it


yeah.


so the system


so the v.a.d. was trained on maybe different set of labels for channel zero and channel one.


uhhuh.


and


was the alignments were were different for certainly different because they were independently trained.


we didn't copy the channel zero alignments to channel one.


but for the new alignments what you generated you just copied the channel zero to channel one right?


right.


um


and uh hhh


actually when we look at at the v.a.d. for some utterances it's almost perfect.


i mean it just dropped one frame.


the first frame of speech.


or


so there are some utterances where it's almost one hundred percent v.a.d. performance.


huh.


uh


but yeah.


huh


yep.


so the next thing is um i have the spreadsheet for three different system.


but for this you only have to look right now on the speechdat-car performance.


uh because i didn't test


so i didn't test the spectral subtraction on t.i. digits yet.


uh so you have three sheets.


one is the um proposal one system.


actually it's not exactly proposal one.


it's the system that sunil just described.


um


but with uh wiener filtering from um france telecom included.


um so this gives like fifty seven point seven percent uh uh error rate reduction on the speechdat-car data.


and then i have two sheets where it's for a system where


so it's again the same system.


but in this case we have spectral subtraction.


with a maximum overestimation factor of two point five.


uh there is smoothing of the gain trajectory with some kind of uh low pass filter.


which has forty milliseconds latency.


and then after subtraction um i add a constant to the energies.


and i have two cases where the first case is where the constant is twenty five d.b. below the mean speech energy.


and the other is thirty d.b. below.


and for these two system we have like fifty five point uh five percent improvement.


and fifty eight point one.


so again it's around fifty six fifty seven.


uh


because i notice the t.i. digits number is exactly the same for these last two.