i assume but they're all doing communicator stuff.


or


oh okay.


so


huh.


and it's part of the talk itself.


so it's really not a big part.


yeah i see.


i mean the talk's only twenty minutes.


it's just going to be a couple of


so you know all this time we spent on it's going to be thirty seconds of screen time.


i know.


oh my gosh.


right uhhuh


oh!


but it's a good tool to have.


yeah


the the the meeting i r stuff is actually


actually both of them.


you know because we're going to want to be playing with the


uhhuh.


uh


help my brain.


with the uh


prosody?


prosody stuff anyway.


um so actually at some point we should talk to users of that technology who are actually going to be using it.


and see what they would like to see.


yeah have you guys seen the display?


have you guys seen either of the demos?


they're pretty cool.


no.


nope.


like the prosody demo we basically loaded up um word alignments.


so instead of having um utterances in the bottom it's like just word by word.


so you can see exactly where the overlaps are.


oh.


and then um adam did something where um we converted these feature files into wave files.


so you can display them.


display the features.


instead of the wave file?


yeah instead of the wave file.


oh okay.


and then using multi wave you can add a file and play that sound.


uhhuh.


play the audio corresponding to that um um feature file.


okay.


and it's all like aligned and stuff.


uh great.


so you see like the f zero contour?


yeah so you see the pitch contours.


and you and when you hit play like there's a line that goes through it.


you know and it's all aligned with like the the audio and the feature file.


oh that's great.


audio.


yeah.


and the stylized f naught features are pretty cool.


huh.


so these does a piece wise linear.


and so normally if you look at an f naught track it jumps all over the place.


this is a nice smooth one.


oh neat.


and it really does a good job.


and there's something like a median filter in it or something?


and uh and it's it's a linear fit or whatever.


yeah there's a


it's a combination.


there's a median filtering and then there's a piece wise linear fit based on some criteria.


i'm not sure.


okay.


uh


actually could you email me a reference to that paper if you have it?


yeah sure.


because i'm sort of curious how what criteria it has.


i should email morgan about it too.


because he was asking about that.


and yeah.


i mean so there's obviously a trade off between the number of knots you pick and the best fit.


i mean so if you're doing a piece wise linear you have to figure out how many pieces you want.


yeah.


and so there's a trade off.


because the best the best mean squared fit would be with an infinite number of knots.


right well then it's not a linear fit.


yeah yeah.


you're not fitting anything then.


well sure it is.


but


well


so so you there's a trade off.


yeah it


it's kind of a useless one.


and i'm wondering what criteria they use.


yeah i actually don't know off the top of my head.


uhhuh.


was this code that you downloaded to do this?


or was it


this was um kemal sonmez at s r i.


who's a co worker with uh liz.


huh.


dot com.


at s r i dot com?


yeah did i just


no i you said at s r i.


no


so i had to add the dot com.


it sounded like a u r l the way you were saying it.


oh oh.


yeah.


sorry.


um yeah but so kamal has uh


he did a paper on this for some speech verification uh project.


and we basically extrapolated what he what he did for that work and applied it to our files.


and that's how we do that's how we get our feature files and everything for all the prosody.


so what i mean is like the implementation that you used was something that you coded up here?


or you grabbed code from somewhere else?


that


no we grabbed most of it.


like we grabbed the linear fitting stuff.


uh.


i see.


we did all the alignments.


and all the matching and stuff.


uhhuh.


and


but the actual coding of the


the math was done somewhere else.


these are our f zero candidates which i'm supposed to use for the synthesis thing?


i suppose?


uh so that's what you were working on.


right?


right that's the same software i used to get the data.


yeah okay.


okay.


i i gave you a couple examples.


yeah i didn't have time to to look at that.


huh.


so


right right.


yeah.


so other topic is uh the i b m transcription status stuff.


so we've sent a few to them.


we've gotten a few back.


um


i sort of wish jane was were here.


because i think she gave


didn't she give one of them to one of the transcribers?


yeah it's done now.


and how was it?


she said it was much faster.


great.


it really helped a lot to have that.


so um you know there were a


much faster than doing everything alone?


hand.


from scratch.


okay.


yeah so


from end to like from the beginning of the process to the end of the process?


no no so from the point where we take it it's faster.


well but that's what we wanted.


yeah.


right?


we knew it would be longer to have multiple groups.


ooo yeah.


so it's sort of like doing things in parallel.


yeah right.


right.


you know so if you know if they can be working on that while we're working on the other things then when we get them in they don't have to do that one from scratch basically.


well pipelined.


uhhuh.


the idea is that we're taking less time of the linguists.


of the transcribers.


so


yeah right now it is taking a bit of time to turn around the meetings from i b m.


there's there's several problems.


one is um we were giving them these large files.


which they then had to split up and put onto cassette tapes.


so either ninety minute or sixty minute tapes.


oh wow.


so they had to try to find an appropriate place to break this large file that we gave them.


so one of the things that we did is adam made a modification to his script that generates this so you can tell it that you want uh basically uh chunks that are no more than either thirty or forty five minutes.


and then we can give those to i b m.


and it'll make it easier for them to put them on tape.


um then the other thing is that these pool of transcribers that they're using um sort of are not dedicated to to our project.


but are in use for all i b m projects.


and i b m has recently giving given them uh a whole bunch of stuff.


so it's going to sort of delay us.


we might want to re visit hiring our own external transcription company.


because i think that would be cheaper than having our people do it.


but do it with the same process with these beep files.


huh that's an interesting idea.


i mean because originally i had discounted that.


because it was just going to be too hard.


but now we have this procedure with the beep files worked out.


yeah so the only difference would be we would be putting the stuff on tape versus i b m.


then maybe it would be okay.


yeah.


yep.


and then we could send if if we had that process down of putting the audio onto tape


we can send i b m the tapes or we can send this transcription place the


and we might want to do that anyway.


because it seems like that's being a bottleneck and that's silly.


it shouldn't be a bottleneck.


i mean it would once we got it set up it it


well


so i don't know where the bottleneck was.


was the bottleneck in breaking this large file up into the appropriate size chunks or was it actually putting it onto the tape?


it may well have been partially that you know.


because they tried it.


and then they would have to listen to the tape and find out where it bleeps.


exactly.


and go forward in the file et cetera et cetera.


but uh


yeah.


it just seems like it's not hard for us to do.


and that would just make things go faster.


yeah so how do we do that?


how would we put


get a tape deck.


plug it in to the computer.


plug it into the sound card.


hit record.


on the tape deck.


and just do it all all uh analog.


so plug so like the headphone output in the back of the computer?


well line out.


line out yeah.


line out into the line in on a tape player.


uhhuh.


huh.


yep.


and


oh well that sounds pretty easy.


so i mean that's the easy way.


they might have a more complex set up at i b m.


i've seen digital digital analog tapes where it's controlled by the computer.


yeah but the easy way is


but but we don't need to do that.


yeah i know.


no.


yeah.


the the disadvantage of doing it this way is it's real time.


so you would have to sit there for an hour while it's recording.


yeah but if it was at your desk or something


but


yeah but you


yeah yeah.


right it's not a big deal.


so what's the turn around period right now from i b m?


well they there was


should we be discussing this


okay.


so th they um they gave they've given us


since we started doing this new uh beep format with beep number beep


uhhuh.


they've given us uh i think two full uh transcripts.


they have a third one.


and brian just sent me a note saying oh that third one somehow slipped through the cracks of their of the uh transcriptionists' uh company.


huh.


and so there's going to be delay of a week before we get that third one back.


and i've already given him three more to work on.


but he said those new three won't get done until this big chunk of data has been processed by the company.


and the transcribers here are still uh are working though.


yeah yeah they're working away.


they're just doing their own thing like while they're waiting.


they're just doing


right right.


so what we do is uh when i go to um select meetings for i b m i go into the um


well into the the uh master sheet here with the statuses


master.


and i pick ones that thilo has um pre segmented.


and i change the status to i p i b m.


so that you know when jane or whoever goes to select the next meeting they won't choose one of those.


and um


so i'm not picking huge chunks right now of meetings for them.


so


we should at least ask um brian if there are any particular requirements for putting things on tape.


uhhuh.


but


is there a particular type of tape that they want to use or anything else like that?


uhhuh that's a good idea.


and uh


and then we should see how much of a pain it is for one of us.


i mean the the other problem is you would want to do it on an unloaded machine.


right?


because you wouldn't want it to start stuttering.


yep.


yeah.


huh.


and that means that that's more of a problem.


and all


because then you have to be there.


and all of the p cs are connected to the network too here?


what


or uh


right.


you're just talking about playing it and recording it like real time?


bad


oh.


so when that thing when it stutters like that is it because there's a delay getting the data off a disk or is it because there's a delay in the d to a?


uhhuh.


it could be either.


i mean there are lots of different places.


okay.


it can be delay on the disk.


or it can be just load gets high on the machine.


yeah.


um either i o load or processor load.


i see.


you know it's a problem with these multi user systems is that lots of processes are running.


and you don't really have control.


maybe we could do it from a a p c or something.


but they are also on the networks.


so


if you used a line out too you have to worry about if you click around and you hit a beep or something it'll beep onto onto the tape too.


maybe.


that's true.


it'll


that's right so you really do want to be using a machine that's not.


uhhuh.


and you have to record every channel separately.


no because we're doing the beep files.


the beep files.


well you know i i think david might have a bunch of old p cs.


oh okay.


because he's replacing them with these new ones.


maybe we could get one.


and just dedicate it to doing this.


not even hook it


well i guess we need to hook it to the network.


we


so we can get the files to it.


but


yep.


uh


or you could just you could have it no active connection.


just s c p them.


yeah right.


and then just use that machine to hook up to the tape.


actually it wouldn't be bad to have that plus put on that machine a c d burner.


yeah.


we don't need a powerful machine to do these things.


right?


no.


so one of these old pentiums that we have


huh!


should talk to him about that.


or even a d v d burner.


a d v d burner yeah.


uhhuh.


i guess the real question is where would we put it and who would do it?


i guess whoever records the meeting.


i mean it's not a big deal.


but it's yet another thing you have to do.


yeah yeah.


that's why i wanted to avoid having to do that if we could.


but


it's just another


they couldn't use c ds?


no because the transcription company does what they do.


they have a um a device that you put the cassette in.


and it has a foot pedal that lets you go back and forth and stuff like that.


oh i see.


i see.


so


i guess they haven't got a


actually there are a few i saw a web based one at one point.


web based?


where you could send them audio files over the web.


and they send you back text files over the web.


really?


i should look that up again.


can you send them huge audio files?


you can.


but they never will be able to read them.


yeah.


i assume that you can send them anything that they can access with h t t p.


huh!


well they must be dealing with large amounts of data if they're transcribing anyway.


that's interesting.


i mean people probably aren't sending them you know an utterance or two.


well that's true.


are how big are these files that we send to brian?


uh pfff!


half a gig.


that big?


oh no wait.


i thought it was like a hundred megs.


they're the beep files.


yeah it's a hundred meg.


that's right.


so they're smaller.


yeah.


that's still pretty dang big.


yes.


send it via email.


and we are sending them the compressed ones.


right?


i don't know.


whatever you generate


yeah the last ones you did were compressed.


were shortened.


wonder if the ones before that were shortened.


the beep files actually the shorten doesn't save a whole lot.


because it's all


because the whole point is that it's extracting parts that are loud anyway.


it's all speech.


yeah yeah.


uhhuh.


huh!


so


huh.


i think it only saves like half.


only.


okay so i think we just got to let things slide for now.


and see see what's going to happen.


yeah.


nothing else really to do.


um


we have new equipment that i'm going to try to set up tomorrow.


new wireless.


what will that do?


and so uh


is that going to


give us a couple more wireless channels.


oh so we can get rid of the uh


yep.


oh that'll be nice.


but it involves rewiring.


and so if i could borrow someone tomorrow afternoon to help my brain


i should be around.


you know just have someone to bounce the instructions off of.


and make sure i don't do anything too stupid.


yeah sure.


did i have another topic on the on the list?


experiments?


isn't that next week?


that's


yeah i guess that's probably next


yeah.


these are next week.


what did i have on the agenda i mailed out?


i think it was


oh yeah i don't know.


the demo.


something about


the demo was one of them.


demo and i b m i think.


demo.


yeah that's it.


yeah i think that's it.


that was it.


short meeting?


short meeting.


does anyone have anything else?


oh i made a


oh but that's not specific to meeting recorder.


i was going to say i took some a suggestion of adam's.


and i um created a c g i script that lets you see the current status of the speech disks.


uhhuh.


so if you go to the speech local web page and then there's a disk status page.


you can click on that.


and it'll query abbott.


and give you a a list of all the disks that we have.


and then uh what's on each one.


and then it shows how big the disk is.


and what percent full it is.


cool.


so


and that's you know each time you go there it's updated.


so the new disks are installed now?


so what what


or


the new disks are installed.


if you need space just let me know.


i have to create the appropriate sub directory.


so we're kind of trying to keep even the scratch ones we're keeping a little organized where we put a a u doctor speech data.


uhhuh.


and then i'll either create a sub directory for like if it's meeting recorder project or hub five or whatever.


so we


i just have to create those sub directories.


okay.


do we have new d d disks also?


and


no.


darn.


nnn.


so it's all just scratch.


it's all scratch.


yeah.


how are we doing on


uh d e which is where we're putting all the meeting stuff


no d e is full.


we must be doing it on d f.


no it's d e.


d e i think.


d e's not full.


yeah?


d d is full.


d d's full.


that was the first one.


or that was the last one that filled up.


and d e still had like


okay i guess i'm just one off.


last time i looked it was like seven gigs or something.


okay.


so


and each meeting is roughly half a gig.


half.


and so


so that means we're getting pretty close.


we're getting close.


yeah.


shoot!


are we just going to keep recording?


like at some point


morgan said something about stopping at the end of the year.


like when we get around a hundred meetings.


um because there's going to be data coming from u w hopefully by then.


because


i just hate not to get data.


how


yeah it feels funny.


huh?


you're a data fiend.


if we have the capability.


yeah it's like you build this room up for so long.


and then it's just like you stop.


now that we're used to recording it's like uh well why not keep going?


i the other thing we could do is stop doing the regular ones.


and try to convince other topics to come in and do some.


because i think that would be nice.


uhhuh.


we have right now like seventy five or seventy six hours of meetings.


yeah the other thing we were talking about also is once we once the disk has been backed up we don't actually need it online.


so another option is to copy it to tape manually.


so we have the back up copy and the tape copy.


the archive copy and the back up copy.


and then just take them off line.


but aren't we still using the uh compressed


once we have the expanded versions.


so if we have a ton of scratch disk maybe the thing to do is leave those up.


oh.


yeah.


and then use and then not necessarily have the original data online.


yeah we need to figure something out.


because the abbott is basically


full.


we can't add anymore disks to it.


we can increase the size of the disks that are on it.


but that will only take us so far.


so um


david had the suggestion about you know new servers and things like that.


and so i'm going to see if we can talk to morgan about getting some money for a new disk server.


and uh


well that would be pretty tight in the machine room.


yeah it would have to replace abbott basically.


eesh!


well he david's planning to get new servers anyways for all of the main servers.


uhhuh.


i think he wants to sort of go with the same type of machine for all of these.


but um


yeah it's full in there.


have you seen it recently with all of the new machines that we got


uh the sun blade one hundreds.


they're like stacked up on benches and things all over.


it's really crowded.


so


but they are not yet accessible?


the new machines.


oh yeah yeah yeah.


yeah they are?


they're in the yeah if you do a p make


oop!


what are they called?


what are they named?


yeah.


oh i've forgotten now.


um


i assume they're in the p make pool.


yeah they're in the p make pool.


so if you do a reginfo minus show attribute and


uh


a what?


a


p make stuff.


p make magic.


yeah


or customs magic.


oh god!


you can get a list of all the machines that are available to p make by using the command reginfo.


okay.


reginfo okay.


minus show a t t r.


maybe you can send me that.


yeah.


and then um


if you do man customs it has almost all of that.


and i i'm not sure what attributes we attach to the sun blade one hundreds.


but it could be s.b. one hundred.


and so if you query for all machines that have that attribute then you should see the names of the new machines.


but


do we have an attribute for non interactive machines?


because i was thinking that would be a good one to have.


non interactive


ones that aren't on people's desks.


well there's sort of.


there's this one called there's the attribute called what's that called uh uh no evict.


which is roughly that.


right.


it's although it's like my machine has it on my desk.


so i do get jobs running on my machine all the time.


um but uh


so so that's that's sort of what it is.


yep.


uhhuh.


anyway.


shall we do digits?


sure.


okay.


so uh i also copied uh the results that we all got in the mail i think from uh from o g i.


and we'll go go through them also.


so where are we on on uh our runs?


uh uh we so as i was already said we we mainly focused on uh four kind of features.


excuse me.


the p l p the p l p with j rasta the m s g and the m f c c from the baseline aurora.


uh and we focused for the the test part on the english and the italian.


um we've trained uh several neural networks on


so on the t i digits english and on the italian data.


and also on the broad uh english uh french and uh spanish databases.


huh so there's our result tables here for the tandem approach.


and um actually what we we observed is that if the network is trained on the task data it works pretty well.


chicken on the grill.


our our uh there's a we're pausing for a photo.


try that corner.


how about over from the front of the room?


yeah it's longer.


we're pausing for a photo opportunity here.


uh uh


so


oh wait wait wait wait wait wait.


get out of the


yeah.


hold on.


okay.


let me give you a black screen.


he's facing this way.


what?


okay this this would be a good section for our silence detection.


uhhuh.


um oh.


musical chairs everybody.


okay so um you were saying about the training data.


so if the network is trained on the task data um tandem works pretty well.


and uh actually we have uh


results are similar only on


do you mean if it's trained only on on data from just that task?


that language?


just that task.


but actually we didn't train network on uh both types of data.


i mean uh phonetically


phonetically balanced uh data and task data.


we only did either task task data or uh broad data.


huh.


um yeah.


so


so how


i mean clearly it's going to be good then.


so what's


but the question is how much worse is it if you have broad data?


i mean my


from what i saw from the earlier results uh i guess last week was that um if you trained on one language and tested on another say that the results were were relatively poor.


huh yeah.


but but the question is if you train on one language but you have a broad coverage and then test in another does that is that improve things in comparison?


if we use the same language?


no no no.


different


so um if you train on t i digits and test on italian digits you do poorly let's say.


uhhuh.


i don't have the numbers in front of me.


but yeah but i did not uh do that.


so i'm just imagining.


so you didn't train on timit and test on on italian digits say?


we no we did four four kind of of testing actually.


the first testing is with task data.


so with nets trained on task data.


so for italian on the italian speech.


the second test is trained on a single language um with broad database.


but the same language as the task data.


okay.


but for italian we choose spanish which we assume is close to italian.


the third test is by using um the three language database.


and the fourth is


which in


it has three languages.


that's including the the the


this includes


the one that it's


yeah.


but not digits.


i mean it's


right.


the three languages is not digits.


it's the broad data.


and the fourth test is uh excluding from these three languages the language that is the task language.


oh okay.


yeah so that is what i wanted to know.


i just wasn't saying it very well i guess.


uh yeah.


so um for uh t i digits for


example uh when we go from t i digits training to timit training uh we lose uh around ten per cent.


uh the error rate increase of of of ten per cent relative.


relative.


so this is not so bad.


and then when we jump to the multilingual data it's uh it become worse.


and well around uh let's say twenty twenty per cent further.


about how much?


so


yeah.


twenty per cent further?


twenty to to thirty per cent further.


and so remind me.


the multilingual stuff is just the broad data.


right?


it's not the digits.


so it's the combination of two things there.


it's removing the task specific training and it's adding other languages.


yeah yeah.


okay.


but the first step is already removing the task specific from from


already.


right right right.


and we lose


so they were sort of building here.


uh so basically when it's trained on the the multilingual broad data um or number.


so


the the ratio of


our error rates uh with the baseline error rate is around uh one point one.


yes and it's something like one point three of of the uh


if you compare everything to the first case at the baseline.


you get something like one point one for the for the using the same language but a different task.


and something like one point three for three three languages broad stuff.


no no no.


uh same language we are at uh for at english at o point eight.


so it improves compared to the baseline.


but


let me


i i i'm sorry.


task data


we are


i i i meant something different by baseline.


yeah.


so let me let me


um so um


huh.


okay fine.


let's let's use the conventional meaning of baseline.


i i by baseline here i meant uh using the task specific data.


oh yeah the yeah okay.


but uh uh because that's what you were just doing with this ten per cent.


so i was just i just trying to understand that.


sure.


so if we call a factor of just one just normalized to one the word error rate that you have for using t i digits as as training and t i digits as test


uhhuh.


uh different words i'm sure.


but but uh uh the same task and so on.


if we call that one then what you're saying is that the word error rate for the same language but using uh different training data than you're testing on say timit and so forth it's one point one.


yeah it's around one point one.


right.


and if it's


yeah.


you do go to three languages including the english it's something like one point three.


that's what you were just saying i think.


uh more actually.


one point four?


if i yeah.


so it's an additional thirty per cent?


what would you say?


around one point four.


okay.


and if you exclude english from this combination what's that?


if we exclude english um there is not much difference with the data.


with english.


so yeah.


aha!


that's interesting that's interesting.


do you see because uh


uh


so no that that's important.


so what what it's saying here is just that


yes.


there is a reduction in performance when you don't um have the when you don't have um


task data.


wait a minute.


the


huh.


no actually it's interesting.


so it's so when you go to a different task there's actually not so different.


it's when you went to these


so what's the difference between two and three?


between the one point one case and the one point four case?


i'm confused.


it's multilingual.


yeah.


the only difference it's is that it's multilingual.


um


because in both in both both of those cases you don't have the same task.


yeah sure.


so is is the training data for the for this one point four case


does it include the training data for the one point one case?


uh yeah.


yeah a fraction of it.


a part of it.


how how much bigger is it?


um it's two times.


yeah um


actually


um the english data no the multilingual databases are two times the broad english data.


we just wanted to keep this well not too huge.


so


so it's two times.


but it includes the but it includes the broad english data.


i think so.


do you


and the broad english data is what you got this one point one with.


so that's timit basically.


right?


yeah.


uhhuh.


so it's band limited timit.


this is all eight kilohertz sampling.


right.


so you have band limited timit gave you uh as good as a result as using t i digits on a t i digits test.


okay.


huh.


um and um


but when you add in more training data but keep the neural net the same size it um performs worse on the t i digits.


okay now all of this is this is noisy t i digits i assume.


yup.


both training and test?


um


okay well we we we may just need to uh


so i mean it's interesting that going to a different different task didn't seem to hurt us that much.


and going to a different language um


it doesn't seem to matter


the difference between three and four is not particularly great.


so that means that whether you have the language in or not is not such a big deal.


it sounds like um uh we may need to have more of uh things that are similar to a target language.


or i mean you have the same number of parameters in the neural net.


you haven't increased the size of the neural net.


and maybe there's just just not enough complexity to it to represent the increased variability in the in the training set.


that that could be


um so what about


so these are results with uh that you're describing now that they are pretty similar for the different features or or uh


uh let me check.


uh


yeah.


so this was for the p l p.


um the


for the p l p with j rasta the the we


this is quite the same tendency with a slight increase of the error rate uh if we go to to timit.


and then it's it gets worse with the multilingual.


um yeah.


there there is a difference actually with between p l p and j rasta.


is that j rasta seems to perform better with the highly mismatched condition but slightly slightly worse for the well matched condition.


huh.


i have a suggestion actually.


even though it'll delay us slightly.


would would you mind running into the other room and making copies of this?


because we're all sort of


yeah yeah.


if we if we could look at it while we're talking i think it'd be


okay.


uh uh i'll i'll sing a song or dance or something while you do it too.


all right.


so um


go ahead.


uh while you're gone i'll ask some of my questions.


yeah.


um


uh this way and just slightly to the left.


the um what was was this number forty?


or it was roughly the same as this one he said?


when you had the two language versus the three language.


that's what he was saying.


that's where he removed english.


right?


right.


it sometimes actually depends on what features you're using.


but but it sounds like


um but he uhhuh.


i mean that's interesting.


because it it seems like what it's saying is not so much that you got hurt uh because you uh didn't have so much representation of english.


because in the other case you don't get hurt any more.


at least when it seemed like uh it it might simply be a case that you have something that is just much more diverse.


uhhuh.


but you have the same number of parameters representing it.


i wonder.


were um all three of these nets using the same output?


this multi language uh labelling labelling?


he was using uh sixty four phonemes from sampa.


okay okay.


yeah.


so this would from this you would say well it doesn't really matter if we put finnish into the training of the neural net if there's going to be you know finnish in the test data.


right?


well it's it sounds i mean we have to be careful.


because we haven't gotten a good result yet.


and comparing different bad results can be tricky.


huh.


but i i i i think it does suggest that it's not so much uh uh cross language as cross type of speech.


it's it's uh


um but we did


oh yeah.


the other thing i was asking him though is that i think that in the case


you you do have to be careful.


because of compounded results.


i think we got some earlier results in which you trained on one language and tested on another.


and you didn't have three but you just had one language.


so you trained on one type of digits and tested on another.


wasn't there something of that?


where you say you trained on spanish and tested on on t i digits?


or the other way around?


something like that?


no.


i thought there was something like that that he showed me last week.


we'll have to wait till he gets back.


until we get


yeah that would be interesting.


um this may have been what i was asking before stephane.


but but um wasn't there something that you did where you trained on one language and tested on another?


i mean no no mixture but just


i'll get it for you.


uh no no.


we've never just trained on one


training on a single language you mean?


and testing on the other one?


yeah.


not yet.


uh no.


so the only task that's similar to this is the training on two languages.


and that


but we've done a bunch of things where we just trained on one language.


right?


i mean you haven't you haven't done all your tests on multiple languages.


either this is test with uh the same language but from the broad data.


or it's test with uh different languages.


also from the broad data.


excluding the


the early experiment that


so it's it's three or three and four.


did you do different languages from digits?


you mean training digits on one language and using the net to recognize on the other?


digits on another language.


no.


see i thought you showed me something like that last week.


you had a you had a little


uh no i don't think so.


um what


these numbers are uh ratio to baseline?


so i mean what's the


so


this this chart this table that we're looking at is um is all testing for t i digits?


or


bigger is worse.


so you have uh basically two uh parts.


this is error rate i think.


ratio.


no no.


the upper part is for t i digits.


yeah yeah yeah.


and it's divided in three rows of four four rows each.


uhhuh.


yeah.


and the first four rows is well matched.


then the the second group of four rows is mismatched.


and finally highly mismatched.


and then the lower part is for italian.


and it's the same the same thing.


so so the upper part is training t i digits?


so it's it's the h t k results i mean.


so it's h t k training testings with different kind of features.


uh.


and what appears in the uh left column is the networks that are used for doing this.


huh.


so uh yeah.


well what was is that what was it that you had done last week when you showed


do you remember?


when you showed me the your table last week.


it it was part of these results.


so where is the baseline for the t i digits located in here?


you mean the h t k aurora baseline?


yeah.


it's uh the one hundred number.


it's well all these numbers are the ratio with respect to the baseline.


uh!


uh okay okay.


so this is word word error rate.


so a high number is bad.


yeah this is a word error rate ratio.


so it's yeah.


okay i see.


so seventy point two means that we reduced the error rate uh by thirty thirty per cent.


so


okay okay gotcha.


huh.


okay so if we take


uh um


let's see.


p l p uh with online normalization and delta l


so that's this thing you have circled here in the second column.


yeah.


um and multi english refers to what?


to timit.


then you have uh m f m s and m e.


which are for french spanish and english.


and


actually i i uh forgot to say that the multilingual net are trained on uh features without the derivatives.


uh but with increased frame numbers.


and we can we can see on the first line of the table that it it it's slightly slightly worse when we don't use data.


delta but it's not not that much.


right.


so so i'm sorry i missed that.


what's m f m s and m e?


multi french.


so multi french multi spanish and multi english.


multi spanish.


uh okay.


so it's uh broader vocabulary.


yeah.


then and


okay.


so i think what i'm what i saw in your smaller chart that i was thinking of was was there were some numbers i saw i think that included these multiple languages.


and the it and i was seeing that it got worse.


i i think that was all it was.


you had some very limited results that at that point.


which showed having in these these other languages.


in fact it might have been just this last category having two languages broad that were where where english was removed.


so that was cross language.


and the and the result was quite poor.


what i we hadn't seen yet was that if you added in the english it's still poor.


uh um now what's the noise condition um of the training data?


still poor.


well i think this is what you were explaining.


the noise condition is the same.


it's the same.


uh aurora noises uh in all these cases for the training.


yeah yeah.


so there's not a statistical a strong statistically different noise characteristic between uh the training and test.


no these are the same noises.


and yet we're seeing some kind of effect


yeah.


at least at least for the first for the well matched


well matched condition.


right.


so there's some kind of a a an effect from having these uh this broader coverage.


um


now i guess what we should try doing with this is try testing these on this same sort of thing.


on


you probably must have this lined up to do.


to try the same with the exact same training do testing on the other languages.


huh.


on on um


so um


oh i well wait a minute.


you have this here for the italian.


that's right.


okay.


so so


yeah so for the italian the results are stranger.


uh um huh


so what appears is that perhaps spanish is not very close to italian.


because uh well when using the the network trained only on spanish it's the error rate is almost uh twice the baseline error rate.


uhhuh.


huh uh.


well i mean let's see.


is there any difference in


so it's in the uh


so you're saying that when you train on english and uh and and test on


yeah.


no you don't have training on english testing.


there there is another difference is that the noise the noises are different.


well for for the italian part i mean.


in in what?


the uh the um networks are trained with noise from aurora t i digits.


aurora two.


huh.


and the noise is different in


and perhaps the noise are quite different from the noises in the speech that italian.


do we have any um test sets uh in any other language that um have the same noise as in the aurora?


and


huh no.


no.


can i ask something real quick?


in in the upper part in the english stuff it looks like the very best number is sixty point nine.


and that's in the uh the third section in the upper part under p l p j rasta.


sort of the middle column.


yeah.


is that a noisy condition?


so that's matched training.


is that what that is?


it's


the third part.


so it's uh highly mismatched.


so training and test noise are different.


so why do you get your best number in


wouldn't you get your best number in the clean case?


well it's relative to the um baseline mismatching.


uh!


yeah yeah.


okay.


so these are not


all right i see.


and then so in the in the um in the non mismatched clean case your best one was under m f c c?


that sixty one point four?


yeah but it's not a clean case.


it's a noisy case.


but uh training and test noises are the same.


oh so this upper third


so yeah.


uh that's still noisy?


yeah.


uh okay.


so it's always noisy basically.


uhhuh.


and well the


i see.


huh.


okay.


um so uh


i think this will take some looking at thinking about.


but what is uh what is currently running that's uh that just filling in the holes here?


or or pretty much?


uh no we don't plan to fill the holes.


but actually there is something important.


is that um we made a lot of assumption concerning the online normalization.


and we just noticed uh recently that uh the approach that we were using was not uh leading to very good results when we used the straight features to h t k.


um huh.


so basically if you look at the at the left of the table the first uh row with eighty six one hundred and forty three and seventy five


these are the results we obtained for italian uh with straight huh p l p features using online normalization.


huh and the huh what's in the table just at the left of the p l p twelve online normalization column.


so the numbers seventy nine fifty four and uh forty two are the results obtained by uh pratibha with uh his online normalization it's uh error her online normalization approach.


where is that?


seventy nine fifty?


uh it's just sort of sitting right on the uh the column line.


fifty one?


so


this


uh yeah.


oh i see okay.


just uh


yeah.


so these are the results of o g i with online normalization.


and straight features to h t k.


and the previous result eighty six and so on are with our features straight to h t k.


yes.


so what we see that is there is that um uh the way we were doing this was not correct.


but still the networks are very good.


when we use the networks our number are better that uh pratibha results.


we improve.


so do you know what was wrong with the online normalization or?


there were there were different things.


and basically the first thing is the huh alpha uh value.


so the recursion uh part


um i used point five per cent which was the default value in the in the programs here.


and pratibha used five per cent.


uh uhhuh.


so it adapts more quickly.


yeah.


um but


i assume that this was not important because uh previous results from from dan and show that basically the both both values give the same same uh results.


it was true on uh as t i digits but it's not true on italian.


uhhuh.


uh second thing is the initialization of the stuff.


actually uh what we were doing is to start the recursion from the beginning of the utterance.


and using initial values that are the global mean and variances measured across the whole database.


right.


and pratibha did something different is.


that he uh she initialed the um values of the mean and variance by computing this on the twenty five first frames of each utterance.


huh there were other minor differences.


the fact that she used fifteen dissities instead instead of thirteen.


and that she used c zero instead of log energy.


uh but the main difference is differences concerns the recursion.


so uh i changed the code.


uh and now we have a baseline that's similar to the o g i baseline.


okay.


we it it's slightly uh different.


because i don't exactly initialize the same way she does.


actually i start huh i don't wait to a fifteen twenty five twenty five frames before computing a mean and the variance to to to start the recursion.


uhhuh.


yeah.


i i use the online scheme.


and only start the recursion after the twenty five twenty fifth frame.


but well it's similar.


so uh i retrained the networks with these.


well the the the networks are retaining with these new features.


and


so basically what i expect is that these numbers will a little bit go down.


but perhaps not not so much.


right.


because i think the neural networks learn perhaps to


to even if the features are not normalized it it will learn how to normalize.


and uh


okay but i think that given the pressure of time we probably want to draw because of that especially we want to draw some conclusions from this.


do some reductions in what we're looking at.


and make some strong decisions for what we're going to do testing on before next week.


yeah i i'd


so do you are you did you have something going on on the side with uh multi band or on on this?


no.


or


i we plan to start this.


uh so actually we have discussed uh um these.


what we could do more as a as a research.


and and we were thinking perhaps that uh the way we use the tandem is not


uh well there is basically perhaps a flaw in the in the the stuff.


because we trained the networks


if we trained the networks on the on a language and a or a specific task


uhhuh.


um what we ask is to the network is to put the bound the decision boundaries somewhere in the space.


huh.


and uh huh and ask the network to put one at one side of the for for a particular phoneme at one side of the boundary decision boundary.


and one for another phoneme at the other side.


and so there is kind of reduction of the information there that's not correct.


because if we change task and if the phonemes are not in the same context in the new task obviously the decision boundaries are not should not be at the same place.


but the way the feature gives the the way the network gives the features is that it reduce completely the it removes completely the information a lot of information from the the features by uh uh placing the decision boundaries at optimal places for one kind of data.


i


but this is not the case for another kind of data.


it's a trade off.


so


right?


anyway go ahead.


yeah.


so uh we were thinking about is perhaps um one way to solve this problem is increase the number of outputs of the neural networks.


doing something like um um phonemes within context and


well basically context dependent phonemes.


maybe.


i mean i i think you could make the same argument.


it'd be just as legitimate for hybrid systems as well.


yeah but we know that


and in fact things get better with context dependent versions.


yeah but here it's something different.


we want to have features.


uh well um


yeah but it's still true that what you're doing is


you're ignoring


you're you're coming up with something to represent whether it's a distribution part of the probability distribution or features.


you're coming up with a set of variables that are representing uh things that vary over context.


uhhuh.


uh and you're putting it all together.


ignoring the differences in context.


that that's true for the hybrid system.


it's true for a tandem system.


so for that reason when you in in in a hybrid system when you incorporate context one way or another you do get better scores.


yeah.


okay.


but i it's it's a big deal to get that.


i i'm i'm sort of


and once you the other thing is that once you represent start representing more and more context it is uh much more um specific to a particular task in language.


so uh the the acoustics associated with uh a particular context


for instance you may have some kinds of contexts that will never occur in one language and will occur frequently in the other.


so the


the issue of getting enough training for a particular kind of context becomes harder.


we already actually don't have a huge amount of training data.


um


yeah but huh i mean the


the way we we do it now is that we have a neural network.


and basically the network is trained almost to give binary decisions.


right.


and uh binary decisions about phonemes.


nnn uh it's


almost.


but i mean it it it does give a distribution.


yeah.


it's and and it is true that if there's two phones that are very similar that uh the it may prefer one but it will give a reasonably high value to the other too.


yeah sure.


but


uh so basically it's almost binary decisions.


and um the idea of using more classes is to get something that's less binary decisions.


oh no.


but it would still be even more of a binary decision.


it it'd be even more of one.


because then you would say that in that this phone in this context is a one but the same phone in a slightly different context is a zero.


but yeah but


that would be even even more distinct of a binary decision.


i actually would have thought you'd want to go the other way and have fewer classes.


yeah but if


uh i mean for instance the the thing i was arguing for before but again which i don't think we have time to try is something in which you would modify the code so you could train to have several outputs on and use articulatory features.


huh.


uhhuh.


because then that would that would go that would be much broader and cover many different situations.


but if you go to very very fine categories it's very binary.


yeah but i think


yeah perhaps you're right.


but you have more classes.


so you you have more information in your features.


so um you have more information in the uh posterior spectrum.


true.


posteriors vector


um


which means that


but still the information is relevant.


because it's it's information that helps to discriminate.


if it's possible to be able to discriminate amongst uh among the phonemes in context.


well it's


but the


it's it's an interesting thought.


i mean we we could disagree about it at length.


huh.


but the the real thing is if you're interested in it you'll probably try it.


and and we'll see.


but but what i'm more concerned with now as an operational level is uh you know


what do we do in four or five days?


uh and so we have to be concerned with


are we going to look at any combinations of things?


you know once the nets get retrained so you have this problem out of it.


um are we going to look at multi band?


are we going to look at combinations of things?


uh what questions are we going to ask?


uh now that


i mean we should probably turn shortly to this o g i note.


um how are we going to combine with what they've been focusing on?


uh uh we haven't been doing any of the l d a rasta sort of thing.


uhhuh.


and they although they don't talk about it in this note.


um there's um the issue of the um new mu law business uh versus the logarithm.


um so


so what what is going on right now?


what's


right.


you've got nets retraining.


are there is there are there any h t k trainings testings going on?


i i i'm trying the h t k with uh p l p twelve online delta delta and m s g filter together.


the combination.


i see.


the combination yeah.


but i haven't result at this moment.


m s g and and p l p.


yeah.


and is this with the revised online normalization?


uh with the old old older.


old one.


so it's using all the nets for that.


but again we have the hope that it we have the hope that it maybe it's not making too much difference.


but we can


know soon.


maybe.


but but


i don't know.


uh okay.


uh so there is this combination.


working on combination obviously.


uhhuh.


um i will start work on multi band.


and we plan to work also on the idea of using both features and net outputs.


yup.


um and we think that with this approach perhaps we could reduce the number of outputs of the neural network.


um so get simpler networks.


because we still have the features.


so we have um come up with um different kind of broad phonetic categories.


and we have basically we have three types of broad phonetic classes.


well something using place of articulation.


which which leads to nine i think broad classes.


uh another which is based on manner.


which is is also something like nine classes.


and then something that combine both.


and we have twenty twenty five?


twenty seven.


twenty seven broad classes.


so like uh


oh i don't know.


like back vowels front vowels


so what you do


um for the moments we do not don't have nets.


um um i just want to understand.


so you have two net or three nets?


was this


how many how many nets do you have?


i mean it's just were we just changing the labels to retrain nets with fewer outputs.


no nets.


begin to work in this.


we are


right.


and then


but but i didn't understand


uhhuh.


uh the software currently just has uh a allows for i think the one one hot output.


so you're having multiple nets and combining them?


or


uh how are you how are you coming up with


if you say uh if you have a place characteristic and a manner characteristic how do you


it it's the single net.


i think they have one output.


yeah.


oh it's just one net.


it's one net with um twenty seven outputs.


if we have twenty seven classes.


i see.


i see okay.


so it's well it's basically a standard net with fewer classes.


so you're sort of going the other way of what you were saying a bit ago.


instead of


but i think


but including the features.


including the features yeah.


i don't think this will work alone.


i think it will get worse.


because well i believe the effect that of of too reducing too much the information is basically basically what happens.


uhhuh.


and


but you think if you include that plus the other features


but


yeah.


because there is perhaps one important thing that the net brings.


and o g i showed that this is the distinction between speech and silence.


because these nets are trained on well controlled condition.


i mean the labels are obtained on clean speech and we add noise after.


so this is one thing.


but perhaps something intermediary using also some broad classes could could bring so much more information.


uh


so so again then we have these broad classes.


and well somewhat broad.


i mean it's twenty seven instead of sixty four basically.


and you have the original features.


which are p l p or something.


uhhuh.


and then uh just to remind me all of that goes into uh that all of that is transformed by uh uh k k l or something or?


there will probably be


mu.


yeah.


one single k l to transform everything.


or uh


right.


no.


transform the p l p.


and only transform the other.


i'm not sure.


this is still something that


well no.


i think


we don't know


i see.


so there's a question of whether you would


two it's one.


whether you would transform together or just one.


might want to try it bother both ways.


but that's interesting.


so that's something that you're you haven't trained yet but are preparing to train and


yeah.


um yeah.


huh.


so i think hynek will be here monday.


on their


monday or tuesday.


so


uh yeah.


so i think you know we need to choose the choose the experiments carefully.


so we can get uh key key questions answered uh before then.


uhhuh.


and leave other ones aside even if it leave leaves incomplete tables some place.


someplace uh uh it's it's really time to time to choose.


um let me pass this out by the way.


um these are


did did did i interrupt you?


yeah i have one.


were there other things that you wanted to


uh no i don't think so.


oh thanks.


uh okay okay we have lots of them.


we have one.


okay so um something i asked


so they're they're doing the the v a d.


i guess they mean voice activity detection so again it's the silence.


so they've just trained up a net which has two outputs i believe.


um i asked uh hynek whether


i haven't talked to sunil.


i asked hynek whether they compared that to just taking the nets we already had and summing up the probabilities.


uhhuh.


uh to get the speech voice activity detection or else just using the silence if there's only one silence output.


um and he didn't think they had.


um but on the other hand maybe they can get by with a smaller net.


then and maybe sometimes you don't run the other.


maybe there's a computational advantage to having a separate net anyway.


so um their uh the results look pretty good.


yeah.


um i mean not uniformly.


i mean there's a an example or two that you can find where it made it slightly worse but uh in in all but a couple examples.


huh.


uh


but they have a question of the result.


um how are trained the the l d a filter?


how obtained the l d a filter?


i'm sorry.


i don't understand your question.


yes.


um the l d a filter needs some training set to obtain the filter.


maybe


i don't know exactly how they are obtained.


it's on training.


training with the training test of each


you understand me?


no.


yeah.


uh for example l d a filter need a set of a set of training to obtain the filter.


and maybe for the italian for the t d t e on for finnish these filter are are obtained with the um chinese their own training set.


yes i don't know.


that's that's so that's a that's a very good question.


then now that it i understand it it's


where does the l d a come from?


in the in earlier experiments they had taken l d a from a completely different database.


right?


yeah because maybe it the same situation that the neural network training with their own


huh.


set.


so that's a good question.


where does it come from?


yeah i don't know.


um but uh to tell you the truth i wasn't actually looking at the l d a so much when i i was looking at it.


i was mostly thinking about the the v a d.


and um it it


oh what does what does a s p


oh that's


the features.


yeah.


i don't understand also.


it says baseline a s p.


what this


is what is the difference between a s p and uh over?


a s p.


this is


oh.


anybody know any


there it is.


um because there's baseline aurora above it.


uhhuh.


and it's this is mostly better than baseline.


although in some cases it's a little worse in a couple cases.


well it says baseline a s p is twenty three mill minus thirteen.


yeah it says what it is.


but i don't how that's different from


from the baseline okay.


i think this was i think this is the same point we were at when when we were up in oregon.


yeah.


i think i think it's the c zero using c zero instead of log energy.


uh okay uhhuh.


yeah it's this.


oh okay.


it should be that.


because


shouldn't it be


they they say in here that the v a d is not used as an additional feature.


does does anybody know how they're using it?


yeah so so what they're doing here is


if you look down at the block diagram um they estimate they get a they get an estimate of whether it's speech or silence.


but that


and then they have a median filter of it.


uhhuh.


and so um basically they're trying to find stretches.


the median filter is enforcing a it having some continuity.


you find stretches where the combination of the frame wise v a d and the the median filter say that there's a stretch of silence.


and then it's going through and just throwing the data away.


huh.


right?


so um


so it's it's


i don't understand.


you mean it's throwing out frames?


before


it's throwing out chunks of frames yeah.


there's the the median filter is enforcing that it's not going to be single cases of frames or isolated frames.


yeah.


so it's throwing out frames.


and the thing is um what i don't understand is how they're doing this with h t k.


this is


that's what i was just going to ask.


how can you just throw out frames?


well you you can.


i


i mean you you


it stretches again.


for single frames i think it would be pretty hard.


but if you say speech starts here speech ends there.


uhhuh.


right?


huh.


yeah you can basically remove the the frames from the feature feature files.


yeah so i mean in the in the in the decoding you're saying that we're going to decode from here to here.


i think they're they're they're treating it you know like uh


well it's not isolated word.


but but connected.


you know the the


in the text they say that this this is a tentative block diagram of a possible configuration we could think of.


so that sort of sounds like they're not doing that yet.


well no they they have numbers though.


so i think they're they're doing something like that.


i think that they're they're


i think what i mean by that is they're trying to come up with a block diagram that's plausible for the standard.


in other words it's uh


i mean from the point of view of of uh reducing the number of bits you have to transmit it's not a bad idea to detect silence anyway.


yeah yeah.


um


i'm just wondering what exactly did they do up in this table if it wasn't this.


but it's the thing is it's that that that that's that's i i


certainly it would be tricky about it in in transmitting voice uh is uh for listening to is that these kinds of things uh cut speech off a lot.


right?


uhhuh.


and so um


plus it's going to introduce delays.


it does introduce delays.


but they're claiming that it's it's uh within the the boundaries of it.


huh.


and the l d a introduces delays.


and what he's suggesting this here is a parallel path.


so that it doesn't introduce uh anymore any more delay.


it introduces two hundred milliseconds of delay.


but at the same time the l d a down here.


i don't know


what's the difference between t l d a and s l d a?


temporal and spectral.


uh thank you.


temporal l d a.


yeah you would know that.


yeah


so um the temporal l d a does in fact include the same.


so that i think he well by by saying this is a a tentative block diagram i think means if you construct it this way this this delay would work in that way.


and then it'd be okay.


uh.


they they clearly did actually remove silent sections.


in order because they got these word error rate results.


so um i think that it's it's nice to do that in this.


because in fact it's going to give a better word error result.


and therefore will help within an evaluation.


whereas to whether this would actually be in a final standard i don't know.


um uh as you know part of the problem with evaluation right now is that the word models are pretty bad.


and nobody wants has has approached improving them.


so it's possible that a lot of the problems with so many insertions and so forth would go away if they were better word models to begin with.


so this might just be a temporary thing.


but but on the other hand and maybe maybe it's a decent idea.


so um the question we're going to want to go through next week when hynek shows up i guess is given that we've been


if you look at what we've been trying


we're uh looking at uh


by then i guess combinations of features and multi band.


uh and we've been looking at cross language cross task issues.


and they've been not so much looking at the cross task uh multiple language issues that


but they've been looking at uh at these issues.


at the online normalization and the uh voice activity detection.


and i guess when he comes here we're going to have to start deciding about um what do we choose from what we've looked at to um blend with some group of things in what they've looked at?


and once we choose that how do we split up the effort?


uh because we still have even once we choose we've still got uh another month or so.


i mean there's holidays in the way but but uh i think the evaluation data comes january thirty first.


so there's still a fair amount of time to do things together.


it's just that they probably should be somewhat more coherent between the two sites in that that amount of time.


when they removed the silence frames did they insert some kind of a marker so that the recognizer knows it's knows when it's time to back trace or something?


well see they


i i think they're


um i don't know the the specifics of how they're doing it.


they're they're getting around the way the recognizer works because they're not allowed to um change the scripts for the recognizer i believe.


oh right.


so um


maybe they're just inserting some nummy frames or something?


uh you know that's what i had thought.


but i don't i don't think they are.


huh.


i mean that's sort of what the way i had imagined would happen is that on the other side


yeah.


you put some low level noise or something.


probably don't want all zeros.


most recognizers don't like zeros.


but but you know put some epsilon in or some


sorry.


epsilon random variable in or something.


some constant vector.


i mean


maybe not a constant but it doesn't uh don't like to divide by the variance of that.


or something


but i mean it's


that's right.


but something that


what i mean is something that is very distinguishable from speech.


uhhuh.


so that the the silence model in h t k will always pick it up.


yeah.


so i i that's what i thought they would do.


or else uh uh maybe there is some indicator to tell it to start and stop.


i don't know.


huh.


but whatever they did i mean they have to play within the rules of this specific evaluation.


we we can find out.


because you got to do something otherwise if it's just a bunch of speech stuck together


no they're


it would do badly.


and it didn't so badly.


yeah right.


right?


so they did something.


yeah yeah.


uh so okay.


so i think this brings me up to date a bit.


it hopefully brings other people up to date a bit.


and um um i think uh i want to look at these numbers off line a little bit.


and think about it and and talk with everybody uh outside of this meeting.


um but uh


no i mean it sounds like i mean there there there are the usual number of of little little problems and bugs and so forth.


but it sounds like they're getting ironed out.


and now we're seem to be kind of in a position to actually uh look at stuff and and and compare things.


so i think that's that's pretty good.


um i don't know what the


one of the things i wonder about coming back to the first results you talked about is is how much uh things could be helped by more parameters.


and uh and uh how many more parameters we can afford to have in terms of the uh computational limits.


because and anyway when we go to twice as much data and have the same number of parameters


particularly when it's twice as much data and it's quite diverse.


um i wonder if having twice as many parameters would help.


uhhuh.


uh just have a bigger hidden layer.


uh but


i doubt it would help by forty per cent.


but but uh


yeah.


just curious.


how are we doing on the resources?


disk and


i think we're all right.


um not much problems with that.


okay.


computation?


it's okay.


well this table took uh more than five days to get back.


we


yeah well


but


are were you folks using gin?


that's a that just died.


you know?


huh no.


you were using gin perhaps?


yeah.


no.


no no.


oh that's good.


it just died.


okay.


yeah we're going to get a replacement server that'll be a faster server actually.


yes.


huh uhhuh.


that'll be it's a seven hundred fifty megahertz uh sun.


tonic.


uh but it won't be installed for a little while.


go ahead.


do we do we have that big new i b m machine?


the i think in


we have the little tiny i b m machine that might someday grow up to be a big i b m machine.


it's got slots for eight.


uh i b m was donating five.


i think we only got two so far.


processors.


we had originally hoped we were getting eight hundred megahertz processors.


they ended up being five fifty.


so instead of having eight processors that were eight hundred megahertz we ended up with two that are five hundred and fifty megahertz.


and more are supposed to come soon.


and there's only a moderate amount of of memory.


so i don't think anybody has been sufficiently excited by it to spend much time um uh with it.


but uh hopefully they'll get us some more parts soon.


and


uh yeah i think that'll be


once we get it populated that'll be a nice machine.


i mean we will ultimately get eight processors in there.


and uh and uh a nice amount of memory.


uh so it'll be a pretty fast linux machine.


and if we can do things on linux some of the machines we have going already like swede.


uhhuh.


um it seems pretty fast.


but i think fudge is pretty fast too.


yeah i mean you can check with uh dave johnson.


i mean it it's i think the machine is just sitting there.


and it does have two processors.


you know?


and somebody could do you know uh check out uh the multi threading libraries.


and i mean it's possible that the


and i mean i guess the prudent thing to do would be for somebody to do the work on on getting our code running on that machine with two processors even though there aren't five or eight.


there's there's there's going to be debugging hassles.


and then we'd be set for when we did have five or eight to have it really be useful.


but notice how i said somebody in


and turned my head your direction.


that's one thing you don't get in these recordings.


you don't get the don't get the visuals.


but


is it um um the neural network trainings that are um us down or the h t k runs that are slowing us down?


uh i think yes.


uh isn't that right?


i mean i think you're you're sort of held up by both.


right?


if the if the neural net trainings were a hundred times faster you still wouldn't be anything.


running through these a hundred times faster.


because you'd be stuck by the h t k trainings.


huh.


yeah.


but if the h t k


i mean i think they're both it sounded like they were roughly equal.


is that about right?


because um think that'll be running linux.


and swede and fudge are already running linux.


so um could try to get um the neural network trainings or the h t k stuff running under linux and to start with i'm wondering which one i should pick first.


uh probably the neural net.


because it's probably it it's it's um


well i i don't know.


they both


h t k we use for uh um this aurora stuff.


um um i think it's not clear yet what we're going to use for trainings.


uh


well there's the trainings.


uh is it the training that takes the time or the decoding?


uh is it about equal between the two?


for for aurora?


for h t k?


for


yeah for the aurora?


uh training is longer.


okay.


yeah.


well i don't know how we can


i don't know how to


do we have h t k source?


is that


huh.


you would think that would fairly trivially


the training would anyway.


the testing uh i don't i don't think would parallelize all that well.


but i think that you could certainly do um distributed sort of uh


no it's the each individual sentence.


is pretty tricky to parallelize.


but you could split up the sentences in a test set.


they have a they have a thing for doing that.


and they have for awhile in h t k.


and you can parallelize the training.


and run it on several machines.


aha!


and it just basically keeps counts.


and there's something a final thing that you run.


and it accumulates all the counts together.


i see.


i don't what their scripts are set up to do for the aurora stuff.


but


yeah.


something that we haven't really settled on yet is other than this aurora stuff uh what do we do large vocabulary training slash testing for uh tandem systems?


because we hadn't really done much with tandem systems for larger stuff.


because we had this one collaboration with c m u and we used sphinx.


uh we're also going to be collaborating with s r i.


and we have their have theirs.


um so i don't know.


um so i i think the the advantage of going with the neural net thing is that we're going to use the neural net trainings no matter what.


okay.


for a lot of the things we're doing.


whereas exactly which h m m gaussian-mixture-based h m m thing we use is going to depend uh


so with that maybe we should uh go to our digit recitation task.


and it's about eleven fifty.


canned.


uh i can i can start over here.


so everybody everybody's on?


so you guys had a a meeting with uh with hynek which i unfortunately had to miss.


um


huh.


and uh somebody


uh and uh


i guess chuck you weren't there either.


so uh


i was there.


oh you were there.


with hynek?


yeah.


so everybody knows what happened except me.


maybe somebody should tell me.


oh yeah.


all right.


well uh first we discussed about some of the points that i was addressing in the mail i sent last week.


uhhuh.


so


about the um well the downsampling problem.


uh and about the the length of the filters.


and


what was the what was the downsampling problem again?


so we had


i forget.


so the fact that there there is no uh low pass filtering before the downsampling.


well.


there is because there is l d a filtering but that's perhaps not uh the best.


depends what it's frequency characteristic is yeah.


is the system on?


so you could do a you could do a stricter one.


maybe.


yeah.


so we discussed about this about the um


was there any conclusion about that?


uh try it.


i see.


i guess.


uh


so again this is this is the downsampling uh of the uh the feature vector stream.


and


um yeah i guess the the uh l d a filters they were doing do have


um uh let's see so the the the feature vectors are calculated every ten milliseconds.


so


uh the question is how far down they are at fifty fifty hertz.


uh um


uhhuh.


sorry at twenty five hertz since they're downsampling by two.


so does anybody know what the frequency characteristic is?


we don't have yet.


um so yeah.


oh okay.


we should have a look first at perhaps the modulation spectrum.


okay.


yeah.


um so there is this


there is the um length of the filters.


um so the this idea of trying to find filters with shorter delays.


um we started to work with this.


huh huh.


huh.


and the third point um was the um


yeah the online normalization where


well the recursion recursion for the mean estimation is a filter with some kind of delay.


and that's not taken into account right now.


um yeah.


and there again yeah.


for this the conclusion of hynek was well we can try it but


uhhuh.


um


try try what?


so try to um um take into account the delay of the recursion for the mean estimation.


okay.


and this we've not uh worked on this yet.


and so while discussing about these these l d a filters some issues appeared like


well the fact that if we look at the frequency response of these filters it's


uh well we don't know really what's the important part in the frequency response.


and there is the fact that in the very low frequency these filters don't don't really remove a lot compared to the to the uh standard rasta filter.


uh and that's probably a reason why yeah online normalization helps because it it


right.


yeah it removed this mean.


um yeah but perhaps everything could should be could be in the filter.


i mean uh the the mean normalization and


yeah.


so


so basically that was that's all we discussed about.


we discussed about good things to do.


also uh well generally good stuff to do for the research.


uhhuh.


and this was this l d a uh tuning perhaps and hynek proposed again to his uh traps.


okay.


um


i mean i i guess the key thing for me is is figuring out how to better coordinate between the two sides.


because because um


uh i was talking with hynek about it later.


and the the sort of had the sense sort of that that neither group of people wanted to to bother the other group too much.


and and i don't think anybody is you know closed in in their thinking or are unwilling to talk about things.


but i think that you were sort of waiting for them to tell you that they had something for you.


and and that and expected that they would do certain things.


and they were they didn't want to bother you.


and they were sort of waiting for you.


and and and uh we ended up with this thing where they they were filling up all of the possible latency themselves.


and they just hadn't thought of that.


so


uh i mean it's true that maybe maybe no one really thought about that that this latency thing would be such a a strict issue.


yeah.


well but.


well


in in uh the other


i don't know what happened really but


i guess it's it's also so uh the time constraints.


because well we discussed about that about this problem.


and they told us well we will do all that's possible to have enough space for a network.


but then yeah perhaps they were too short with the time and


then they couldn't.


i see.


uh yeah.


but there was also problem perhaps a problem of communication.


so yeah.


now we will try to


just talk more.


yeah slikes and send mails.


uh so


yeah.


uh


okay.


so there's um


all right.


well maybe we should just


uh i mean you're you're other than that you folks are busy doing all the all the things that you're trying that we talked about before.


right?


and this machines are busy.


and you're busy.


and


basically.


um


oh.


let's let's i mean i think that as as we said before that one of the things that we're imagining is that uh there there will be uh in the system we end up with there'll be something to explicitly uh uh do something about noise.


uh.


in addition to the uh other things that we're talking about.


and that's probably the best thing to do.


and there was that one email that said that it sounded like uh uh things looked very promising up there.


in terms of uh i think they were using ericsson's approach or something.


and in addition to


they're doing some noise removal thing.


yeah yeah.


so yeah we're will start to do this also.


yeah.


uh so carmen is just looking at the ericsson ericsson code.


we


uhhuh.


yeah i modified it.


and


well modify no.


i studied barry's sim code more or less.


to take only the first step the spectral subtraction.


and we have some the feature for italian database.


and we will try with this feature with the filter to find the result.


but we haven't result until this moment.


yeah sure.


but well we are working in this also.


and maybe try another type of spectral subtraction i don't


when you say you don't have a result yet you mean it's it's just that it's in process?


or that you it finished and it didn't get a good result?


no.


no no we have we have do the experiment.


only have the feature the feature but the experiment have


yeah.


we have not make the experiment.


oh.


and


okay.


maybe will be good result or bad result.


we don't know.


so um i suggest actually now we we we sort of move on and and hear what's what's what's happening in in other areas.


like what's what's happening with your investigations about echos and so on.


oh um


well um i haven't started writing the test yet i'm meeting with adam today.


uhhuh.


um and he's going show me the scripts he has for um running recognition on meeting recorder digits.


uh i also um haven't got the code yet.


i haven't asked hynek for for the for his code yet.


because i looked at uh avendano's thesis and i don't really understand what he's doing yet.


but it it it sounded like um the channel normalization part um of his thesis um was done in a a bit of i don't know what the word is a a bit of a rough way.


um it sounded like he um


he he it it wasn't really fleshed out.


and maybe he did something that was interesting for the test situation.


but i i'm not sure if it's what i'd want to use.


so i have to i have to read it more.


i don't really understand what he's doing yet.


okay.


it's my


yeah i haven't read it in a while so i'm not going to be too much help unless i read it again.


i know this is mine here.


oh yeah.


so


um the um


so you and then you're also going to be doing this echo cancelling between the the close mounted and the and the the the what we're calling a cheating experiment uh of sorts.


uh i'm


between the distant


right.


well or i'm hoping i'm hoping espen will do it.


um


uh.


delegate.


that's good.


it's good to delegate.


i i think he's at least planning to do it for the close mike cross talk.


and so maybe i can just take whatever setup he has and use it.


great.


yeah actually um he should uh


i wonder who else is


i think maybe it's dan ellis is going to be doing uh a different cancellation.


um one of the things that people working in the meeting task want to get at is they would like to have cleaner close miked recordings.


so uh this is especially true for the lapel.


but even for the close close miked uh cases.


um we'd like to be able to have um other sounds from other people and so forth removed from


so when someone isn't speaking you'd like the part where they're not speaking to actually be


so what they're talking about doing is using uh echo cancellation like techniques.


it's not really echo but uh just um uh


taking the input from other mikes and using uh uh a uh an adaptive filtering approach to remove the effect of that uh other speech.


so


um what was it?


there was there was some some some point where uh uh eric or somebody was was speaking and he had lots of silence in his channel.


and i was saying something to somebody else uh which was in the background.


and it was not


it was recognizing my words which were the background speech on the close close mike.


huh.


oh the what we talked about yesterday?


yes.


yeah that was actually my


i was wearing the i was wearing the lapel and you were sitting next to me.


oh you.


it was you i was


yeah.


and i only said one thing but you were talking and it was picking up all your words.


so they would like clean channels.


uh and for that huh uh that purpose uh they'd like to pull it out.


so i think i think dan ellis or somebody who was working with him was going to uh work on that.


so


okay.


right?


um and uh i don't know if we've talked lately about the the plans you're developing that we talked about this morning.


uh i don't remember if we talked about that last week or not.


but maybe just a quick reprise of of what we were saying this morning.


um so continuing to um extend


uh


what about the stuff that um mirjam has been doing?


and and shawn yeah.


oh.


so they're training up nets to try to recognize these acoustic features?


i see.


but that's uh uh all that's is a a certainly relevant uh study.


and you know what are the features that they're finding?


we have this problem with the overloading of the term feature.


yeah.


uh what are the variables?


what we're calling this one?


what are the variables that they're found finding useful?


huh.


um


and their their targets are based on canonical mappings of phones to acoustic features.


for


right.


and that's certainly one thing to do and we're going to try and do something more more fine than that.


but uh


so


so i guess you know what i was trying to remember some of the things we were saying.


do you still have that


oh yeah.


there's those that


uh


yeah some of some of the issues we were talking about was in just getting a good handle on on uh what good features are.


and


what does what did um larry saul use for it was the sonorant uh detector?


right?


how did he


how did he do that?


what was his detector?


uhhuh.


oh okay.


so how did he combine all these features?


what what huh classifier did he use?


huh.


oh right.


you were talking about that yeah.


i see.


and the other thing you were talking about is is is where we get the targets from.


so i mean there's these issues of what are the what are the variables that you use.


and do you combine them using the soft and or or you do something you know more complicated.


um and then the other thing was so where do you get the targets from.


the initial thing is just the obvious that we're discussing is starting up with phone labels from somewhere and then uh doing the transformation.


but then the other thing is to do something better.


and uh why don't you tell us again about this this database?


this is the


huh!


and then tell them to talk naturally?


yeah yeah.


pierced tongues and


yeah.


you could just mount it to that and they wouldn't even notice.


weld it.


zzz.


maybe you could go to these parlors and and you could you know you know have have you know reduced rates if you if you can do the measurements.


i


that's right.


you could what you could do is you could sell little rings and stuff with embedded you know transmitters in them and things.


and


yeah be cool and help science.


okay.


there's a bunch of data that around


that people have done studies like that way way back.


right?


i mean i can't remember where uh wisconsin or someplace that used to have a big database of


i remember there was this guy at a t and t.


randolph?


or what was his name?


do you remember that guy?


um researcher at a t and t a while back that was studying trying to do speech recognition from these kinds of features.


i can't remember what his name was.


dang.


now i'll think of it.


do you mean uh but you i mean


that's interesting.


well he was the guy the guy that was using


you mean when was was mark randolph there or


mark randolph.


yeah he's he's he's at motorola now.


oh is he?


oh okay.


yeah.


is it the guy that was using the pattern of pressure on the tongue?


or


i can't remember exactly what he was using now.


but i know i just remember it had to do with you know uh positional parameters.


what


uhhuh.


and trying to you know do speech recognition based on them.


so the only the only uh hesitation i had about it since i mean i haven't see the data is it sounds like it's it's continuous variables and a bunch of them.


huh.


and so


i don't know how complicated it is to go from there.


what you really want are these binary labels and just a few of them.


and maybe there's a trivial mapping if you want to do it.


and it's but it


i i i worry a little bit that this is a research project in itself.


whereas um if you did something instead that like um having some manual annotation by uh you know linguistics students.


this would there'd be a limited set of things that you could do as per our discussions with with john before.


but the things that you could do like nasality and voicing and a couple other things you probably could do reasonably well.


and then there would it would really be uh this uh uh binary variable.


course then that's the other question is do you want binary variables.


so


i mean the other thing you could do is boot trying to to uh get those binary variables.


and take the continuous variables from uh the uh uh the data itself there.


but i i'm not sure


could you cluster the just do some kind of clustering?


guess you could yeah.


bin them up into different categories and


yeah.


so anyway that's that's uh that's another whole direction that could be looked at.


um um i mean in general it's going to be for new data that you look at.


it's going to be hidden variable because we're not going to get everybody sitting in these meetings to wear the pellets and


right.


um so


so you're talking about using that data to get


uh


instead of using canonical mappings of phones.


so you'd use that data to give you sort of what the the true mappings are for each phone?


uhhuh.


i see.


so yeah where this fits into the rest in in my mind i guess is that um we're looking at different ways that we can combine uh different kinds of of front end representations um in order to get robustness under difficult or even you know typical conditions.


and part of it this robustness seems to come from uh multi stream or multi band sorts of things.


and saul seems to have a reasonable way of looking at it at least for one one um articulatory feature.


the question is is can we learn from that to change some of the other methods we have since


i mean one of the things that's nice about what he had i thought was that that it it um


the decision about how strongly to train the different pieces is based on uh a a reasonable criterion with hidden variables.


rather than um just assuming that you should train every detector uh with equal strength towards uh it being this phone or that phone.


huh.


right?


so it so um he's got these um uh uh


he and's between these different features.


it's a soft and i guess.


but in in principle you you want to get a strong concurrence of all the different things that indicate something.


and then he or's across the different soft or's across the different uh multi band channels.


and um the weight


yeah the target for the training of the and and'ed things is something that's kept uh as a hidden variable.


and is learned with e m.


so he doesn't have


whereas what we were doing is is uh taking the phone target and then just back propagating from that.


which means that it's it's uh it could be for instance that for a particular point in the data you don't want to um uh train a particular band train the detectors for a particular band.


you you want to ignore that band because that's a band is a noisy noisy measure.


uhhuh.


and we don't


we're we're still going to try to train it up.


in our scheme we're going to try to train it up to do as well well as it can at predicting.


uh maybe that's not the right thing to do.


so he doesn't have to have truth marks?


or


right.


and uh he doesn't have to have hard labels.


well at the at the tail end yeah he has to know what's where it's sonorant.


for the full band.


but he's but what he's but what he's not training up uh what he doesn't depend on as truth is


um i guess one way of describing would be


if if a sound is sonorant is it sonorant in this band?


is it sonorant in that band?


it's hard to even answer that.


what you really mean is that the whole sound is sonorant.


uhhuh.


so


okay.


then it comes down to you know to what extent should you make use of information from particular band towards making your decision.


i see.


and um uh we're making in a sense sort of this hard decision that you should you should use everything uh with with uh equal strength.


and uh because in the ideal case we would be going for posterior probabilities.


if we had uh enough data to really get posterior probabilities.


and if the if we also had enough data so that it was representative of the test data.


then we would in fact be doing the right thing to train everything as hard as we can.


but um this is something that's more built up along an idea of robustness from from the beginning.


and so you don't necessarily want to train everything up towards the


so where did he get his uh his his uh high level targets about what's sonorant and what's not?


from uh canonical mappings um at first.


yeah.


and then it's unclear um uh


using timit?


or using


using timit.


right right.


uhhuh.


and then uh he does some fine tuning um for um special cases.


i mean we we have a kind of iterative training because we do this embedded viterbi.


uh so there is something that's suggested based on the data.


but it's it's not


i think it doesn't seem like it's quite the same because of this.


because then whatever that alignment is it's that for all all bands.


well no that's not quite right we did actually do them separate tried to do them separately.


so that would be a little more like what he did.


um


but it's still not quite the same because then it's it's um setting targets based on where you would say the sound begins in a particular band.


where he's this is not a labeling per se.


might be closer i guess if we did a soft soft target uh uh embedded neural net training like we've done a few times.


uh the forward um do the forward calculations to get the gammas and train on those.


huh.


uh what's next?


i could say a little bit about stuff i've been playing with.


oh.


i um


you're playing?


huh?


yes i'm playing.


um so i wanted to do this experiment to see um uh what happens if we try to uh improve the performance of the back end recognizer for the aurora task.


and see how that affects things.


and so i had this um i think i sent around last week a this plan i had for an experiment.


this matrix where i would take the um the original um the original system.


so there's the original system trained on the mel cepstral features.


and then and then uh optimize the h t k system and run that again.


so look at the difference there.


and then uh do the same thing for the i.c.s.i.o.g.i. front end.


what which test set was this?


this is


that i looked at?


uhhuh.


uh i'm looking at the italian right now.


so as far as i've gotten is i've uh been able to go through from beginning to end the um full h t k system for the italian data.


and got the same results that um that uh stephane had.


so um i started looking to and now i'm i'm sort of at the point where i want to know what should i change in the h t k back end in order to try to uh to improve it.


so


one of the first things i thought of was the fact that they use the same number of states for all of the models.


and so i went online and i uh found a pronunciation dictionary for italian digits.


and just looked at you know the number of phones in each one of the digits.


um you know sort of the canonical way of setting up a an h m m system is that you use um three states per phone.


and um so then the the total number of states for a word would just be you know the number of phones times three.


and so when i did that for the italian digits i got a number of states ranging on the low end from nine to the high end eighteen.


um now you have to really add two to that because in h t k there's an initial null and a final null.


so when they use uh models that have eighteen states there're really sixteen states.


they've got those initial and final null states.


and so um their guess of eighteen states seems to be pretty well matched to the two longest words of the italian digits.


the four and five which um according to my you know sort of off the cuff calculation should have eighteen states each.


uhhuh.


and so they had sixteen.


so that's pretty close.


um but for the most of the words are much shorter.


so the majority of them want to have nine states.


and so theirs are sort of twice as long.


so my guess uh


and then if you i i printed out a confusion matrix um uh for the well matched case.


and it turns out that the longest words are actually the ones that do the best.


so my guess about what's happening is that you know if you assume a fixed the same amount of training data for each of these digits.


and a fixed length model for all of them.


but the actual words for some of them are half as long.


you really um have you know half as much training data for those models.


because if you have a long word and you're training it to eighteen states uh you've got you know you've got the same number of gaussians you've got to train in each case.


but for the shorter words you know the total number of frames is actually half as many.


so it could be that you know for the short words there's because you have so many states you just don't have enough data to train all those gaussians.


so um i'm going to try to um create more word specific um uh prototype h m m's to start training from.


yeah i mean it's not at all uncommon you do worse on long on short words than long words anyway just because you're accumulating more evidence for the for the longer word.


uhhuh.


but


yeah so i'll i'll the next experiment i'm going to try is to just um you know create uh models that seem to be more matched to my guess about how long they should be.


and as part of that um i wanted to see sort of how the um


how these models were coming out you know what when we train up uh you know the model for one which wants to have nine states you know.


what is the uh what do the transition probabilities look like in the self loops look like in in those models?


and so i talked to andreas and he explained to me how you can calculate the expected duration of an h m m just by looking at the transition matrix.


and so i wrote a little matlab script that calculates that.


and


so i'm going to sort of print those out for each of the words to see what's happening you know how these models are training up.


you know the long ones versus the short ones.


i i did


quickly i did the silence model


and and um


that's coming out with about one point two seconds as its average duration.


and the silence model's the one that's used at the beginning and the end of each of the string of digits.


wow!


lots of silence.


yeah yeah.


and so the s p model which is what they put in between digits i i haven't calculated that for that one yet.


but um


so they basically their their model for a whole digit string is silence digit s p digit s p blah blah blah and then silence at the end.


and so


are the s p's optional?


i mean skip them?


i have to look at that but i'm not sure that they are.


now the one thing about the s p model is really it only has a single emitting state to it.


uhhuh.


so if it's not optional you know it's it's not going to hurt a whole lot.


i see.


and it's tied to the center state of the silence model so it's not its own.


um it doesn't require its own training data.


it just shares that state.


so it i mean it's pretty good the way that they have it set up.


so i want to play with that a little bit more.


i'm curious about looking at you know how these models have trained and looking at the expected durations of the models.


and i want to compare that in the the well matched case to the unmatched case and see if you can get an idea of


just from looking at the durations of these models you know what what's happening.


yeah i mean i think that uh as much as you can it's good to sort of not do anything really tricky.


not do anything that's really finely tuned.


but just sort of uh you know you you


yeah.


the premise is kind of if you have a a good person look at this for a few weeks and what do you come up with.


huh.


uhhuh.


and uh


and hynek when i told him about this he had an interesting point.


and that was um the the final models that they end up training up have i think probably something on the order of six gaussians per state.


so they're fairly you know hefty models.


and hynek was saying that well probably in a real application you wouldn't have enough compute to handle models that are very big or complicated.


so in fact what we may want are simpler models.


could be.


and compare how they perform to that.


but you know it depends on what the actual application is.


and it's really hard to know what your limits are in terms of how many gaussians you can have.


right.


and that i mean at the moment that's not the limitation.


so


i mean i i i what i thought you were going to say but which i was thinking was um


where did six come from?


probably it came from the same place eighteen came from.


yeah.


you know?


uh that's another parameter.


right?


yeah yeah.


that that maybe you know uh you really want three or nine or


well one thing i mean if i if if i start um reducing the number of states for some of these shorter models that's going to reduce the total number of gaussians.


so in a sense it'll be a simpler system.


but i think right now again the idea is doing just very simple things.


how much better can you make it?


uhhuh.


and um since they're only simple things there's nothing that you're going to do that is going to blow up the amount of computation.


right.


um so


if you found that nine was better than six that would be okay i think actually.