right right right.


that's great.


so


yeah i i shweta mentioned that although she said it's a secret.


okay.


hi.


right.


the faculty aren't faculty aren't supposed to know.


no faculty.


wednesday's much better for me yeah.


but um


i'm sufficiently clueless that i count as a


yeah right.


it's as if we didn't tell anyone at all.


bhaskara.


so uh you see don the unbridled excitement of the work that we have on this project.


it's just uh


um


uh you know it doesn't seem like a bad idea to have that information.


and i'm surprised i sort of i'm surprised i forgot that.


but uh i think that would be a good thing to add.


yeah i i'd i think it's


after i just printed out a zillion of them.


yeah well that's


um so i i do have a an agenda suggestion.


uh we i think the things that we talk about in this meeting uh tend to be a mixture of uh procedural uh mundane things and uh research points.


and um i was thinking i think it was a meeting a couple of weeks ago that we we spent much of the time talking about the mundane stuff because that's easier to get out of the way.


and then we sort of drifted into the research and maybe five minutes into that andreas had to leave.


so uh i'm suggesting we turn it around.


and and uh sort of we have anybody has some mundane points that we could send an email later uh hold them for a bit and let's talk about the the research y kind of things.


um so um the one one thing i know that we have on that is uh we had talked a a couple weeks before um uh about the uh the stuff you were doing with with uh um uh attempting to locate events.


we had a little go around trying to figure out what you meant by events.


but i think you know what we had meant by events i guess was uh points of overlap between speakers.


but i i gather from our discussion a little earlier today that you also mean uh interruptions with something else.


like some other noise.


yeah.


yes?


uhhuh yeah.


you mean that as an event also?


so at any rate you were you've you've done some work on that.


right.


and um then the other thing would be it might be nice to have a preliminary discussion of some of the other uh research uh areas that uh we're thinking about doing.


um i think especially since you you haven't been in in these meetings for a little bit.


maybe you have some discussion of some of the the plausible things to look at now that we're starting to get data.


uh and one of the things i know that also came up uh is some discussions that that uh that uh jane had with lokendra.


uh about some some some um uh work about


i i i i i don't want to try to say because i i'll say it wrong.


but anyway some some potential collaboration there about about the about the working with these data.


oh.


so


sure.


so uh


you want to just go around?


uh well i don't know if we if this is sort of like everybody has something to contribute sort of thing.


i think there's just just a couple a couple people primarily.


um but um


uh why don't


actually i think that that last one i just said we could do fairly quickly.


so why don't you you start with that?


okay.


shall i shall i just start?


yeah just explain what it was.


um so uh he was interested in the question of you know relating to his to the research he presented recently um of inference structures.


and uh the need to build in um this this sort of uh mechanism for understanding of language.


and he gave the example in his talk about how um


i'm remembering it just off the top of my head right now.


but it's something about how um joe slipped you know john had washed the floor or something like that.


and i don't have it quite right.


but that kind of thing.


where you have to draw the inference that okay there's this time sequence.


but also the the the causal aspects of the uh floor and and how it might have been the cause of the fall.


and that um it was the other person who fell than the one who cleaned it.


and it these sorts of things.


so i looked through the transcript that we have so far and um identified a couple different types of things of that type.


and um one of them was something like uh during the course of the transcript um um we had gone through the part where everyone said which channel they were on and which device they were on.


and um the question was raised well should we restart the recording at this point.


and and dan ellis said well we're just so far ahead of the game right now we really don't need to.


now how would you interpret that without a lot of inference?


so the inferences that are involved are things like okay so how do you interpret ahead of the game.


you know so it's the it's what you what you what you draw you know the conclusions that you need to draw are that space is involved in recording.


huh metaphorically.


that um that we have enough space.


and he continues like we're so ahead of the game because now we have built in down sampling.


so you have to sort of get the idea that um ahead of the game is speaking with respect to space limitations.


that um that in fact down sampling is gaining us enough space.


and that therefore we can keep the recording we've done so far.


but there are a lot of different things like that.


so do you think his interest is in using this as a data source?


or training material?


or what?


well i i should maybe interject to say this started off with a discussion that i had with him.


so um we were trying to think of ways that his interests could interact with ours.


uhhuh.


and um uh i thought that if we were going to project into the future when we had a lot of data uh and um such things might be useful for that in before we invested too much uh effort into that he should uh with jane's help look into some of the data that we're already have.


and see is there anything to this at all.


is there any point which you think that you know you could gain some advantage and some potential use for it.


because it could be that you'd look through it and you say well this is just the wrong task for for him to pursue his


wrong.


yeah.


and and uh i got the impression from your mail that in fact there was enough things like this just in the little sample that that you looked at that that it's plausible at least.


it's possible.


uh he was he he you know we met and he was going to go and uh you know look through them more systematically.


and then uh meet again.


so it's you know not a matter of a


but yeah i think i think it was optimistic.


so anyway that's that's a quite different thing from anything we've talked about that you know might might might come out from some of this.


but he can use text basically.


i mean he's talking about just using text.


that's his major


i mentioned several that had to do with implications drawn from intonational contours.


pretty much?


or


and that wasn't as directly relevant to what he's doing.


okay.


he's interested in these these knowledge structures.


yeah interesting.


inferences that you draw from.


i mean he certainly could use text.


but we were in fact looking to see if there is there is there something in common between our interest in meetings and his interest in in in this stuff.


so


and i imagine that transcripts of speech i mean text that is speech probably has more of those than sort of prepared writing.


i i don't know whether it would or not but it seems like it would.


i don't know.


probably probably depends on what the prepared writing was.


but


yeah i don't think i would make that leap.


because in narratives you know i mean if you spell out everything in a narrative it can be really tedious.


uhhuh.


yeah i'm just thinking you know when you're when you're face to face you have a lot of backchannel.


and and


oh!


that aspect.


yeah.


and so i think it's just easier to do that sort of broad inference jumping if it's face to face.


i mean so if i just read that dan was saying we're ahead of the game in that in that context.


well


i might not realize that he was talking about disk space as opposed to anything else.


i you know i i had several that had to do with backchannels and this wasn't one of them.


this this one really does um make you leap from


so he said you know we're ahead of the game we have built in down sampling.


and the inference if you had it written down would be


i guess it would be the same.


but there are others that have backchanneling.


it's just he was less interested in those.


can i


sorry to interrupt.


um i i've a minute uh several minutes ago i like briefly was was not listening.


and so who is he in this context?


yeah there's a lot of pronoun


okay.


so i was just realizing we've you guys have been talking about he um for at least uh i don't know three three four minutes without ever mentioning the person's name again.


i believe it.


yeah actually to make it worse uh morgan uses you and you.


so this is this is this is going to be a big big problem if you want to later do uh you know indexing or speech understanding of any sort.


it's in my notes.


with gaze and no identification.


or


i just wrote this down.


you just wrote this?


yeah actually because morgan will say well you had some ideas.


yeah.


and he never said he looked


well i think he's doing that intentionally.


right.


aren't you?


so it's great.


so this is really great.


because the thing is because he's looking at the even for addressees in the conversation.


uhhuh.


i bet you could pick that up in the acoustics.


just because your gaze is also correlated with the directionality of your voice.


could be.


yeah that would be


oh that would be interesting.


can


yeah so that i mean to even know um when


yeah if you have the p z m's you should be able to pick up what a person is looking at from their voice.


well especially with morgan with the way we have the microphones arranged i'm sort of right on axis.


and it would be very hard to tell.


uh


oh but you'd have the


put morgan always like this.


and


you'd have fainter


well these


wouldn't you get fainter reception out here?


sure but i think if i'm talking like this


right now i'm looking at jane and talking.


now i'm looking at chuck and talking.


i don't think the microphones would pick up that difference.


but you don't have this this problem.


i see.


morgan is the one who does this most.


so if i'm talking at you or i'm talking at you.


i probably been no i i think i've been affected by too many conversations where we were talking about lawyers and talking about and concerns about oh gee is somebody going to say something bad and so on.


lawyers.


and so i so i'm i'm tending to stay away from people's names even though uh


i am too.


even though you could pick up later on just from the acoustics who you were who you were looking at.


and we did mention who he was.


yeah.


early in the conversation.


right but i missed it.


but it was uh


do can i say


yeah yeah.


or or is that just too sensitive?


yeah no no there's


no no it isn't sensitive at all.


well


i was just i was just i was overreacting just because we've been talking about it.


and in fact it is it is it is sensitive.


it's okay to


no but that


it's interesting.


i i came up with something from the human subjects people that i wanted to mention.


i mean it fits into the area of the mundane.


but they did say


you know i asked her very specifically about this clause of how um you know it says no individuals will be identified.


uh in any publication using the data.


okay well individuals being identified.


let's say you have a a snippet that says joe uh thinks such and such about about this field but i think he's wrongheaded.


now i mean we're we're going to be careful not to have the wrongheaded part in there.


but but you know let's say we say you know joe used to think so-and-so about this area in his publication he says that but i think he's changed his mind or whatever.


but i


then the issue of of being able to trace joe because we know he's well known in this field and all this and and tie it to the speaker whose name was just mentioned a moment ago can be sensitive.


so i think it's really really kind of adaptive and wise to not mention names any more than we have to.


because if there's a slanderous aspect to it then how much to we want to be able to have to remove.


yeah well there's that.


but i i mean i think also to some extent it's just educating the human subjects people in a way.


because there's if uh you know there's court transcripts there's there's transcripts of radio shows.


i mean people say people's names all the time.


so i think it it can't be bad to say people's names.


it's just that i mean you're right that there's more if we never say anybody's name then there's no chance of of of slandering anybody.


but then it won't i mean if we if we


but


it's not a meeting.


yeah i mean we should do whatever's natural in a meeting if if we weren't being recorded.


yeah.


right so i so my behavior is probably not natural.


so


if person x


well my feeling on it was that it wasn't really important who said it.


you know?


well if you since you have to um go over the transcripts later anyway you could make it one of the jobs of the people who do that to mark


well we we we talked about this during the anonymization.


if we want to go through and extract from the audio and the written every time someone says a name.


right.


and i thought that our conclusion was that we didn't want to do that.


yeah we really can't.


but actually i'm sorry i really would like to push finish this off.


i understand.


no i just i just was suggesting that it's not a bad policy potentially.


so it's


so we need to talk about this later.


yeah i i didn't intend it an a policy though.


it was it was just it was just unconscious well semi conscious behavior.


uhhuh.


i sort of knew i was doing it but it was


well i still don't know who he is.


i i i don't remember who he is.


no you have to say you still don't know who he is with that prosody.


uh.


uh we were talking about dan at one point and we were talking about lokendra at another point.


oh.


yeah depends on which one you mean.


and i don't i don't remember which which part.


it's ambiguous so it's okay.


uh i think


well the inference structures was lokendra.


but no the inference stuff was was was lokendra.


okay.


yeah yeah yeah.


that makes sense yeah.


um


and the down sampling must have been dan.


yeah.


good.


it's an inference.


yeah you could do all these inferences.


um i i would like to move it into into uh what jose uh has been doing.


because he's actually been doing something.


uhhuh.


so right.


as opposed to the rest of us.


i i remind that me my first objective uh in the project is to to study difference parameters.


to to find a a good solution to detect uh the overlapping zone in uh speech recorded.


but uh tsk uh in that way i i i begin to to study and to analyze the ehn the recorded speech uh the different session.


to to find and to locate and to mark uh the the different overlapping zone.


and uh so uh i was uh i am transcribing the the first session.


and i i have found uh uh one thousand acoustic events.


uh besides the overlapping zones uh i i i mean the uh breaths uh aspiration uh uh talk uh uh clap.


uh i don't know what is the different names uh you use to to name the the speech.


non speech sounds.


yeah.


oh i don't think we've been doing it at that level of detail.


so


uh i i i i don't need to to to huh to to label the the different acoustic.


but i prefer because uh i would like to to study if uh i i will find uh uh a good uh parameters uh to detect overlapping.


i would like to to to test these parameters uh with the another uh uh acoustic events.


to nnn to uh to find what is the um the false uh the false uh hypothesis uh nnn which uh are produced when we use the the um this uh parameter uh i mean pitch uh uh difference uh feature.


uhhuh.


you know i think some of these um that are the non speech overlapping events may be difficult even for humans to tell that there's two there.


so it was


i mean if it's a tapping sound you wouldn't necessarily or you know something like that it'd be it might be hard to know that it was two separate events.


well you weren't talking about just overlaps.


were you?


you were just talking about acoustic events.


i i i i i i talk uh about uh acoustic events in general.


someone starts someone stops.


but uh my my objective uh will be uh to study uh overlapping zone.


oh.


uh uh in twelve minutes i found uh uh one thousand acoustic events.


how many overlaps were there uh in it?


no no how many of them were the overlaps of speech though?


how many?


uh almost uh three hundred uh in one session.


oh god!


in five uh in forty five minutes.


ugh.


three hundred overlapping speech.


three hundred overlapping zone.


overlapping speech.


with the overlapping zone overlapping speech speech what uh different duration.


uhhuh.


sure.


does this


so if you had an overlap involving three people how many times was that counted?


yeah three people two people.


uh um i would like to consider uh one people with difference noise uh in the background.


no no but i think what she's asking is if at some particular for some particular stretch you had three people talking instead of two did you call that one event?


oh.


yeah.


i consider one event uh for for that uh for all the zone.


this i i i i consider i consider uh an acoustic event the overlapping zone the period where three speaker or uh are talking together.


well so let's


so let's say me and jane are talking at the same time.


and then liz starts talking also over all of us.


how many events would that be?


i don't understand.


so two people are talking and then a third person starts talking.


yeah?


is there an event right here?


uh no no no.


for me is the overlapping zone.


because because you you have you have more one uh more one voice uh uh produced in a in in a moment.


so if two or more people are talking


i see.


okay.


yeah so i think yeah we just wanted to understand how you're defining it.


yeah.


so then in the region between since there is some continuous region in between regions where there is only one person speaking.


uhhuh.


and one contiguous region like that you're calling an event.


is it are you calling the beginning or the end of it the event?


or are you calling the entire length of it the event?


i consider the the nnn the nnn nnn uh the entirety.


uh uh all all the time there were the voice has overlapped.


this is the idea.


but uh i i don't distinguish between the the numbers of uh speaker.


uh i'm not considering uh the the um uh the fact of uh uh for example what did you say.


uh at first uh uh two talkers are uh uh speaking.


and uh uh a third person uh join to to that.


for me it's uh it's uh all overlap zone with uh several numbers of speakers is uh uh the same acoustic event.


but uh without any mark between the zone of the overlapping zone with two speakers uh speaking together.


and the zone with the three speakers.


that would just be one.


it one one.


okay.


uh with uh a beginning mark and the ending mark.


got it.


because uh for me is the is the zone with uh some kind of uh distortion the spectral.


i don't mind by the moment by the moment.


well but but you could imagine that three people talking has a different spectral characteristic than two.


i i don't


yeah but uh but uh i have to study what will happen in a general way.


could.


so you had to start somewhere.


yeah we just


i i don't know what uh will will happen with the


so there's a lot of overlap.


yep.


so


that's a lot of overlap.


yeah?


so again that's that's three three hundred in forty five minutes that are that are speakers just speakers.


for forty five minutes.


yeah.


uhhuh okay yeah.


but a a a


so that's about eight per minute.


but a thousand events in twelve minutes that's


but


but that can include taps.


uh yeah.


well but a thousand taps in eight minutes is a in twelve minutes is a lot.


general


actually


i i i consider i consider acoustic events uh the silent too.


silent.


silence starting or silence ending


yeah silent ground.


to to detect uh because i consider acoustic event all the things are not uh speech.


oh okay.


uhhuh.


oh.


in in in in a general point of view.


okay so how many of those thousand were silence?


all right.


not speech not speech or too much speech.


in the


too much speech.


right.


so how many of those thousand were silence silent sections?


yeah.


uh silent.


i i i i don't i i haven't the


uh i i would like to to do a stylistic study.


and give you uh with the report uh from uh the the study from the the the session one session.


and i i found that uh another thing.


when uh uh i i i was uh look at uh nnn the difference speech file.


um for example uh if uh we use the um the mixed file to to transcribe the the events and the words i i saw that uh the uh speech signal collected by the uh this kind of mike uh of this kind of mike uh are different from the uh mixed signal uh we uh collected by headphone.


yep.


and it's right.


but the problem is the following.


the the the


i i i knew that uh the signal uh uh would be different.


but uh the the problem is uh uh we uh detected uh difference events in the speech file uh collected by by that mike uh compared with the mixed file.


and so if when you transcribe uh only uh using the nnn the mixed file it's possible uh if you use the transcription to evaluate a different system.


it's possible you uh in the uh


and you use the uh speech file collected by the uh fet mike to uh to nnn to do the experiments with the the system.


uhhuh.


right.


its possible to evaluate uh uh or to consider uh acoustic events that which you marked uh in the mixed file but uh they don't appear in the uh speech signal uh collected by the by the mike.


the the reason that i generated the mixed file was for i b m to do word level transcription not speech event transcription.


yeah.


yeah oh it's a good idea.


it's a good idea i think.


so i agree that if someone wants to do speech event transcription that the mixed signals here.


i mean if i'm tapping on the table you it's not going to show up on any of the mikes but it's going to show up rather loudly in the p z m.


so and i i i say uh that uh uh or this uh only because uh i i i in my opinion it's necessary to uh to uh to put the transcription on the speech file collected by the objective signal.


so


i mean the the the signal collected by the uh the real mike in the future in the prototype to to uh correct the initial uh segmentation uh with the uh real speech.


the the the far field.


and you have to to analyze.


you have to to process.


because i i found a difference.


yeah well just i mean just in that that one ten second or whatever it was example that adam had that that we we passed on to others a few months ago.


there was that business where i i guess it was adam and jane were talking at the same time.


and and uh in the close talking mikes you couldn't hear the overlap and in the distant mike you could.


so yeah it's clear that if you want to study if you want to find all the places where there were overlap it's probably better to use a distant mike.


that's good.


on the other hand there's other phenomena that are going on at the same time for which it might be useful to look at the close talking mikes.


yeah.


so it's


but why can't you use the combination of the close talking mikes time aligned?


if you use the combination of the close talking mikes you would hear jane interrupting me.


but you wouldn't hear the paper rustling.


and so if you're interested in


i i mean if you're interested in speakers overlapping other speakers and not the other kinds of non speech that's not a problem.


some of it's masking masked.


were you interrupting him or was he interrupting you?


right.


although the other issue is that the mixed close talking mikes


i mean i'm doing weird normalizations and things like that.


but it's known.


yep.


i mean the normalization you do is over the whole conversation.


isn't it?


over the whole meeting.


so if you wanted to study people overlapping people that's not a problem.


i i i think uh


i saw the nnn the uh


but uh i uh i have uh any results.


i i i saw the the speech file collected by uh the fet mike.


and uh uh signal uh to uh to noise uh relation is uh low.


uhhuh.


it's low.


it's very low you would if we compare it with uh the headphone.


and


i i found that nnn that uh um probably


did did you


i'm not sure uh by the moment.


but it's it's probably that uh a lot of uh uh


for example in the overlapping zone on uh in in several uh parts of the files where you you can find uh uh uh smooth uh uh speech uh from uh one uh uh talker in the in the meeting.


it's probably in in that uh in in those files you you can not find you can not process.


because uh it's confused with with noise.


there are a lot of


i think.


but i have to study with more detail.


but uh my idea is to to process only nnn this uh nnn this kind of of uh speech.


because i think it's more realistic.


i'm not sure it's a good idea but uh


well it's more realistic but it'll it'll be a lot harder.


no


well it'd be hard but on the other hand as you point out if your if if if your concern is to get uh the overlapping people people's speech you will you will get that somewhat better.


yeah.


um are you making any use


uh you were you were working with the data that had already been transcribed.


does it uh


with by jane.


yes.


now um did you make any use of that?


see i was wondering because we we have these ten hours of other stuff that is not yet transcribed.


do you


the the transcription by jane uh uh i i i want to use to to nnn uh to put


it's a reference for me.


but uh the transcription uh for example i i don't i i'm not interested in the in the in the words transcription words uh transcribed uh uh in uh follow in the in the in the speech file but uh uh jane uh for example uh put a mark uh at the beginning uh of each uh talker.


in the in the meeting.


um uh she she nnn includes information about the zone where uh there are uh there is an overlapping zone.


uhhuh.


but uh there isn't any any mark time temporal mark to to uh to huh e-heh to label the beginning and the end of the of the


okay.


right so she is


i'm i i i think uh we need this information to


right.


so the twelve you you it took you twelve hours of course this included maybe some some time where you were learning about what what you wanted to do.


but but uh it took you something like twelve hours to mark the forty five minutes your


twelve minutes.


twelve minutes!


twelve.


i thought you did forty five minutes of


no forty five minutes is the is the session.


all the session.


oh you haven't done the whole session.


oh!


yeah.


this is just twelve minutes.


all is the the session.


twelve hours of work to to segment uh and label uh twelve minutes from a session of part of


so let me back up again.


so the when you said there were three hundred speaker overlaps


that's in twelve minutes?


no no no.


i i consider all the all the session because uh i i count the nnn the nnn the overlappings marked by by jane.


oh okay.


okay.


oh i see.


in in in in the in in the forty five minutes.


so it's three hundred in forty five minutes.


but you have you have time uh uh marked twelve minute the the the um overlaps in twelve minutes of it.


got it.


well not just the overlaps everything.


so can i ask can i ask whether you found uh you know how accurate uh jane's uh uh labels were as far as


you know did she miss some overlaps or did she


but by by the moment i i don't compare my my temporal mark with uh jane but uh i i want to do it.


because uh uh perhaps i have uh errors in the in the marks.


i and if i i compare with uh jane it's probably i i i can correct and and and to get uh uh a more accurately uh uh transcription in the file.


yeah.


well also jane jane was doing word level.


so we weren't concerned with exactly when an overlap started and stopped.


right right.


i'm i'm not expecting


well


well not only a word level but actually


i mean you didn't need to show the exact point of interruption.


no it's


you just were showing at the level of the phrase or the level of the speech spurt or


right.


uhhuh.


yep.


well yeah yeah i would say time bin.


so my my goal is to get words with reference to a time bin beginning and end point.


and and sometimes you know it was like you could have an overlap where someone said something in the middle.


but yeah it just wasn't important for our purposes to have it that disrupt that unit in order to have you know the words in the order in which they were spoken.


yeah.


it would have it would have been hard with the interface that we have.


now my adam's working on a of course on a revised overlapping interface.


i i i think it's it's a good uh work.


but


but uh i think we need uh uh more information.


no of course.


i expect you to find more overlaps than than jane.


always need more for


no no i i have to go to


because you're looking at it at a much more detailed level.


i want uh i wanted to uh compare the the transcription.


but if it takes sixty to one


i have


well i but i have a suggestion about that.


um obviously this is very very time consuming and you're finding lots of things.


which i'm sure are going to be very interesting.


but in the interests of making progress uh might i


how how would it affect your time if you only marked speaker overlaps?


only.


yes.


do not mark any other events.


yeah.


but only mark speaker.


uhhuh.


do you think that would speed it up quite a bit?


okay .


okay i i i i i i wanted to


do do you think that would speed it up?


uh speed up your your your marking?


nnn i don't understand very


it took you a long time to mark twelve minutes.


oh yeah yeah.


now my suggestion was for the other thirty three


only to mark only to mark overlapping zone but


yeah and my question is if you did that if you followed my suggestion would it take much less time?


oh.


yeah okay.


sure.


yeah sure.


then i think it's a good idea.


then i think it's a good idea because


sure sure.


because i i need a lot of time to to put the label or to do that.


yeah.


yeah i mean we we know that there's noise.


uhhuh.


there's there's uh continual noise uh from fans and so forth.


and there is uh more impulsive noise from uh taps and so forth.


and and something in between with paper rustling.


we know that all that's there.


and it's a worthwhile thing to study.


but obviously it takes a lot of time to mark all of these things.


whereas i would think that uh you we can study more or less as a distinct phenomenon the overlapping of people talking.


okay.


so then you can get the because you need


if it's three hundred uh it sounds like you probably only have fifty or sixty or seventy events right now that are really


and and you need to have a lot more than that to have any kind of uh even visual sense of of what's going on much less any kind of reasonable statistics.


right.


now why do you need to mark speaker overlap by hand if you can infer it from the relative energy in the


well that's


that's what i was going to bring up.


i mean you shouldn't need to do this completely by hand.


um okay yeah so let's back up because you weren't here for an earlier conversation.


right?


i'm sorry.


so the idea was that what he was going to be doing was experimenting with different measures.


such as the increase in energy such as the energy in the l p c residuals such as


i mean there's a bunch of things i increased energy is sort of an obvious one.


uhhuh.


in the far field mike.


yeah.


oh okay.


um and uh it's not obvious.


i mean you could you could do the dumbest thing and get get it ninety percent of the time.


but when you start going past that and trying to do better it's not obvious what combination of features is going to give you the you know the right detector.


so the idea is to have some ground truth first.


and so the the idea of the manual marking was to say okay this you know it's it's really here.


but i think liz is saying why not get it out of the transcripts.


what i mean is get it from the close talking mikes.


uh yeah.


or get a first pass from those.


we we we we talked about that.


and then go through sort of


it'd be a lot faster probably to


and you can


yeah that's his uh


we we we talked about that.


but so it's a bootstrapping thing and the thing is


yeah i just


the idea was we we thought it would be useful for him to look at the data anyway.


and and then whatever he could mark would be helpful.


right.


and we could uh


it's a question of what you bootstrap from.


you know do you bootstrap from a simple measurement which is right most of the time and then you do better?


or do you bootstrap from some human being looking at it and then then do your simple measurements uh from the close talking mike?


i mean even with the close talking mike you're not going to get it right all the time.


well that's what i wonder because um


or how bad it is.


well


i'm working on a program to do that and


um because that would be interesting.


especially because the bottleneck is the transcription.


right?


i mean we've got a lot more data than we have transcriptions for.


we have the audio data.


yeah.


we have the close talking mike.


so i mean it seems like one kind of project


that's not perfect but


um that you can get the training data for pretty quickly is you know


if you infer form the close talking mikes where the on off points are of speech.


right we discussed that.


you know how can we detect that from a far field?


and


oh.


i've i've written a program to do that.


okay.


and it uh


i'm sorry i missed the


it's okay.


and so but it's it's doing something very very simple.


it just takes a threshold based on on the volume.


uhhuh.


or you can set the threshold low and then weed out the false alarms by hand.


right.


by hand yeah.


yeah.


um and then it does a median filter and then it looks for runs.


and it seems to work.


i've i'm sort of fiddling with the parameters to get it to actually generate something.


and i haven't


i don't what i'm working on was working on was getting it to a form where we can import it into the user interface that we have into transcriber.


and so i told i said it would take about a day.


i've worked on it for about half a day.


i have to go.


so give me another half day and we'll have something we can play with.


okay.


see this is where we really need the meeting recorder query stuff to be working.


because we've had these meetings and we've had this discussion about this and i'm sort of remembering a little bit about what we decided.


i'm sorry!


i just


but i couldn't remember all of it.


it


so i think it was partly that you know give somebody a chance to actually look at the data and see what these are like partly that we have some ground truth to compare against you know when when he he gets his thing going.


but


uh and


well it's definitely good to have somebody look at it.


i was just thinking as a way to speed up you know the amount of


but that was that was exactly the notion that that that we discussed.


uhhuh.


thanks.


another thing we discussed was um that


so


it looks good.


i'll be in touch.


see ya.


yeah.


was that um there there was this already a script i believe uh that dan had written that uh handle bleedthrough.


i mean because you have this this close you have contamination from other people who speak loudly.


and i haven't tried using that.


it would probably help the program that i'm doing to first feed it through that.


it's a cross correlation filter.


so i i haven't tried that but that if it it might be something it might be a good way of cleaning it up a little.


so some thought of maybe having


having that be a preprocessor and then run it through yours.


exactly yep.


but but that's a refinement.


that's what we were discussing.


and i think we want to see try the simple thing first.


because you add this complex thing up uh afterwards that does something good you sort of want to see what the simple thing does first.


yep.


but uh having having somebody have some experience again with with uh with marking it from a human standpoint.


we're i mean i don't expect jose to to do it for uh fifty hours of of speech.


but i mean we if uh if he could speed up what he was doing by just getting the speaker overlaps so that we had it say for forty five minutes then at least we'd have three hundred examples of it.


sure.


and when when uh adam was doing his automatic thing he could then compare to that and see what it was different.


oh yeah definitely.


you know i did i did uh something almost identical to this at one of my previous jobs and it works pretty well.


i mean almost exactly what you described an energy detector with a median filter you look for runs.


and uh


you know you can


it seemed like the right thing to do.


yeah i mean you you can get i mean you get them pretty close.


that was with zero literature search.


and so i think doing that to generate these possibilities and then going through and saying yes or no on them would be a quick way to to do it.


that's good validation.


yeah.


is this proprietary?


yeah do you have a patent on it?


uh no no.


it was when i was working for the government.


oh then everybody owns it.


it's the people.


well i mean is this something that we could just co opt?


or is it


no.


nah.


okay.


well i he's pretty close anyway i think i think it's


yeah he's it it doesn't take a long time.


right.


i just thought if it was tried and true then and he's gone through additional levels of of development.


just output.


although if you if you have some parameters like what's a good window size for the median filter?


oh i have to remember.


i'll think about it.


and try to remember.


and it might be different for government people.


that's all right.


yeah good enough for government work as they say.


they they


different different bandwidth.


they


i was doing pretty short you know tenth of a second sorts of numbers.


okay.


uh i don't know it if if we want to uh


so uh maybe we should move on to other other things in limited time.


can i ask one question about his statistics?


so so in the twelve minutes um if we took three hundred and divided it by four which is about the length of twelve minutes um i'd expect like there should be seventy five overlaps.


yeah.


did you find uh more than seventy five overlaps in that period or


more than?


more than


how many overlaps in your twelve minutes?


how many?


uh not


only i i transcribe uh only twelve minutes from the


but uh i i don't uh i don't count uh the the overlap.


the overlaps.


i consider i i the the nnn the the three hundred is uh considered only you your transcription.


i have to to finish transcribing.


so


i i bet they're more because the beginning of the meeting had a lot more overlaps than than sort of the middle.


middle or end.


because we're we're dealing with the uh in the early meetings.


i'm not sure.


we're recording while we're saying who's talking on what microphone and things like that.


yeah.


and that seems to be a lot of overlap.


i think it's an empirical question.


i think we could find that out.


yep.


i'm i'm not sure that the beginning had more.


so so i was going to ask i guess about any any other things that that that either of you wanted to talk about.


especially since andreas is leaving in five minutes.


that that you want to go with.


can i just ask about the data?


like very straightforward question is where we are on the amount of data and the amount of transcribed data?


just because i'm


i wanted to get a feel for that.


to sort of be able to know what what can be done first.


and like how many meetings are we recording?


right so there's this this there's this forty five minute piece that jane transcribed.


and


that piece was then uh sent to i b m so they could transcribe so we have some comparison point.


then there's a larger piece that's been recorded and uh put on c d rom and sent uh to i b m.


right?


and then we don't know.


how many meetings is that?


what's that?


like how many


that was about ten hours and there was about


ten it's like ten meetings or something?


yeah something like that.


uhhuh.


and then then we


ten meetings that have been sent to i b m?


well i haven't sent them yet because i was having this problem with the missing files.


yeah.


oh.


oh that's right that had those have not been sent.


how many total have we recorded now altogether?


we're saying about twelve hours.


about twelve by now twelve or thirteen.


and we're recording only this meeting like continuously.


we're only recording this one now or


no.


nope.


no so the the that's the that's the biggest one uh chunk so far.


okay.


it was the morning one.


but there's at least one meeting recorded of uh the uh uh natural language guys.


jerry.


do they meet every week?


and then there


uh they do.


or every


and we talked to them about recording some more.


and we're going to


uh we've started having a morning meeting today uh starting a a week or two ago on the uh front end issues.


and we're recording those.


uh there's a network services and applications group here who's agreed to have their meetings recorded.


great.


and we're going to start recording them.


they're they meet on tuesdays we're going to start recording them next week.


so actually we're going to start having a a pretty significant chunk.


and


so you know adam's sort of struggling with trying to get things to be less buggy and come up quicker when they do crash and stuff things like that.


now that uh the things are starting to happen.


so right now yeah i i'd say the data is predominantly meeting meetings.


but there are scattered other meetings in it and that that amount is going to grow.


uh so that the meeting meetings will probably ultimately


if we're if we collect fifty or sixty hours the meeting meetings it will probably be you know twenty or thirty percent of it.


not not not eighty or ninety.


but


so there's probably there's three to four a week.


that's what we're aiming for.


that we're aiming for.


yeah.


and they're each about an hour or something.


yeah yeah.


although


we'll find out tomorrow whether we can really do this or not.


so


yeah and the the other thing is i'm not


okay.


i'm sort of thinking as we've been through this a few times that i really don't know


maybe you want to do it once for the novelty.


but i don't know if in general we want to have meetings that we record from outside this group do the digits.


right.


because it's just an added bunch of weird stuff.


and you know we we we're highly motivated.


uh in fact the morning group is really motivated.


because they're working on connected digits.


so it's


actually that's something i wanted to ask.


is i have a bunch of scripts to help with the transcription of the digits.


yeah.


we don't have to hand transcribe the digits because we're reading them and i have those.


and so i have some scripts that let you very quickly extract the sections of each utterance.


but i haven't been i haven't been doing that.


um if i did that is someone going to be working on it?


uh yeah i think.


i mean is it something of interest?


definitely absolutely.


yeah whoever we have working on the acoustics for the meeting recorder are going to start with that.


okay.


i mean i'm i'm interested in it.


i just don't have time to do it now.


i was these meetings i'm sure someone thought of this but these this uh reading of the numbers would be extremely helpful to do um adaptation.


so


yep yep.


um


actually i have


i i would really like someone to do adaptation.


uhhuh.


so if we got someone interested in that i think it would be great for meeting recorder.


well


i mean one of the things i wanted to do uh that i i talked to to don about is one of the possible things he could do or also we could have someone else do it is to do block echo cancellation.


since it's the same people over and over.


to try to get rid of some of the effects of the the the far field effects.


um i mean we have the party line has been that echo cancellation is not the right way to handle the situation.


because people move around.


and uh if if it's if it's uh not a simple echo like a cross talk kind of echo but it's actually room acoustics it's it's it's you can't really do inversion.


and even echo cancellation is going to uh be something it may you someone may be moving enough that you are not able to adapt quickly.


and so the tack that we've taken is more lets come up with feature approaches and multi stream approaches and so forth that will be robust to it for the recognizer and not try to create a clean signal.


uh that's the party line.


but it occurred to me a few months ago that uh party lines are always you know sort of dangerous.


it's good good to sort of test them actually.


and so we haven't had anybody try to do a good serious job on echo cancellation and we should know how well that can do.


so that's something i'd like somebody to do at some point just take these digits take the far field mike signal and the close uh mike signal and apply really good echo cancellation.


um there was a have been some nice talks recently by by lucent on on their


huh.


the block echo cancellation particularly appealed to me.


uh you know trying and change it sample by sample but you have some reasonable sized blocks and um you know


what is the um the artifact you try to you're trying to get rid of when you do that?


ciao.


uh so it's it you have a a direct uh


what's the difference in if you were trying to construct a linear filter that would


um


i'm signing off.


yeah.


that would subtract off the um uh parts of the signal that were the aspects of the signal that were different between the close talk and the distant.


you know so so uh um i guess in most echo cancellation.


yeah so you given that um


yeah so you're trying to so you'd there's a a distance between the close and the distant mikes so there's a time delay there.


and after the time delay there's these various reflections.


and if you figure out well what's the


there's a a least squares algorithm that adjusts itself adjusts the weight so that you try to subtract essentially to subtract off uh different uh different reflections.


right so let's take the simple case where you just had you had some uh some delay in a satellite connection or something.


and then there's a there's an echo.


it comes back and you want to adjust this filter so that it will maximally reduce the effect of this echo.


so that would mean like if you were listening to the data that was recorded on one of those.


uh just the raw data.


you would you might hear kind of an echo?


and and then this noise cancellation would


well i'm i'm i'm saying that's a simplified version of what's really happening what's really happening is


well when i'm talking to you right now you're getting the direct sound from my speech.


but you're also getting uh the indirect sound that's bounced around the room a number of times.


okay?


so now if you um try to you to completely remove the effect of that is sort of impractical for a number of technical reasons.


but i but


not to try to completely remove it that is invert the the room response.


but just to try to uh uh eliminate some of the the effect of some of the echos.


um a number of people have done this.


so that say if you're talking to a speakerphone uh it makes it more like it would be if you were talking right up to it.


so this is sort of the the straight forward approach.


you say i i i want to use this uh this item but i want to subtract off various kinds of echos.


so you construct a filter and you have this this filtered version uh of the speech um gets uh uh gets subtracted off from the original speech.


then you try to you try to minimize the energy in some sense.


and so um uh with some constraints.


kind of a clean up thing that


it's a clean up thing right.


okay.


so echo cancelling is is you know commonly done in telephony.


and and and it's sort of the obvious thing to do in this situation if you if you know you're going to be talking some distance from a mike.


when uh i would have meetings with the folks in cambridge when i was at b b n over the phone they had a um some kind of a special speaker phone.


and when they would first connect me it would come on and we'd hear all this noise.


yeah.


and then it was uh and then it would come on and it was very clear.


you know?


right so it's taking samples it's doing adaptation it's adjusting weights.


and then it's getting the sum.


so um uh anyway that's that's kind of a reasonable thing that i'd like to have somebody try.


somebody look and and the digits would be a reasonable thing to do that with.


i think that'd be enough data plenty of data to do that with.


and for that sort of task you wouldn't care whether it was uh large vocabulary speech or anything.


uh um


is brian kingsbury's work related to that?


or is it a different type of reverberation?


brian's kingsbury's work is an example of what we did from the opposite dogma.


right?


which is what i was calling the party line which is that uh doing that sort of thing is not really what we want.


we want something more flexible uh where people might change their position.


and there might be you know


there's also um


oh yeah!


noise.


so the echo cancellation does not really allow for noise.


it's if you have a clean situation but you just have some delays.


then we'll figure out the right the right set of weights for your taps for your filter in order to produce the effect of those those echos.


but um if there's noise then the very signal that it's looking at is corrupted.


so that it's decision about what the right you know right right uh delays are is uh is right delayed signal is is is uh is incorrect.


and so in a noisy situation um also in a in a situation that's very reverberant with long reverberation times and really long delays it's it's sort of typically impractical.


so for those kind of reasons


and also a a a complete inversion


if you actually i mentioned that it's kind of hard to really do the inversion of the room acoustics.


um that's difficult because um often times the the um the system transfer function is such that when it's inverted you get something that's unstable.


and so if you you do your estimate of what the system is and then you try to invert it you get a filter that actually uh you know rings and and uh goes to infinity.


so it's so there's there's there's that sort of technical reason and the fact that things move and there's air currents.


i mean there's all sorts of all sorts of reasons why it's not really practical.


so for all those kinds of reasons uh we we we sort of um concluded we didn't want to do inversion.


and we're even pretty skeptical of echo cancellation which isn't really inversion.


and um we decided to do this approach of taking uh just picking uh features which were uh will give you more something that was more stable in the presence of or absence of room reverberation.


and that's what brian was trying to do.


so um let me just say a couple things that i was i was going to bring up.


uh let's see i guess you you actually already said this thing about the uh about the consent forms.


which was that we now don't have to


so this was the human subjects folks who said this or that that


the apparently i mean we're going to do a revised form of course.


um


but once a person has signed it once then that's valid for a certain number of meetings.


she wanted me to actually estimate how many meetings and put that on the consent form.


i told her that would be a little bit difficult to say.


so i think from a practical standpoint maybe we could have them do it once every ten meetings or something.


it won't be that many people who do it that often.


but um just you know so long as they don't forget that they've done it i guess.


okay.


um back on the data thing.


so there's this sort of one hour ten hour a hundred hour sort of thing that that we have.


we have we have an hour uh that that is transcribed.


we have we have twelve hours that's recorded but not transcribed.


and at the rate we're going uh by the end of the semester we'll have i don't know forty or fifty or something if we if this really.


uh


well do we have that much?


not really.


let's see.


we have


it's three to four per week.


so that's what you know that


uh eight weeks uh is


so that's not a lot of hours.


eight weeks times three hours is twenty four so that's yeah so like thirty thirty hours.


three three hours.


yeah.


i mean is there


i know this sounds tough but we've got the room set up.


um i was starting to think of some projects where you would use


well similar to what we talked about with uh energy detection on the close talking mikes.


there are a number of interesting questions that you can ask about how interactions happen in a meeting that don't require any transcription.


so what are the patterns the energy patterns over the meeting?


and i'm really interested in this but we don't have a whole lot of data.


so i was thinking you know we've got the room set up.


and you can always think of also for political reasons if icsi collected you know two hundred hours that looks different than forty hours even if we don't transcribe it ourselves.


but i don't think we're going to stop at the end of this semester.


so


right?


so i i think that if we are able to keep that up for a few months we are going to have more like a hundred hours.


especially meetings that have some kind of conflict in them or some kind of


i mean that are less


well i don't


uh that have some more emotional aspects to them?


or strong


we had some good ones earlier.


there's laughter.


um i'm talking more about strong differences of opinion meetings.


maybe with manager types or


i think it's hard to record those.


to be allowed to record them?


yeah people will get


okay.


it's also likely that people will cancel out afterwards.


but i but i wanted to raise the k p f a idea.


well if there is any way.


yeah i was going to mention that.


oh that's a good idea.


that's that would be a good match.


yeah so yeah so i i uh i i'd mentioned to adam and that was another thing i was going to talk uh mention to them before that uh there's uh it it it occurred to me that we might be able to get some additional data by talking to uh acquaintances in local broadcast media.


because you know we had talked before about the problem about using found data that that uh it's just set up however they have it set up.


and we don't have any say about it.


and it's typically one microphone.


in a uh uh or and and so it doesn't really give us the the the uh characteristics we want.


um and so i do think we're going to continue recording here and record what we can.


but um it did occur to me that we could go to friends in broadcast media and say hey you have this panel show or this you know this discussion show and um can you record multi channel.


and uh they may be willing to record it uh with


lapel mikes or something?


well they probably already use lapel.


but they might be able to have it it wouldn't be that weird for them to have another mike that was somewhat distant.


right.


it wouldn't be exactly this setup but it would be that sort of thing.


and what we were going to get from u w you know assuming they they they start recording isn't also is not going to be this exact setup.


no i think that'd be great.


so i i i i was thinking of looking into that.


if we can get more data.


the other thing that occurred to me after we had that discussion in fact is that it's even possible since of course many radio shows are not live uh that we could invite them to have like some of their record some of their shows here.


wow!


well or the thing is they're not as averse to wearing one of these head mount


right as we are.


i mean they're on the radio.


right?


so um i think that'd be fantastic.


because those kinds of panels and those have interesting


yeah.


that's an a side of style a style that we're not collecting here.


so


and and the i mean the other side to it was the what which is where we were coming from i'll i'll talk to you more about it later is that is that there's there's uh


it'd be great.


the radio stations and television stations already have stuff worked out presumably uh related to you know legal issues and and permissions and all that.


i mean they already do what they do do whatever they do.


so it's uh it's so it's so it's another source.


so i think it's something we should look into.


you know we'll collect what we collect here.


hopefully they will collect more at u w also.


and um and maybe we have this other source.


but yeah i think that it's not unreasonable to aim at getting you know significantly in excess of a hundred hours.


i mean that was sort of our goal.


the thing was i was hoping that we could in the under this controlled situation we could at least collect you know thirty to fifty hours.


and at the rate we're going we'll get pretty close to that i think this semester.


and if we continue to collect some next semester i think we should uh


right.


yeah i was mostly trying to think okay if you start a project within say a month you know how much data do you have to work with.


and you you want to you want to sort of freeze your your data for awhile.


so um right now


and we don't have the transcripts back yet from i b m.


right?


well we don't even have it for this you know forty five minutes that was


do oh do we now


so um not complaining i was just trying to think you know what kinds of projects can you do now versus uh six months from now.


yeah.


and they're pretty different because


yeah so i was thinking right now it's sort of this exploratory stuff where you you look at the data you use some primitive measures and get a feeling for what the scatter plots look like.


um


and and and uh and meanwhile we collect and it's more like yeah three months from now or six months from now you can you can do a lot of other things.


right right.


because i'm not actually sure just logistically that i can spend


you know i don't want to charge the time that i have on the project too early before there's enough data to make good use of the time.


and that's and especially with the student


right.


uh for instance this guy who seems.


uh


anyway i shouldn't say too much.


but um if someone came that was great and wanted to do some real work and they have to end by the end of this school year in the spring how much data will i have to work with with that person?


and so it's


yeah so i would think exploratory things now uh three months from now


um i mean the transcriptions i think are a bit of an unknown because we haven't gotten those back yet as far as the timing.


but i think as far as the collection it doesn't seem to me like uh unreasonable to say that uh in january you know roughly uh which is roughly three months from now we should have at least something like you know twenty five thirty hours.


and we just don't know about the transcription part of that.


so that's


yeah we need to i think that there's a possibility that the transcript will need to be adjusted afterwards.


so


i mean it


and uh especially since these people won't be uh used to dealing with multi channel uh transcriptions.


yeah.


so i think that we'll need to adjust some.


and also if we want to add things like um well more refined coding of overlaps then definitely i think we should count on having an extra pass through.


i wanted to ask another aspect of the data collection.


there'd be no reason why a person couldn't get together several uh you know friends.


and come and argue about a topic if they wanted to.


right?


if they really have something they want to talk about as opposed to something uh


i mean what we're trying to stay away from was artificial constructions.


but i think if it's a real


why not?


i mean i'm thinking politically.


stage some political debates.


you could do this.


well yeah.


you know you could.


or just if you're if you if there are meetings here that happen that we can record even if we don't um have them do the digits or maybe have them do a shorter digit thing like if it was you know uh one string of digits or something they'd probably be willing to do.


we don't have to do the digits at all if we don't want to.


then having the data is very valuable.


because i think it's um politically better for us to say we have this many hours of audio data especially with the i t r if we put in a proposal on it.


it'll just look like icsi's collected a lot more audio data.


um whether it's transcribed or not um is another issue.


but there's there are research questions you can answer without the transcriptions or at least that you can start to answer.


yep.


it seems like you could hold some meetings.


so


you know you and maybe adam.


you you could you could maybe hold some additional meetings if you wanted.


would it help at all


i mean we're already talking about sort of two levels of detail in meetings.


one is uh um without doing the digits or i guess the full blown one is where you do the digits and everything and then talk about doing it without digits.


what if we had another level just to collect data which is without the headsets and we just did the table mounted stuff?


need the close talking mikes.


you do okay.


i mean absolutely.


yeah yeah.


yeah i'm really scared


it seems like it's a big part of this corpus is to have the close talking mikes.


um or at least like me personally i would i couldn't use that data.


i see okay.


yeah.


i agree.


um


and mari also.


we had this came up when she was here.


that's important.


so it's a great idea.


i i by the by the way i don't think the transcriptions are actually in the long run such a big bottleneck.


and if it were true than i would just do that.


but it's not that bad.


like the room is not the bottleneck.


and we have enough time in the room.


it's getting the people to come in and put on the and get the setup going.


i think the issue is just that we're we're blazing that path.


right?


and and um do you have any idea when when uh the you'll be able to send uh the ten hours to them?


well i've been burning two c d's a day which is about all i can do with the time i have.


yeah.


so it'll be early next week.


yeah okay.


so early next week we send it to them.


and then then we check with them to see if they've got it.


and we we start you know asking about the timing for it.


yep.


so i think once they get it sorted out about how they're going to do it which i think they're pretty well along on because they were able to read the files and so on.


yeah but


well


yeah who knows where they are?


have they ever responded to you?


nope.


yeah but you know so they they they have you know they're volunteering their time and they have a lot of other things to do.


what if


yeah you we can't complain.


right?


but they but at any rate they'll i i think once they get that sorted out they're they're making cassettes there then they're handing it to someone who they who's who is doing it.


and uh i think it's not going to be i don't think it's going to be that much more of a deal for them to do thirty hours then to do one hour.


i think it's not going to be thirty


yep i think that's probably true.


really?


so it's the amount of


it's it's just getting it going.


it's pipeline pipeline issues.


right.


what about these lunch meetings?


once the pipeline fills


i mean i don't know if there's any way without too much more overhead even if we don't ship it right away to i b m even if we just collect it here for awhile to record you know two or three more meeting a week.


just to have the data even if they're um not doing the digits but they do wear the headphones.


but the lunch meetings are pretty much one person getting up and


no i meant


um sorry the meetings where people eat their lunch downstairs.


maybe they don't want to be recorded but


oh and we're just chatting?


just the the chatting.


yeah we have a lot of those.


i actually i actually think that's useful data um the chatting.


yeah the problem with that is i would i think i would feel a little constrained to you know uh some of the meetings.


but


okay.


you don't want to do it because


you know our soccer ball meeting?


i guess none of you were there for our soccer ball meeting.


all right.


all right so i'll just throw it out there.


that was hilarious.


if anyone knows of one more or two more meetings per week that happen at icsi um that we could record i think it would be worth it.


yeah well we should also check with mari again because they because they were really intending you know maybe just didn't happen but they were really intending to be duplicating this in some level.


so then that would double what we had.


and there's a lot of different meetings at u w.


uh i mean really a lot more than we have here right because we're not right on campus.


right.


so


is the uh notion of recording any of chuck's meetings dead in the water?


or is that still a possibility?


uh they seem to have some problems with it we can we can talk about that later.


um but again jerry is jerry's open.


so i mean we have two speech meetings one uh network meeting.


uh jerry was open to it but i


one of the things that i think is a little a little bit of a limitation there is a think when the people are not involved uh in our work we probably can't do it every week.


you know i i i i think that that people are going to feel uh are going to feel a little bit constrained.


now it might get a little better if we don't have them do the digits all the time.


yep.


and the then so then they can just really sort of try to put the mikes on and then just charge in and


what if we give people you know we cater a lunch in exchange for them having their meeting here or something?


well you know i i do think eating while you're doing a meeting is going to be increasing the noise.


okay.


all right all right all right.


but i had another question which is um you know in principle um i know that you don't want artificial topics.


but um it does seem to me that we might be able to get subjects from campus to come down and do something that wouldn't be too artificial.


i mean we could political discussions or or something or other.


no definitely.


and


you know people who are


because you know there's also this constraint we


it's like you know the the uh goldibears goldi goldilocks.


it's like you don't want meetings that are too large but you don't want meetings that are too small.


and um and it just seems like maybe we could exploit the human subject pool in the positive sense of the word.


well even i mean coming down from campus is sort of a big thing but what about


we could pay subjects.


or what about people in the in the building?


yeah i was thinking there's all these other


i mean there's the state of california downstairs and


yeah.


i mean


i just really doubt that uh any of the state of california meetings would be recordable and then releasable to the general public.


oh.


uhhuh.


so i i mean i talked with some people at the haas business school who are who are interested in speech recognition.


all right well


and they sort of hummed and hawed and said well maybe we could have meetings down here.


but then i got email from them that said no we decided we're not really interested and we don't want to come down and hold meetings.


so i think it's going to be a problem to get people regularly.


what about joachim maybe he can


but but we but i think you know we get some scattered things from this and that.


and i i i do think that maybe we can get somewhere with the with the radio.


uh i have better contacts in radio than in television but


you could get a lot of lively discussions from those radio ones.


yep.


well and they're already they're these things are already recorded.


yeah.


we don't have to ask them to


even and i'm not sure how they record it but they must record from individual


well


no i'm not talking about ones that are already recorded.


i'm talking about new ones.


because because because we would be asking them to do something different.


why why not?


well we can find out.


i know for instance mark liberman was interested uh in in l d c.


getting data.


right that's the found data idea.


uh and


but what i'm saying is uh if i talk to people that i know who do these who produce these things we could ask them if they could record an extra channel let's say of a distant mike.


uhhuh.


and i think routinely they would not do this.


so since i'm interested in the distant mike stuff i want to make sure that there is at least that somewhere.


right.


great okay.


great.


okay.


and uh


but if we ask them to do that they might be intrigued enough by the idea that they uh might be willing to the i might be able to talk them into it.


we're getting towards the end of our disk space.


so we should think about trying to wrap up here.


okay well i don't why don't we why why don't we uh uh turn them turn


that's a good way to end a meeting.


okay leave leave them on for a moment until i turn this off because that's when it crashed last time.


oh!


that's good to know.


turning off the microphone made it crash.


well


so uh


all right.


um so i wanted to discuss digits briefly but that won't take too long.


oh good.


right.


okay agenda items.


uh we have digits.


what else we got?


new version of the presegmentation.


new version of presegmentation.


um do we want to say something about the an update of the uh transcript?


yeah why don't you summarize the


update on transcripts.


and i guess that includes some the filtering for the the a s i refs too.


huh.


filtering for what?


for the references that we need to go from the the fancy transcripts to the sort of brain dead.


it'll it'll be basically it'll be a recap of a meeting that we had jointly this morning.


uhhuh.


with don as well.


got it.


anything else more pressing than those things?


so so why don't we just do those?


you said yours was brief so


okay.


okay well the uh as you can see from the numbers on the digits we're almost done.


the digits goes up to about four thousand.


um and so uh we probably will be done with the t i digits in um another couple weeks um depending on how many we read each time.


so there were a bunch that we skipped.


you know someone fills out the form and then they're not at the meeting and so it's blank.


um but those are almost all filled in as well.


and so once we're it's done it would be very nice to train up a recognizer and actually start working with this data.


so we'll have a corpus that's the size of t i digits?


and so


one particular test set of t i digits.


test set okay.


so i i extracted there was a file sitting around which people have used here as a test set.


it had been randomized and so on.


uhhuh.


and that's just what i used to generate the order of these particular ones.


great.


um.


so i'm impressed by what we could do is take the standard training set for t i digits train up with whatever you know great features we think we have uh for instance and then test on uh this test set.


and presumably uh it should do reasonably well on that.


and then presumably we should go to the distant mike and it should do poorly.


yeah.


right.


and then we should get really smart over the next year or two and it that should get better.


and increase it by one or two percent yeah.


yeah yeah.


um but in order to do that we need to extract out the actual digits.


um so that the reason it's not just a transcript is that there're false starts and misreads and miscues and things like that.


and so i have a set of scripts and x waves where you just select the portion hit r.


um it tells you what the next one should be and you just look for that.


you know so it it'll put on the screen the next set is six nine nine two two.


and you find that and hit the key and it records it in a file in a particular format.


so is this


and so the the question is should we have the transcribers do that or should we just do it?


well some of us.


i've been i've done eight meetings something like that just by hand.


just myself rather.


so it will not take long.


um


uh what what do you think?


my feeling is that


we discussed this right before coffee.


and i think it's a it's a fine idea partly because um it's not unrelated to their present skill set.


but it will add for them an extra dimension it might be an interesting break for them.


and also it is contributing to the uh composition of the transcript.


because we can incorporate those numbers directly and it'll be a more complete transcript.


so i'm i think it's fine that part.


there is there is


so you think it's fine to have the transcribers do it?


uhhuh.


yeah okay.


there's one other small bit which is just entering the information which at which is at the top of this form onto the computer to go along with the where the digits are recorded automatically.


good.


yeah.


and so it's just you know typing in name times time date and so on.


um which again either they can do but it is you know firing up an editor or again i can do.


or someone else can do.


and that you know i'm not that that one i'm not so sure if it's into the the things that i wanted to use the hours for.


because the the time that they'd be spending doing that they wouldn't be able to be putting more words on.


huh.


but that's really your choice it's your


so are these two separate tasks that can happen?


or do they have to happen at the same time before


no they don't have this you have to enter the data before you do the second task but they don't have to happen at the same time.


so it's it's just i have a file which has this information on it and then when you start using my scripts for extracting the times it adds the times at the bottom of the file.


okay.


and so um i mean it's easy to create the files and leave them blank and so actually we could do it in either order.


oh okay.


um it's it's sort of nice to have the same person do it just as a double check to make sure you're entering for the right person.


but either way.


yeah.


yeah just by way of uh uh a uh order of magnitude uh um we've been working with this aurora uh data set.


and uh the best score on the nicest part of the data that is where you've got training and test set that are basically the same kinds of noise and so forth uh is about.


uh


i think the best score was something like five percent uh error per digit.


so that


per digit.


you're right.


so if you were doing ten digit uh recognition you would really be in trouble.


uhhuh.


so so the the point there and this is uh car noise uh uh things but but real real situation.


well real


um the uh there's one microphone that's close that they have as as this sort of thing close versus distant.


uh but in a car.


instead of instead of having a projector noise it's it's car noise.


uh but it wasn't artificially added to get some some artificial signal to noise ratio.


it was just people driving around in a car.


so that's that's an indication


that was with many sites competing and this was the very best score and so forth.


so more typical numbers like


although the models weren't that good.


right?


i mean the models are pretty crappy?


i think that we could have done better on the models.


but the thing is that we got this this is the kind of typical number for all of the uh uh things in this task all of the um languages.


and so i i think we'd probably the models would be better in some than in others.


huh.


um so uh anyway just an indication once you get into this kind of realm even if you're looking at connected digits it can be pretty hard.


it's going to be fun to see how we compare at this.


yeah.


how did we do on the t i digits?


very exciting.


well the prosodics are so much different.


it's going to be strange.


i mean the prosodics are not the same as t i digits for example.


so i'm i'm not sure how much of effect that will have.


what do you mean the prosodics?


how do


um just what we were talking about with grouping.


that with these the grouping there's no grouping at all and so it's just the only sort of discontinuity you have is at the beginning and the end.


so what are they doing in aurora?


are they reading actual phone numbers?


aurora i don't know.


i don't know what they do in aurora.


or a a digit at a time or


uh i'm not sure how


because it's


no no i mean it's connected it's connected uh digits.


connected.


yeah.


so there's also the not just the prosody but the cross the cross word modeling is probably quite different.


but


but right.


but in t i digits they're reading things like zip codes and phone numbers and things like that.


how


right.


so it's going to be different.


how do we do on t i digits?


i don't remember.


i mean very good.


right?


yeah i mean we were in the


one and a half percent two percent something like that?


uh i no i think we got under a percent.


oh really?


but it was but it's but i mean the very best system that i saw in the literature was a point two five percent or something that somebody had at at bell labs or uh but but uh sort of pulling out all the stops.


okay.


it strikes me that there are more each of them is more informative because it's so random.


all right.


huh.


but i think a lot of systems sort of get half a percent or three quarters a percent.


and we're we're in there somewhere.


but that i mean it's really it's it's close talking mikes no noise clean signal just digits i mean everything is good.


yeah.


it's the beginning of time in speech recognition.


yes exactly.


and we've only recently got it to anywhere near human.


it's like the single cell you know it's the beginning of life.


prehistory.


and it's still like an order of magnitude worse than what humans do.


right.


so


when when they're wide awake yeah.


um.


after coffee.


after coffee right.


not after lunch.


okay so um what i'll do then is i'll go ahead and enter this data.


and then hand off to jane and the transcribers to do the actual extraction of the digits.


one question i have that that i mean we wouldn't know the answer to now but might do some guessing but i was talking before about doing some modeling of uh uh marking of articulatory features with overlap and so on.


huh.


and and um


on some subset.


one thought might be to do this uh on on the digits or some piece of the digits.


uh it'd be easier uh and so forth.


the only thing is i'm a little concerned that maybe the kind of phenomena in


the reason for doing it is because the the argument is that certainly with conversational speech the stuff that we've looked at here before um just doing the simple mapping from um the phone to the corresponding features that you could look up in a book uh isn't right.


it isn't actually right.


in fact there's these overlapping processes where some voicing some up and then some you know some nasality is comes in here and so forth.


and you do this gross thing saying well i guess it's this phone starting there.


so uh that's the reasoning.


but it could be that when we're reading digits because it's it's for such a limited set that maybe maybe that phenomenon doesn't occur as much.


i don't know.


anybody do you have any anybody have any opinion about that?


and that people might articulate more.


and you that might end up with more a closer correspondence.


uhhuh.


yeah.


yeah that's i i agree.


that it's just


sort of less predictability.


and you have to


well it's definitely true that when people are reading even if they're re reading what they had said spontaneously that they have very different patterns.


it's a well would this corpus really be the right one to even try that on?


mitch showed that and some dissertations have shown that.


right.


so the fact that they're reading first of all whether they're reading in a room of people or you know just the fact that they're reading will make a difference.


and depends what you're interested in.


see i don't know.


so maybe the thing will be do to take some very small subset mean not have a big program but take a small set uh subset of the conversational speech and a small subset of the digits.


and look and and just get a feeling for it.


um just take a look really.


that could could be an interesting design too because then you'd have the the comparison of the uh predictable speech versus the less predictable speech.


because i don't think anybody is i at least i don't know of anybody uh


well i don't know the answers.


hey


yeah.


and maybe you'd find that it worked in in the case of the of the uh non predictable.


have to think about the particular acoustic features to mark too because i mean some things they wouldn't be able to mark like uh you know uh tense lax.


uhhuh.


some things are really difficult.


you know?


just listening.


i think we can get ohala in to give us some advice on that.


well


also i thought you were thinking of a much more restricted set of features that


yeah but i i i i was like he said i was going to bring john in and ask john what he thought.


yeah sure.


sure.


right.


but i mean you want you want it be restrictive but you also want it to to to have coverage.


you know you should it should be such that if you if you uh if you had um all of the features determined that you that you were uh have chosen that that would tell you uh in the steady state case uh the phone.


so um


okay.


even i guess with vowels that would be pretty hard.


wouldn't it?


to identify actually you know which one it is.


it would seem to me that the points of articulation would be more


uh i mean that's i think about articulatory features i think about points of articulation which means uh rather than vowels.


yeah.


points of articulation?


what do you mean?


so is it uh bilabial or dental or is it you know palatal.


uhhuh.


which which are all like where where your tongue comes to rest.


uvular.


place of place of articulation.


place place.


place.


thank you what whatever i said that's


okay.


i really meant place.


okay i see.


okay we got our jargon then okay.


uh


well it's also there's really a difference between the pronunciation models in the dictionary and the pronunciations that people produce.


and so you get some of that information from steve's work on the on the labeling.


right.


and it really


i actually think that data should be used more.


that maybe although i think the meeting context is great that he has transcriptions that give you the actual phone sequence.


and you can go from not from that to the articulatory features but that would be a better starting point for marking the gestural features then data where you don't have that.


because we you want to know both about the way that they're producing a certain sound and what kinds of you know what kinds of phonemic differences you get between these transcribed sequences and the dictionary ones.


well you might be right that might be the way at getting at what i was talking about.


but the particular reason why i was interested in doing that was because i remember when that happened and john ohala was over here and he was looking at the spectrograms of the more difficult ones.


uh he didn't know what to say about what is the sequence of phones there.


they came up with some compromise.


because that really wasn't what it look like.


it didn't look like a sequence of phones.


it look like this blending thing happening here and here and here.


yeah so you have this feature here and overlap yeah.


there was no name for that.


yeah.


but right.


but it still is there's a there are two steps.


one you know one is going from a dictionary pronunciation of something like going to see you tomorrow.


and or gonta.


right.


it could be going to or going to or gonta you know.


and yeah.


going to see you tomorrow uh guh see you tomorrow.


and that it would be nice to have these intermediate or these some these reduced pronunciations that those transcribers had marked or to have people mark those as well.


uhhuh.


because it's not um that easy to go from the dictionary word the dictionary phone pronunciation to the gestural one without this intermediate or a syllable level kind of representation.


well i don't think morgan's suggesting that we do that though.


do you mean


yeah i mean i'm at the moment of course we're just talking about what to provide as a tool for people to do research who have different ideas about how to do it.


so for instance you might have someone who just has a has words with states and has uh uh comes from articulatory gestures to that.


and someone else might actually want some phonetic uh intermediate thing.


so i think it would be be best to have all of it if we could.


but um


but what i'm imagining is a score like notation where each line is a particular feature.


yeah.


right?


so you would say you know it's voiced through here and so you have label here and you have nasal here.


and they they could be overlapping in all sorts of bizarre ways that don't correspond to the timing on phones.


i mean this is the kind of reason why


i remember when at one of the switchboard workshops that uh when we talked about doing the transcription project dave talkin said can't be done.


right.


he was he was what what he meant was that this isn't you know a sequence of phones.


and when you actually look at switchboard that's not what you see and you know and it


and in in fact the inter annotator agreement was not that good.


on the harder ones?


yeah i mean it


it depends how you look at it and


i i understand what you're saying about this kind of transcription exactly.


because i've seen you know where does the voicing bar start and so forth.


all i'm saying is that it is useful to have that the transcription of what was really said and which syllables were reduced.


uh if you're going to add the features.


it's also useful to have some level of representation which is is a reduced it's a pronunciation variant that currently the dictionaries don't give you.


uhhuh.


because if you add them to the dictionary and you run recognition you you add confusion.


so people purposely don't add them.


so it's useful to know which variant was was produced at least at the phone level.


so it would be it would be great if we had either these kind of labelings on the same portion of switchboard that steve marked or steve's type markings on this data with these.


right.


that's all i mean.


exactly.


yeah.


yeah no i i don't disagree with that.


and steve's type is fairly it's not that slow.


uh uh


i don't know exactly what the timing was but


yeah i don't disagree with it.


the the only thing is that what you actually will end end up with is something


it's all compromised.


right?


so the string that you end up with isn't actually what happened.


but it's it's the best compromise that a group of people scratching their heads could come up with to describe what happened.


uhhuh.


and it's more accurate than phone labels.


but and it's more accurate than the than the dictionary or if you've got a pronunciation uh lexicon that has three or four.


the word.


this might be have been the fifth one that you that you pruned or whatever.


so it's like a continuum.


right.


it's you're going all the way down.


so sure.


that's what i meant is


yeah.


and in some places it would fill in so the kinds of gestural features are not everywhere.


well


so there are some things that you don't have access to either from your ear or the spectrogram.


but you know what phone it was and that's about all you can all you can say.


and then there are other cases where nasality voicing


it's basically just having multiple levels of of information and marking on the signal.


well the other difference is that the the features are not synchronous.


right?


they overlap each other in weird ways.


uhhuh.


so it's not a strictly one dimensional signal.


so i think that's sort of qualitatively different.


right.


you can add the features in uh but it'll be underspecified.


huh.


there'll be no way for you to actually mark what was said completely by features.


well not with our current system but you could imagine designing a system that the states were features rather than phones.


and if you're


well we we've probably have a separate um discussion of uh of whether you can do that.


that's well isn't that i thought that was well but that wasn't that kind of the direction?


yeah.


i thought


yeah so i mean what what where this is i mean i i want would like to have something that's useful to people other than those who are doing the specific kind of research i have in mind.


so it should be something broader.


but the but uh where i'm coming from is uh we're coming off of stuff that larry saul did with with um uh john dalan and muzim rahim in which uh they uh have um a a multi band system that is uh trained through a combination of gradient learning and e m to um estimate uh the uh value for for for a particular feature.


okay?


and this is part of a larger image that john dalan has about how the human brain does it in which he's sort of imagining that individual frequency channels are coming up with their own estimate of of these these kinds of something like this.


might not be you know exact features that jakobson thought of or something.


but i mean you know some something like that.


some kind of low level features which are not fully you know phone classification.


and the the this particular image of how how it's done is that then given all of these estimates at that level there's a level above it then which is is making some kind of sound unit classification such as you know phone.


and and you know you could argue what what a sound unit should be and and so forth.


but that that's sort of what i was imagining doing.


um and but it's still open within that whether you would have an intermediate level in which it was actually phones or not.


you wouldn't necessarily have to.


um but again i wouldn't want to wouldn't want what we we produced to be so know local in perspective that it it was matched what we were thinking of doing one week.


and and and you know what you're saying is absolutely right.


that that if we can we should put in uh another level of of description there if we're going to get into some of this low level stuff.


well you know um i mean if we're talking about having the annotators annotate these kinds of features it seems like


you know you


the the question is do they do that on meeting data?


or do they do that on switchboard?


that's what i was saying.


maybe meeting data isn't the right corpus.


well it seems like you could do both.


i mean i was thinking that it would be interesting to do it with respect to parts of switchboard anyway in terms of


uhhuh.


uh partly to see if you could generate first guesses at what the articulatory feature would be based on the phone representation at that lower level.


it might be a time gain.


but also in terms of comparability of um


well because the yeah and then also if you did it on switchboard you would have the full continuum of transcriptions.


what you gain.


yep.


you'd have it from the lowest level the acoustic features then you'd have the you know the phonetic level that steve did.


yeah that that's all i was thinking about.


and


it is telephone band so the bandwidth might be


yeah.


and you could tell that


it'd be a complete set then.


and you get the relative gain up ahead.


it's so it's a little different.


so i mean you know we'll see how much we can uh get the people to do and how much money we'll have and all this sort of thing.


uhhuh.


but it it might be good to do what jane was saying uh you know seed it with guesses about what we think the features are based on you know the phone or steve's transcriptions or something to make it quicker.


but


might be do both.


all right so based on the phone transcripts they would all be synchronous but then you could imagine nudging them here and there.


adjusting.


yeah exactly.


scoot the voicing over a little because


right.


well i think what i mean i'm i'm a little behind in what they're doing now and uh the stuff they're doing on switchboard now.


but i think that steve and the gang are doing something with an automatic system first and then doing some adjustment.


as i as i recall.


so i mean that's probably the right way to go anyway is to is to start off with an automatic system with a pretty rich pronunciation dictionary that that um you know tries to label it all.


and then people go through and fix it.


so in in our case you'd think about us starting with maybe the regular dictionary entry?


and then


well regular dictionary i mean this is a pretty rich dictionary.


or would we?


it's got got a fair number of pronunciations in it.


or you could start from the if we were going to do the same set of sentences that steve had done we could start with those transcriptions.


but


uhhuh.


yeah.


that's actually what i was thinking is


so i was thinking


right.


the problem is when you run uh if you run a regular dictionary um even if you have variants in there which most people don't you don't always get out the actual pronunciations.


so that's why the human transcriber's giving you the that pronunciation.


actually maybe they're using phone recognizers.


oh.


and so


they they i thought that they were.


is that what they're doing?


they are.


we should catch up on what steve is


oh okay.


uh i think that would be a good good idea.


yeah so i think that we also don't have i mean


we've got a good start on it.


but we don't have a really good meeting recorder or recognizer or transcriber or anything yet.


so


yeah.


so i mean another way to look at this is to is to uh do some stuff on switchboard which has all this other stuff to it.


and then um as we get further down the road and we can do more things ahead of time we can do some of the same things to the meeting data.


uhhuh.


okay.


and i'm and these people might they they are most of them are trained with i p a.


they'd be able to do phonetic level coding or articulatory.


are they busy for the next couple years or


well you know i mean they they they're interested in continuing working with us.


so i mean i and this would be up their alley so we could when the when you meet with with john ohala and find you know what taxonomy you want to apply then they'd be good to train onto it.


anyway this is not an urgent thing at all.


just it came up.


it'd be very interesting though to have that data.


i wonder how would you do a forced alignment?


i think so too.


might


interesting idea.


to to i mean you'd want to iterate somehow.


yeah.


it's interesting thing to think about.


it might be


huh.


i was thinking it might be


i mean you'd you'd want models for spreading.


of the acoustic features?


uhhuh.


well it might be neat to do some phonetic features on these non word words.


are are these kinds of words that people never the huh s and the huh s and the huh and the uh?


these no i'm serious.


there are all these kinds of functional uh elements.


i don't know what you call them.


but not just fill pauses but all kinds of ways of interrupting and so forth.


and some of them are yeah uhhuh s and huh s and huh.


huh okay uh grunts.


uh


that might be interesting.


he's got lip lip smacks.


in the meetings.


we should move on.


yeah.


uh new version of uh presegmentation?


uh oh yeah um i worked a little bit on the on the presegmentation to to get another version which does channel specific uh speech-nonspeech detection.


and what i did is i used some normalized features which uh look into the which is normalized energy uh energy normalized by the mean over the channels and by the minimum over the other.


within each channel.


and to to huh to yeah to normalize also loudness and and modified loudness and things and that those special features actually are in my feature vector.


oh.


and and therefore to be able to uh somewhat distinguish between foreground and background speech in in the different in each channel.


and uh i tested it on on three or four meetings and it seems to work well yeah fairly well i i would say.


there are some problems with the lapel mike.


of course.


wow that's great.


uh yeah.


and


so i i understand that's what you were saying about your problem with minimum.


yeah and and i had i had uh specific problems with


i get it.


so new use ninetieth quartile rather than minimum.


yeah.


wow!


yeah yeah then i i did some some some things like that.


interesting.


as there there are some some problems in when in the channel there they the the speaker doesn't doesn't talk much or doesn't talk at all.


then the yeah there are there are some problems with with with with normalization and then uh there the system doesn't work at all.


so i'm i'm glad that there is the the digit part where everybody is forced to say something.


right.


so that's that's great for for my purpose.


and the thing is i i then the evaluation of of the system is a little bit hard as i don't have any references.


well we did the hand


the one by hand.


yeah that's the one one where i do the training on so i can't do the evaluation on.


so the thing is can the transcribers perhaps do some some some meetings in in terms of speech-nonspeech in in the specific channels?


uh.


well won't you have that from their transcriptions?


well i have


well okay so now we need


no because we need is really tight.


yeah.


so um