Identifying named entities in written text is an essential component of the text processing pipeline used in applications such as text editors to gain a better understanding of the semantics of the text .

However , the typical experimental setup for evaluating Named Entity Recognition ( NER ) systems is not directly applicable to systems that process text in real time as the text is being typed .

Evaluation is performed on a sentence level assuming the end-user is willing to wait until the entire sentence is typed for entities to be identified and further linked to identifiers or coreferenced .

We introduce a novel experimental setup for NER systems for applications where decisions about named entity boundaries need to be performed in an online fashion .

We study how state-of-the-art

perform under this setup in multiple languages and propose adaptations to these models to suit this new

Experimental results show that the best systems that are evaluated on each token after its typed , reach performance within 1 - 5 F 1 points of systems that are evaluated at the end of the sentence .

These show that entity recognition can be performed in this setup and open up the development of other NLP tools in a similar setup .

Automatically identifying named entities such as organizations , people and locations is a key component in processing written text as it aids with understanding the semantics of the text .

Named entity recognition is used as a pre-processing step to subsequent tasks such as linking named entities to concepts in a knowledge graph , identifying the salience of an entity to the text , identifying coreferential mentions , computing sentiment towards an entity , in question answering or for extracting relations .

Figure 1 : An example of the proposed task and setup .

After the word ' Foreign ' is typed , the model immediately predicts an NER label for this word , only using left context ( ' A spokesman for ' ) and the word itself .

The prediction is then compared against the gold label to compute token-level F 1 score .

This token 's prediction will not be changed , even if the model 's internal prediction for it can be revised later as more tokens are typed .

Identifying named entities as they are typed benefits any system that processes text on the fly .

Examples of such applications include : a ) News editorswhere named entities can be highlighted , suggested ( auto-completion ) , co-referenced or linked as the editor is typing ; b ) auto-correct -where named entities that are just typed are less likely to need correction as they may come from a different language or be out-of-vocabulary ( OOV ) ; c ) simultaneous machine translation -where translation of OOV named entities requires different approaches ; d ) live speech-to-text ( e.g. , TV shows ) -where named entities are more likely to be OOV , hence the transcription should focus more on the phonetic transcription rather than on n-gram language modelling .

This paper introduces a novel of Named Entity Recognition systems illustrated in Figure 1 .

In this setup , inference about the span and type of named entities is performed for each token , immediately after it was typed .

The sentence level tag sequence is composed through appending all individual token predictions as they were made .

The current named entity recognition systems that are trained and evaluated to predict full sentences are likely to under-perform in this as they : expect that right context is available , are faced with unseen types of inputs in the form of truncated sentences and can not reconcile the final sentence-level tag sequence across the entire sentence as the result may not be a valid sequence .

The goal of this study is to present a comprehensive analysis of the task of NER in the as-you-type scenario , with the following contributions : a ) A novel for conducting named entity recognition experiments , denoted as the as-you-type scenario ; b ) with state-of-the-art sentence-level approaches to named entity recognition in the as-you-type setup across three languages , which indicate a 1 - 5 F 1 points decrease compared to sentence-level inference ; c ) Tailored methods for as-you-type entity recognition models which reduce the gap to entire sentence-level inference by 9 - 23 % compared to regular approaches ; d ) An extensive analysis of existing data sets in the context of this task and model error analysis , which highlight future modelling opportunities .

This paper introduced and motivated the as-youtype for the popular NER task .

We presented results across three different languages , which show the extent to which sentencelevel state-of-the-art models degrade in this setup .

Through insights gained from data analysis , we proposed modelling improvements to further reduce the gap to the regular sentence-level performance .

Our error analysis highlights the cases that pose challenges to the as-you-type scenario and uncovers insights into way to further improve the modelling of this task .

This setup is tailored for end-applications such as text editors , speech-to-text , machine translation , auto-completion , or auto-correct .

For text editors , the editor would be able to receive suggestions for entities inline , right after they type the entity , which can further be coupled with a linking algorithm .

This would increase the user experience and efficiency of the editor , as they can make selections about entities inline ( similar to a phone 's autocorrect ) , rather than having to go back over the entire sentence after it was completed .

Another avenue of future work would be to couple the NER as-you-type with ASR data and using methods that adapt NER to noisy ASR input ( Benton and Dredze , 2015 ) for building an end-to-end live speech to entities system .

The current experimental setup and evaluation for named entity recognition systems run on complete sentences limiting the real-time applications that process texts as being typed.

They propose an experimental setup where systems are evaluated in an online fashion and perform analysis of state-of-the-art methods in multiple languages.

They find that models degrade performance by 1 to 5 F1 points in their setup and the proposed tailored methods can reduce the gaps.


Almost all current dependency parsers classify based on millions of sparse indicator features .

Not only do these features generalize poorly , but the cost of feature computation restricts parsing speed significantly .

In this work , we propose a novel way of learning a neural network classifier for use in a greedy , transition-based dependency parser .

Because this classifier learns and uses just a small number of dense features , it can work very fast , while achieving an about 2 % improvement in unlabeled and labeled attachment scores on both English and Chinese datasets .

Concretely , our parser is able to parse more than 1000 sentences per second at 92.2 % unlabeled attachment score on the English Penn Treebank .

In recent years , enormous parsing success has been achieved by the use of feature-based discriminative dependency parsers ( Kübler et al . , 2009 ) .

In particular , for practical applications , the speed of the subclass of transition-based dependency parsers has been very appealing .

However , these parsers are not perfect .

First , from a statistical perspective , these parsers suffer from the use of millions of mainly poorly estimated feature weights .

While in aggregate both lexicalized features and higher-order interaction term features are very important in improving the performance of these systems , nevertheless , there is insufficient data to correctly weight most such features .

For this reason , techniques for introducing higher-support features such as word class features have also been very successful in improving parsing performance ( Koo et al . , 2008 ) .

Second , almost all existing parsers rely on a manually designed set of feature templates , which require a lot of expertise and are usually incomplete .

Third , the use of many feature templates cause a less studied problem : in modern dependency parsers , most of the runtime is consumed not by the core parsing algorithm but in the feature extraction step ( He et al . , 2013 ) .

For instance , Bohnet ( 2010 ) reports that his baseline parser spends 99 % of its time doing feature extraction , despite that being done in standard efficient ways .

In this work , we address all of these problems by using dense features in place of the sparse indicator features .

This is inspired by the recent success of distributed word representations in many NLP tasks , e.g. , POS tagging ( Collobert et al . , 2011 ) , machine translation ( Devlin et al . , 2014 ) , and constituency parsing ( Socher et al . , 2013 ) .

Low-dimensional , dense word embeddings can effectively alleviate sparsity by sharing statistical strength between similar words , and can provide us a good starting point to construct features of words and their interactions .

Nevertheless , there remain challenging problems of how to encode all the available information from the configuration and how to model higher-order features based on the dense representations .

In this paper , we train a neural network classifier to make parsing decisions within a transition-based dependency parser .

The neural network learns compact dense vector representations of words , part-of-speech ( POS ) tags , and dependency labels .

This results in a fast , compact classifier , which uses only 200 learned dense features while yielding good gains in parsing accuracy and speed on two languages ( English and Chinese ) and two different dependency representations ( CoNLL and Stanford dependencies ) .

The main contributions of this work are : ( i ) showing the usefulness of dense representations that are learned within the parsing task , ( ii ) developing a neural network architecture that gives good accuracy and speed , and ( iii ) introducing a novel acti-vation function for the neural network that better captures higher-order interaction features .

We have presented a novel dependency parser using neural networks .

Experimental evaluations show that our parser outperforms other greedy parsers using sparse indicator features in both accuracy and speed .

This is achieved by representing all words , POS tags and arc labels as dense vectors , and modeling their interactions through a novel cube activation function .

Our model only relies on dense features , and is able to automatically learn the most useful feature conjunctions for making predictions .

An interesting line of future work is to combine our neural network based classifier with searchbased models to further improve accuracy .

Also , there is still room for improvement in our architecture , such as better capturing word conjunctions , or adding richer features ( e.g. , distance , valency ) .

Existing dependency parsers are based on millions of parse indicator features which have poor generalization ability and restrictions on parsing speed.

They propose to represent words, part-of-speech tags, and dependency labels and train a neural network classifier with an activation function which performs transition-based dependency parsing.

The proposed parser can parse more than 1000 sentences per second on the English Penn Treebank and also achieves both speed and accuracy in Chinese.


Scientific papers revolve around citations , and for many discourse level tasks one needs to know whose work is being talked about at any point in the discourse .

In this paper , we introduce the scientific attribution task , which links different linguistic expressions to citations .

We discuss the suitability of different

metrics and evaluate our classification approach to deciding attribution both intrinsically and in an extrinsic where information about scientific attribution is shown to improve performance on Argumentative Zoning , a rhetorical classification task .

In the recent past , there has been a focus on information management from scientific literature .

In the genetics domain , for instance , information extraction of genes and gene-protein interactions helps geneticists scan large amounts of information ( e.g. , as explored in the TREC Genomics track ( Hersh et al . , 2004 ) ) .

Elsewhere , citation indexes ( Garfield , 1979 ) provide bibliometric data about the frequency with which particular papers are cited .

The success of citation indexers such as CiteSeer ( Giles et al . , 1998 ) and Google Scholar relies on the robust detection of formal citations in arbitrary text .

In bibliographic information retrieval , anchor text , i.e. , the context of a citation can be used to characterise ( index ) the cited paper using terms outside of that paper ( Bradshaw , 2003 ) ; O'Connor ( 1982 ) presents an approach for identifying the area around citations where the text focuses on that citation .

And automatic citation classification ( Nanba and Okumura , 1999 ; Teufel et al . , 2006 ) determines the function that a citation plays in the discourse .

For such information access and retrieval purposes , the relevance of a citation within a paper is often crucial .

One can estimate how important a citation is by simply counting how often it occurs in the paper .

But as Kim and Webber ( 2006 ) argue , this ignores many expressions in text which refer to the cited author 's work but which are not as easy to recognise as citations .

They address the resolution of instances of the third person personal pronoun " they " in astronomy papers : it can either refer to a citation or to some entities that are part of research within the paper ( e.g. , planets or galaxies ) .

Several applications should profit in principle from detecting connections between referring expressions and citations .

For instance , in citation function classification , the task is to find out if a citation is described as flawed or as useful .

Consider : Most computational models of discourse are based primarily on an analysis of the intentions of the speakers [ Cohen and Perrault , 1979 ] [ Allen and Perrault , 1980 ] [ Grosz and Sidner , 1986 ] WEAK .

The speaker will form intentions based on his goals and then act on these intentions , producing utterances .

The hearer will then reconstruct a model of the speaker 's intentions upon hearing the utterance .

This approach has many strong points , but does not provide a very satisfactory account of the adherence to discourse conventions in dialogue .

The three citations above are described as flawed ( detectable by " does not provide a very satisfactory account " ) , and thus receive the label Weak .

However , in order to detect this , one must first realise that " this approach " refers to the three cited papers .

A contrasting hypothesis could be that the citations are used ( thus deserving the label Use ; the cue phrase " based on " might make us think so ( as in the context " our work is based on " ) .

This , however , can be ruled out if we know that " the speaker " is not referring to some aspect of the current paper .

We have described a new reference task -deciding scientific attribution , and demonstrated high human agreement ( α > 0.8 ) on this task .

Our machine learning solution using shallow features achieves an agreement of α M = 0.68 with the human gold standard , increasing to α M = 0.71 if only pronouns need to be resolved .

We have also demonstrated that information about scientific attribution improves results for a discourse classification task ( Argumentative Zoning ) .

We believe that similar improvements can be achieved on other discourse annotation tasks in the scientific literature domain .

In particular , we plan to investigate the use of scientific attribution information for the citation function classification task .

Relevance of a citation in a paper to the cited paper is crucial information for retrieval purposes but computing occurrences would result in low recalls.

They propose the scientific attribution task which links different linguistic expressions to citations and explore both intrinsic and extrinsic evaluation setups.

They show that high human agreement on the proposed task and information about scientific attributions can improve the performance of discourse classification.


Understanding the connotation of words plays an important role in interpreting subtle shades of sentiment beyond denotative or surface meaning of text , as seemingly objective statements often allude nuanced sentiment of the writer , and even purposefully conjure emotion from the readers ' minds .

The focus of this paper is drawing nuanced , connotative sentiments from even those words that are objective on the surface , such as " intelligence " , " human " , and " cheesecake " .

We propose induction algorithms encoding a diverse set of linguistic insights ( semantic prosody , distributional similarity , semantic parallelism of coordination ) and prior knowledge drawn from lexical resources , resulting in the first broad-coverage connotation lexicon .

There has been a substantial body of research in sentiment analysis over the last decade ( Pang and Lee , 2008 ) , where a considerable amount of work has focused on recognizing sentiment that is generally explicit and pronounced rather than implied and subdued .

However in many real-world texts , even seemingly objective statements can be opinion-laden in that they often allude nuanced sentiment of the writer ( Greene and Resnik , 2009 ) , or purposefully conjure emotion from the readers ' minds ( Mohammad and Turney , 2010 ) .

Although some researchers have explored formal and statistical treatments of those implicit and implied sentiments ( e.g.

Wiebe et al .

( 2005 ) , Esuli and Sebastiani ( 2006 ) , Greene and Resnik ( 2009 ) , Davidov et al .

( 2010 ) ) , automatic analysis of them largely remains as a big challenge .

In this paper , we concentrate on understanding the connotative sentiments of words , as they play an important role in interpreting subtle shades of sentiment beyond denotative or surface meaning of text .

For instance , consider the following : Geothermal replaces oil-heating ; it helps reducing greenhouse emissions.1 Although this sentence could be considered as a factual statement from the general standpoint , the subtle effect of this sentence may not be entirely objective : this sentence is likely to have an influence on readers ' minds in regard to their opinion toward " geothermal " .

In order to sense the subtle overtone of sentiments , one needs to know that the word " emissions " has generally negative connotation , which geothermal reduces .

In fact , depending on the pragmatic contexts , it could be precisely the intention of the author to transfer his opinion into the readers ' minds .

The main contribution of this paper is a broadcoverage connotation lexicon that determines the connotative polarity of even those words with ever so subtle connotation beneath their surface meaning , such as " Literature " , " Mediterranean " , and " wine " .

Although there has been a number of previous work that constructed sentiment lexicons ( e.g. , Esuli and Sebastiani ( 2006 ) , Wilson et al .

( 2005a ) , Kaji and Kitsuregawa ( 2007 ) , Qiu et al .

( 2009 ) ) , which seem to be increasingly and inevitably expanding over words with ( strongly ) connotative sentiments rather than explicit sentiments alone ( e.g. , " gun " ) , little

has directly tackled this problem of learning connotation , 2 and much of the subtle connotation of many seemingly objective words is yet to be determined .

POSITIVE NEGATIVE FEMA , Mandela , Intel , Google , Python , Sony , Pulitzer , Harvard , Duke , Einstein , Shakespeare , Elizabeth , Clooney , Hoover , Goldman , Swarovski , Hawaii , Yellowstone Katrina , Monsanto , Halliburton , Enron , Teflon , Hiroshima , Holocaust , Afghanistan , Mugabe , Hutu , Saddam , Osama , Qaeda , Kosovo , Helicobacter , HIV A central premise to

our approach is that it is collocational statistics of words that affect and shape the polarity of connotation .

Indeed , the etymology of " connotation " is from the Latin " com- " ( " together or with " ) and " notare " ( " to mark " ) .

It is important to clarify , however , that we do not simply assume that words that collocate share the same polarity of connotation .

Although such an assumption played a key role in previous work for the analogous task of learning sentiment lexicon ( Velikovich et al . , 2010 ) , we expect that the same assumption would be less reliable in drawing subtle connotative sentiments of words .

As one example , the predicate " cure " , which has a positive connotation typically takes arguments with negative connotation , e.g. , " disease " , when used as the " relieve " sense .

3 Therefore , in order to attain a broad coverage lexicon while maintaining good precision , we guide the induction algorithm with multiple , carefully selected linguistic insights : [ 1 ] distributional similarity , [ 2 ] semantic parallelism of coordination , [ 3 ] selectional preference , and [ 4 ] semantic prosody ( e.g. , Sinclair ( 1991 ) , Louw ( 1993 ) , Stubbs ( 1995 ) , Stefanowitsch and Gries ( 2003 ) ) ) , and also exploit existing lexical resources as an additional inductive bias .

We cast the connotation lexicon induction task as a collective inference problem , and consider approaches based on three distinct types of algorithmic framework that have been shown successful for conventional sentiment lexicon induction : Random walk based on HITS / PageRank ( e.g. , Kleinberg ( 1999 ) Label / Graph propagation ( e.g. , Zhu and Ghahra-(2011 ) but with practical limitations .

See § 3 for detailed discussion .

3 Note that when " cure " is used as the " preserve " sense , it expects objects with non-negative connotation .

Hence wordsense-disambiguation ( WSD ) presents a challenge , though not unexpectedly .

In this work , we assume the general connotation of each word over statistically prevailing senses , leaving a more cautious handling of WSD as future work . mani ( 2002 ) , Velikovich et al .

( 2010 ) ) Constraint optimization ( e.g. , Roth and Yih ( 2004 ) , Choi and Cardie ( 2009 ) , Lu et al .

( 2011 ) ) .

We provide comparative empirical results over several variants of these approaches with comprehensive evaluations including lexicon-based , human judgments , and extrinsic evaluations .

It is worthwhile to note that not all words have connotative meanings that are distinct from denotational meanings , and in some cases , it can be difficult to determine whether the overall sentiment is drawn from denotational or connotative meanings exclusively , or both .

Therefore , we encompass any sentiment from either type of meanings into the lexicon , where non-neutral polarity prevails over neutral one if some meanings lead to neutral while others to non-neutral .

4Our work results in the first broad-coverage connotation lexicon,5 significantly improving both the coverage and the precision of Feng et al .

( 2011 ) .

As an interesting by-product , our algorithm can be also used as a proxy to measure the general connotation of real-world named entities based on their collocational statistics .

Table 1 highlights some example proper nouns included in the final lexicon .

The rest of the paper is structured as follows .

In § 2 we describe three types of induction algorithms followed by

in § 3 .

Then we revisit the induction algorithms based on constraint optimization in § 4 to enhance quality and scalability . § 5 presents comprehensive evaluation with human judges and extrinsic evaluations .

Related work and conclusion are in § 6 and § 7 .

We presented a broad-coverage connotation lexicon that determines the subtle nuanced sentiment of even those words that are objective on the surface , including the general connotation of realworld named entities .

Via a comprehensive , we provided empirical insights into three different types of induction algorithms , and proposed one with good precision , coverage , and efficiency .

Most works on automatic sentiment analysis focus on explicit expression while subtle implicit and implied sentiments have not been investigated.

They regard the connotation lexicon induction task as a collective inference problem and propose induction algorithms that encode a diverse set of linguistic insights.

They show their algorithms can significantly improve the coverage and the precision from the previous method and also measure named entities on the collocational statistics.


For resource-limited language pairs , coverage of the test set by the parallel corpus is an important factor that affects translation quality in two respects : 1 ) out of vocabulary words ; 2 ) the same information in an input sentence can be expressed in different ways , while current phrase-based SMT systems can not automatically select an alternative way to transfer the same information .

Therefore , given limited data , in order to facilitate translation from the input side , this paper proposes a novel

to reduce the translation difficulty using source-side lattice-based paraphrases .

We utilise the original phrases from the input sentence and the corresponding paraphrases to build a lattice with estimated weights for each edge to improve translation quality .

Compared to the baseline system , our method achieves relative improvements of 7.07 % , 6.78 % and 3.63 % in terms of BLEU score on small , medium and largescale English-to-Chinese translation tasks respectively .

The results show that the proposed method is effective not only for resourcelimited language pairs , but also for resourcesufficient pairs to some extent .

In recent years , statistical MT systems have been easy to develop due to the rapid explosion in data availability , especially parallel data .

However , in reality there are still many language pairs which lack parallel data , such as Urdu-English , Chinese-Italian , where large amounts of speakers exist for both languages ; of course , the problem is far worse for pairs such as Catalan-Irish .

For such resourcelimited language pairs , sparse amounts of parallel data would cause the word alignment to be inaccurate , which would in turn lead to an inaccurate phrase alignment , and bad translations would result .

Callison-Burch et al .

( 2006 ) argue that limited amounts of parallel training data can lead to the problem of low coverage in that many phrases encountered at run-time are not observed in the training data and so their translations will not be learned .

Thus , in recent years , research on addressing the problem of unknown words or phrases has become more and more evident for resource-limited language pairs .

Callison-Burch et al .

( 2006 ) proposed a novel method which substitutes a paraphrase for an unknown source word or phrase in the input sentence , and then proceeds to use the translation of that paraphrase in the production of the target-language result .

Their

showed that by translating paraphrases a marked improvement was achieved in coverage and translation quality , especially in the case of unknown words which previously had been left untranslated .

However , on a large-scale data set , they did not achieve improvements in terms of automatic .

Nakov ( 2008 ) proposed another way to use paraphrases in SMT .

He generates nearly-equivalent syntactic paraphrases of the source-side training sentences , then pairs each paraphrased sentence with the target translation associated with the original sentence in the training data .

Essentially , this method generates new training data using paraphrases to train a new model and obtain more useful phrase pairs .

However , he reported that this method results in bad system performance .

By contrast , real improvements can be achieved by merging the phrase tables of the paraphrase model and the original model , giving priority to the latter .

Schroeder et al .

( 2009 ) presented the use of word lattices for multi-source translation , in which the multiple source input texts are compiled into a compact lattice , over which a single decoding pass is then performed .

This lattice-based method achieved positive results across all data conditions .

In this paper , we propose a novel method using paraphrases to facilitate translation , especially for resource-limited languages .

Our method does not distinguish unknown words in the input sentence , but uses paraphrases of all possible words and phrases in the source input sentence to build a source-side lattice to provide a diverse and flexible list of source-side candidates to the SMT decoder so that it can search for a best path and deliver the translation with the highest probability .

In this case , we neither need to change the phrase table , nor add new features in the log-linear model , nor add new sentences in the training data .

The remainder of this paper is organised as follows .

In Section 2 , we define the " translation difficulty " from the perspective of the source side , and then examine how well the test set is covered by the phrase table and the parallel training data .

Section 3 describes our paraphrase lattice method and discusses how to set the weights for the edges in the lattice network .

In Section 4 , we report comparative conducted on small , medium and largescale English-to-Chinese data sets .

In Section 5 , we analyse the influence of our paraphrase lattice method .

Section 6 concludes and gives avenues for future work .

2 What Makes Translation Difficult ?

In this paper , we proposed a novel method using paraphrase lattices to facilitate the translation process in SMT .

Given an input sentence , our method firstly discovers all possible paraphrases from a paraphrase database for N -grams ( 1 < = N < = 10 ) in the test set , and then filters out the paraphrases which do not appear in the phrase table in order to avoid adding new unknown words on the input side .

We then use the original words and the paraphrases to build a word lattice , and set the weights to prioritise the original edges and penalise the paraphrase edges .

Finally , we import the lattice into the decoder to perform lattice decoding .

The experiments are conducted on English-to-Chinese translation using the FBIS data set with small and medium-sized amounts of data , and on a large-scale corpus of 2.1 million sentence pairs .

We also performed comparative for the baseline , the " Para-Sub " system and our paraphrase lattice-based system .

The experimental results show that our proposed system significantly outperforms the baseline and the " Para-Sub " system , and the effectiveness is consistent on the small , medium and large-scale data sets .

As for future work , firstly we plan to propose a pruning algorithm for the duplicate paths in the lattice , which will track the edge generation with respect to the path span , and thus eliminate duplicate paths .

Secondly , we plan to with another feature function in the log-linear model to discount words derived from paraphrases , and use MERT to assign an appropriate weight to this feature function .

Vocabulary coverage is crucial for SMT systems, especially for resource-lean language pairs, however; current systems cannot find alternative solutions.

They propose to use paraphrasing the source side of inputs at test time listing all the possible alternatives to expand the lattice.

They show that their system outperforms the baseline system in the English-Chinese translation task for both resource-lean and sufficient setups.


Traditional approaches to Information Extraction ( IE ) from speech input simply consist in applying text based methods to the output of an Automatic Speech Recognition ( ASR ) system .

If it gives satisfaction with low Word Error Rate ( WER ) transcripts , we believe that a tighter integration of the IE and ASR modules can increase the IE performance in more difficult conditions .

More specifically this paper focuses on the robust extraction of Named Entities from speech input where a temporal mismatch between training and test corpora occurs .

We describe a Named Entity Recognition ( NER ) system , developed within the French Rich Broadcast News Transcription program ESTER , which is specifically optimized to process ASR transcripts and can be integrated into the search process of the ASR modules .

Finally we show how some metadata information can be collected in order to adapt NER and ASR models to new conditions and how they can be used in a task of Named Entity indexation of spoken archives .

Named Entity Recognition ( NER ) is a crucial step in many Information Extraction ( IE ) tasks .

It has been a specific task in several

pro-grams such as the Message Understanding Conferences ( MUC ) , the Conferences on Natural Language Learning ( CoNLL ) , the DARPA HUB-5 program or more recently the French ESTER Rich Transcription program on Broadcast News data .

Most of these conferences have studied the impact of using transcripts generated by an Automatic Speech Recognition ( ASR ) system rather than written texts .

It appears from these studies that unlike other IE tasks , NER performance is greatly affected by the Word Error Rate ( WER ) of the transcripts processed .

To tackle this problem , different ideas have been proposed : modeling explicitly the ASR errors ( Palmer and Ostendorf , 2001 ) or using the ASR system alternate hypotheses found in word lattices ( Saraclar and Sproat , 2004 ) .

However performance in NER decreases dramatically when processing high WER transcripts like the ones that are obtained with unmatched conditions between the ASR training model and the data to process .

This paper investigates this phenomenon in the framework of the NER task of the French Rich Transcription program of Broadcast News ESTER ( Gravier et al . , 2004 ) .

Several issues are addressed : • how to jointly optimize the ASR and the NER models ? • what is the impact in term of ASR and NER performance of a temporal mismatch between the corpora used to train and test the models and how can it be recovered by means of metadata information ? • Can metadata information be used for indexing large spoken archives ?

After a quick overview of related works in IE from speech input , we present the ESTER program ; then we introduce a NER system tightly integrated to the ASR process and show how it can successfully index high WER spoken databases thanks to metadata .

We have presented in this paper a robust Named Entity Recognition system dedicated to process ASR transcripts .

The FSM-based approach allows us to control the generalization capabilities of the system while the statistical tagger provides good labeling decisions .

The main feature of this system is its ability to extract n-best lists of NE hypothesis from word lattices leaving the decision strategy choosing to either emphasize the recall or the precision of the extraction , according to the task targeted .

A comparison between this approach and a standard approach based on the NLP tools Lingpipe validates our hypotheses .

This integration of the ASR and the NER processes is particularly important in difficult conditions like those that can be found in large spoken archives where the training corpus does not match all the documents to process .

A study of the use of metadata information in order to adapt the ASR and NER models to a specific situation showed that if the overall improvement is small , some salient information related to the metadata added can be better extracted by means of this adaptation .

Existing named entity recognition models for speech input process the output from automatic speech recognition disjointly making the process sensitive to the word error rate.

They propose a system which is optimized to process transcripts and can be integrated into the search process of the automatic speech recognition modules.

The proposed method allows the control over generalization capabilities and comparison with a standard approach shows its effectiveness.


This paper presents an study of the use of interlocking phrases in phrase-based statistical machine translation .

We examine the effect on translation quality when the translation units used in the translation hypotheses are allowed to overlap on the source side , on the target side and on both sides .

A large-scale evaluation on 380 language pairs was conducted .

Our results show that overall the use of overlapping phrases improved translation quality by 0.3 BLEU points on average .

Further analysis revealed that language pairs requiring a larger amount of re-ordering benefited the most from our approach .

When the evaluation was restricted to such pairs , the average improvement increased to up to 0.75 BLEU points with over 97 % of the pairs improving .

Our approach requires only a simple modification to the decoding algorithm and we believe it should be generally applicable to improve the performance of phrase-based decoders .

In this paper we examine the effect on machine translation quality of using interlocking phrases to during the decoding process in phrase-based statistical machine translation ( PBSMT ) .

The motivation for this is two-fold .

Firstly , during the phrase-pair extraction process that occurs in the training of a typical PBSMT system , all possible alternative phrase-pairs are extracted that are consistent with a set of alignment points .

As a consequence , the source and target sides of these extracted phrase pairs may over-lap .

However , in contrast to this , the decoding process traditionally proceeds by concatenating disjoint translation units ; the process relies on the language model to eliminate awkward hypotheses with repeated words produced by sequences of translation units that overlap .

Secondly , the transduction process in PBSMT is carried out by generating hypotheses that are composed of sequences of translation units .

These sequences are normally generated independently , as modeling the dependencies between them is difficult due to the data sparseness issues arising from modeling with word sequences .

The process of interlocking is a way of introducing a form of dependency between translation units , effectively producing larger units from pairs of compatible units .

( Karimova et al . , 2014 ) presented a method to extract overlapping phrases offline for hierarchical phrase based SMT .

They used the CDEC SMT decoder ( Dyer et al . , 2010 ) that offers several learners for discriminative tuning of weights for the new phrases .

Their results showed improvements of 0.3 to 0.6 BLEU points over discriminatively trained hierarchical phrase-based SMT systems on two datasets for German-to-English translation .

( Tribble and et al . , 2003 ) proposed a method to generate longer new phrases by merging existing phraselevel alignments that have overlaping words on both source and target sides .

Their experiments on translating Arabic-English text from the news domain were encouraging .

( Roth and McCallum , 2010 ) proposed a conditional-random-field approach to discriminatively train phrase based machine translation in which training and decoding are both cast in a sampling framework .

Different with traditional PB-SMT decoding that infers both a Viterbi alignment and the target sentence , their approach produced a rich overlapping phrase alignment .

Their approach leveraged arbitrary features of the entire source sentence , target sentence and alignment .

( Kääriäinen , 2009 ) proposed a novel phrase-based conditional exponential family translation model for SMT .

The model operates on a feature representation in which sentence level translations are represented by enumerating all the known phrase level translations that occur inside them .

The model automatically takes into account information provided by phrase overlaps .

Although both of the latter two approaches were innovative the translation performance was lower than tranditional PBSMT baselines .

Our proposed approach is most similar to that of ( Tribble and et al . , 2003 ) .

Our approach differs in the interlocking process is less constrained ; phrase pairs can interlock independently on source and target sides , and the interlocking process performed during the decoding process itself , rather than by augmenting the phrase-table .

In this paper we propose and evaluate a simple technique for improving the performance of phrasebased statistical machine translation decoders , that can be implemented with only minor modifications to the decoder .

In the proposed

phrases are allowed to interlock freely on both the source and target side during decoding .

The experimental results , based on a large-scale study involving 380 language pairs provide strong evidence that our approach is genuinely effective in improving the machine translation quality .

The translation quality improved for 77 % of the language pairs tested , and this was increased to over 97 % when the set of language pairs was filtered according to Kendall 's tau distance .

The translation quality improved by an average of up to 0.75 BLEU points on this subset .

This value represents a lower bound on what is possible with this technique and in future work we intend to study the introduction of additional features into the log-linear model to encourage or discourage the use of interlocking phrases during decoding , and investigate the effect of increasing the number of interlocked words .

Existing phrase-based statistical machine translation systems can generate the same words repeatedly because there can be overlaps in extracted phrases.

They first analyze the effect of overlapping phrases and propose a simple modification to the decoder to handle the interlocking phrases.

A large-scale evaluation including 380 language pairs reveals that their modification effectively improves the translation quality.


Existing approaches for table annotation with entities and types either capture the structure of table using graphical models , or learn embeddings of table entries without accounting for the complete syntactic structure .

We propose TabGCN , which uses Graph Convolutional Networks to capture the complete structure of tables , knowledge graph and the training annotations , and jointly learns embeddings for table elements as well as the entities and types .

To account for knowledge incompleteness , TabGCN 's embeddings can be used to discover new entities and types .

Using

on 5 benchmark datasets , we show that TabGCN significantly outperforms multiple state-of-the-art baselines for table annotation , while showing promising performance on downstream table-related applications .

Table data abounds in webpages and organizational documents .

Annotation of table entries , such as columns , cells and rows , using available background knowledge ( e.g.

Yago , DBPedia , Freebase , etc . ) , such as knowledge of entities and their types , helps in better understanding and semantic interpretation of such tabular data .

The challenge , however , is that such web tables do not adhere to any standard format , schema or convention ( Limaye et al . , 2010 ) .

Additionally , knowledge graphs are typically incomplete -entities and types mentioned in tables may not always exist in the knowledge graph .

Therefore , it becomes necessary to expand the knowledge graph with new entities ( Zhang et al . , 2020 ) and types for annotating tables .

Initial research on table annotation ( Limaye et al . , 2010 ; Takeoka et al . , 2019 ; Bhagavatula et al . , 2015 ) used probabilistic graphical models to capture the complete row-column structure of tables and also the knowledge graph for collective annotation .

More recent approaches using embeddings ( Gentile et al . , 2017 ; Zhang and Balog , 2018 ; Zhang et al . , 2019 ; Chen et al . , 2019 ; Yin et al . , 2020 ) only partly capture the syntactic structure of tables , and also ignore the structure of the knowledge graph .

The problem of incompleteness of the knowledge representation ( Zhang et al . , 2020 ) is mostly not addressed .

In this work , we propose the TabGCN model that uses a Graph Convolutional Network ( GCN ) ( Kipf and Welling , 2017 ) to unify the complete syntactic structure of tables ( rows , columns and cells ) and that of the knowledge graph ( entities and types ) via available annotations .

The embeddings of the table elements as well as knowledge graph entities and types are trained jointly and end-to-end .

While GCNs have been used for learning embeddings for many NLP tasks using the syntactic and semantic structure of natural language sentences ( Marcheggiani and Titov , 2017 ; Vashishth et al . , 2019 ) , encoding tabular structure using GCNs has not been addressed before .

The model and embeddings thus trained are used to annotate new tables with known entities and types , while discovering hitherto unseen entities and types .

Additionally , we use the trained embeddings for tables and rows for downstream table-related tasks -identifying similar tables , and identifying the appropriate table for any row .

We demonstrate these capabilities of TabGCN using on 5 benchmark web table datasets comparing against 5 existing models .

We show that WebGCN significantly improves performance for entity and type annotation .

For the other tasks , we show that the same embeddings show impressive performance .

No existing model can perform all of these tasks .

Our contributions are as follows : ( a ) We propose a model called TabGCN based on the GCN architecture that captures the complete syntactic structure of tables as well as the knowledge representation , and learns embeddings of tables , rows , columns and cells , as well as entities and types jointly and in an end-to-end fashion .

( b ) TabGCN addresses incompleteness in the knowledge representation by discovering new entities and types .

( c ) TabGCN significantly outperforms 5 existing approaches in 5 different benchmark datasets for the task of table annotation .

( d ) The trained embeddings show impressive performance in downstream tasks such as identifying similar tables and assignment of rows to appropriate tables .

We have proposed a model for that jointly learns representations of tables , rows , columns and cell , as well as entities and types by capturing the complete syntactic structure of all tables , the relevant entities and types and the available annotations using the Graph Convolutional Network .

As a result , TabGCN unifies the benefits of probabilistic graphical model based approaches and embedding based approaches for table annotation .

Using these embeddings , TabGCN significantly outperforms existing approaches for table annotation , as well as entity and type discovery .

Existing embedding-based approaches for table annotation only partially capture the syntactic structure of tables while ignoring the structure of the knowledge graph.

They propose a graph convolutional network that jointly learns embeddings for table structures, knowledge graphs, and annotations.

The proposed model outperforms 5 state-of-the-art baselines for table annotation and other table-related downstream applications on 5 benchmark datasets


This paper describes a method using morphological rules and heuristics , for the automatic extraction of large-coverage lexicons of stems and root word-forms from a raw text corpus .

We cast the problem of high-coverage lexicon extraction as one of stemming followed by root word-form selection .

We examine the use of POS tagging to improve precision and recall of stemming and thereby the coverage of the lexicon .

We present accuracy , precision and recall scores for the system on a Hindi corpus .

Large-coverage morphological lexicons are an essential component of morphological analysers .

Morphological analysers find application in language processing systems for tasks like tagging , parsing and machine translation .

While raw text is an abundant and easily accessible linguistic resource , high-coverage morphological lexicons are scarce or unavailable in Hindi as in many other languages ( Clément et al . , 2004 ) .

Thus , the development of better algorithms for the extraction of morphological lexicons from raw text corpora is a task of considerable importance .

A root word-form lexicon is an intermediate stage in the creation of a morphological lexicon .

In this paper , we consider the problem of extracting a large-coverage root word-form lexicon for the Hindi language , a highly inflectional and moderately agglutinative Indo-European language spoken widely in South Asia .

Since a POS tagger , another basic tool , was available along with POS tagged data to train it , and since the error patterns indicated that POS tagging could greatly improve the accuracy of the lexicon , we used the POS tagger in our experiments on lexicon extraction .

Previous work in morphological lexicon extraction from a raw corpus often does not achieve very high precision and recall ( de Lima , 1998 ; Oliver and Tadić , 2004 ) .

In some previous work the process of lexicon extraction involves incremental or post-construction manual validation of the entire lexicon ( Clément et al . , 2004 ; Sagot , 2005 ; Forsberg et al . , 2006 ; Sagot et al . , 2006 ; Sagot , 2007 ) .

Our method attempts to improve on and extend the previous work by increasing the precision and recall of the system to such a point that manual validation might even be rendered unnecessary .

Yet another difference , to our knowledge , is that in our method we cast the problem of lexicon extraction as two subproblems : that of stemming and following it , that of root word-form selection .

The input resources for our system are as follows : a ) raw text corpus , b ) morphological rules , c ) POS tagger and d ) word-segmentation labelled data .

We output a stem lexicon and a root wordform lexicon .

We take as input a raw text corpus and a set of morphological rules .

We first run a stemming algorithm that uses the morphological rules and some heuristics to obtain a stem dictionary .

We then create a root dictionary from the stem dictionary .

The last two input resources are optional but when a POS tagger is utilized , the F-score ( harmonic mean of precision and recall ) of the root lexicon can be as high as 94.6 % .

In the rest of the paper , we provide a brief overview of the morphological features of the Hindi language , followed by a description of our method including the specification of rules , the corpora and the heuristics for stemming and root word-form selection .

We then evaluate the system with and without the POS tagger .

We have described a system for automatically constructing a root word-form lexicon from a raw text corpus .

The system is rule-based and utilizes a POS tagger .

Though preliminary , our results demonstrate that it is possible , using this method , to extract a high-precision and high-recall root word-form lexicon .

Specifically , we show that with a POS tagger capable of labelling wordforms with POS categories at an accuracy of about 88 % , we can extract root word-forms with an accuracy of about 87 % and a precision and recall of 94.1 % and 95.3 % respectively .

Though the system has been evaluated on Hindi , the techniques described herein can probably be applied to other inflectional languages .

The rules selected by the system and applied to the wordforms also contain information that can be used to determine the paradigm membership of each root word-form .

Further work could evaluate the accuracy with which we can accomplish this task .

High-coverage morphological lexicons are scarce or unavailable in Hindi calling algorithms that extract morphological lexicons from raw corpora with high recall and precision.

They propose to use morphological rules and heuristics to automatically extract lexicons of stems and root word forms from a raw corpus.

They show that high-precision and high-recall root word-form lexicon can be extracted from raw corpora in Hindi, and a POS tagger improves results.


Automatic evaluation of Machine Translation ( MT ) quality is essential to developing highquality MT systems .

Various evaluation metrics have been proposed , and BLEU is now used as the de facto standard metric .

However , when we consider translation between distant language pairs such as Japanese and English , most popular metrics ( e.g. , BLEU , NIST , PER , and TER ) do not work well .

It is well known that Japanese and English have completely different word orders , and special care must be paid to word order in translation .

Otherwise , translations with wrong word order often lead to misunderstanding and incomprehensibility .

For instance , SMT-based Japanese-to-English translators tend to translate ' A because B ' as ' B because A. ' Thus , word order is the most important problem for distant language translation .

However , conventional evaluation metrics do not significantly penalize such word order mistakes .

Therefore , locally optimizing these metrics leads to inadequate translations .

In this paper , we propose an automatic evaluation metric based on rank correlation coefficients modified with precision .

Our meta-evaluation of the NTCIR-7 PATMT JE task data shows that this metric outperforms conventional metrics .

SMT systems thus fail to find ( R0 ) .

Consequently , the global word order is essential for translation between distant language pairs , and wrong word order can easily lead to misunderstanding or incomprehensibility .

Perhaps , some readers do not understand why we emphasize word order from this example alone .

A few more examples will clarify what happens when SMT is applied to Japanese-to-English translation .

Even the most famous SMT service available on the web failed to translate the following very simple sentence at the time of writing this paper .

Japanese : meari wa jon wo koroshita .

Reference : Mary killed John .

SMT output : John killed Mary .

Since it can not translate such a simple sentence , it obviously can not translate more complex sentences correctly .

Japanese : bobu ga katta hon wo jon wa yonda .

Reference : John read a book that Bob bought .

SMT output : Bob read the book John bought .

Japanese : bobu wa meari ni yubiwa wo kau tameni , jon no mise ni itta .

Reference : Bob went to John 's store to buy a ring for Mary .

SMT output : Bob Mary to buy the ring , John went to the store .

Automatic evaluation of machine translation ( MT ) quality is essential to developing high-quality machine translation systems because human evaluation is time consuming , expensive , and irreproducible .

If we have a perfect automatic evaluation metric , we can tune our translation system for the metric .

BLEU ( Papineni et al . , 2002b ; Papineni et al . , 2002a ) showed high correlation with human judgments and is still used as the de facto standard automatic evaluation metric .

However , Callison-Burch et al .

( 2006 ) argued that the MT community is overly reliant on BLEU by showing examples of poor performance .

For Japanese-to-English ( JE ) translation , Echizen-ya et al .

( 2009 ) showed that the popular BLEU and NIST do not work well by using the system outputs of the NTCIR-7 PATMT ( patent translation ) JE task ( Fujii et al . , 2008 ) .

On the other hand , ROUGE-L ( Lin and Hovy , 2003 ) , Word Error Rate ( WER ) , and IMPACT ( Echizen-ya and Araki , 2007 ) worked better .

In these studies , Pearson 's correlation coefficient and Spearman 's rank correlation ρ with human evaluation scores are used to measure how closely an automatic evaluation method correlates with human evaluation .

This evaluation of automatic evaluation methods is called meta-evaluation .

In human evaluation , people judge the adequacy and the fluency of each translation .

Denoual and Lepage ( 2005 ) pointed out that BLEU assumes word boundaries , which is ambiguous in Japanese and Chinese .

Here , we assume the word boundaries given by ChaSen , one of the standard morphological analyzers ( http://chasenlegacy.sourceforge.jp/ ) following Fujii et al .

( 2008 ) In JE translation , most Statistical Machine Translation ( SMT ) systems translate the Japanese sentence ( J0 ) kare wa sono hon wo yonda node sekaishi ni kyoumi ga atta which means ( R0 ) he was interested in world history because he read the book into an English sentence such as ( H0 ) he read the book because he was interested in world history in which the cause and the effect are swapped .

Why does this happen ?

The former half of ( J0 ) means " He read the book , " and the latter half means " ( he ) was interested in world history . " The middle word " node " between them corresponds to " because . " Therefore , SMT systems output sentences like ( H0 ) .

On the other hand , Rule-based Machine Translation ( RBMT ) systems correctly give ( R0 ) .

In order to find ( R0 ) , SMT systems have to search a very large space because we can not restrict its search space with a small distortion limit .

Most In this way , this SMT service usually gives incomprehensible or misleading translations , and thus people prefer RBMT services .

Other SMT systems also tend to make similar word order mistakes , and special care should be paid to the translation between distant language pairs such as Japanese and English .

Even Japanese people can not solve this word order problem easily : It is well known that Japanese people are not good at speaking English .

From this point of view , conventional automatic evaluation metrics of translation quality disregard word order mistakes too much .

Single-reference BLEU is defined by a geometrical mean of n-gram precisions p n and is modified by Brevity Penalty ( BP ) min(1 , exp(1 -r / h ) ) , where r is the length of the reference and h is the length of the hypothesis .

BLEU = BP × ( p 1 p 2 p 3 p 4 ) 1/4 .

Its range is [ 0 , 1 ] .

The BLEU score of ( H0 ) with reference ( R0 ) is 1.0×(11/11×9/10×6/9×4/8 ) 1/4 = 0.740 .

Therefore , BLEU gives a very good score to this inadequate translation because it checks only ngrams and does not regard global word order .

Since ( R0 ) and ( H0 ) look similar in terms of fluency , adequacy is more important than fluency in the translation between distant language pairs .

Similarly , other popular scores such as NIST , PER , and TER ( Snover et al . , 2006 ) also give relatively good scores to this translation .

NIST also considers only local word orders ( n-grams ) .

PER ( Position-Independent Word Error Rate ) was designed to disregard word order completely .

TER ( Snover et al . , 2006 ) was designed to allow phrase movements without large penalties .

Therefore , these standard metrics are not optimal for evaluating translation between distant language pairs .

In this paper , we propose an alternative automatic evaluation metric appropriate for distant language pairs .

Our method is based on rank correlation coefficients .

We use them to compare the word ranks in the reference with those in the hypothesis .

There are two popular rank correlation coefficients : Spearman 's ρ and Kendall 's τ ( Kendall , 1975 ) .

In Isozaki et al .

( 2010 ) , we used Kendall 's τ to measure the effectiveness of our Head Finalization rule as a preprocessor for English-to-Japanese translation , but we measured the quality of translation by using conventional metrics .

It is not clear how well τ works as an automatic

metric of translation quality .

Moreover , Spearman 's ρ might work better than Kendall 's τ .

As we discuss later , τ considers only the direction of the rank change , whereas ρ considers the distance of the change .

The first objective of this paper is to examine which is the better metric for distant language pairs .

The second objective is to find improvements of these rank correlation-metrics .

Spearman 's ρ is based on Pearson 's correlation coefficients .

Suppose we have two lists of numbers x = [ 0.1 , 0.4 , 0.2 , 0.6 ] , y = [ 0.9 , 0.6 , 0.2 , 0.7 ] .

To obtain Pearson 's coefficients between x and y , we use the raw values in these lists .

If we substitute their ranks for their raw values , we get x = [ 1 , 3 , 2 , 4 ] and y = [ 4 , 2 , 1 , 3 ] .

Then , Spearman 's ρ between x and y is given by Pearson 's coefficients between x and y .

This ρ can be rewritten as follows when there is no tie : ρ = 1 -i d 2 i n+1 C 3 .

Here , d i indicates the difference in the ranks of the i-th element .

Rank distances are squared in this formula .

Because of this square , we expect that ρ decreases drastically when there is an element that significantly changes in rank .

But we are also afraid that ρ may be too severe for alternative good translations .

Since Pearson 's correlation metric assumes linearity , nonlinear monotonic functions can change its score .

On the other hand , Spearman 's ρ and Kendall 's τ uses ranks instead of raw scores , and simple application of monotonic functions can not change them ( use of other operations such as averaging sentence scores can change them ) .

When Statistical Machine Translation is applied to distant language pairs such as Japanese and English , word order becomes an important problem .

SMT systems often fail to find an appropriate translation because of a large search space .

Therefore , they often output misleading or incomprehensible sentences such as " A because B " vs. " B because A. " To penalize such inadequate translations , we presented an automatic evaluation method based on rank correlation .

There were two questions for this approach .

First , which correlation coefficient should we use : Spearman 's ρ or Kendall 's τ ?

Second , how should we solve the overestimation problem caused by the nature of one-to-one correspondence ?

We answered these questions through our experiments using the NTCIR-7 PATMT JE translation data .

For the first question , τ was slightly better than ρ , but ρ was improved by precision .

For the second question , it turned out that BLEU 's Brevity Penalty was counter-productive .

A precision-based penalty gave a better solution .

With this precisionbased penalty , both ρ and τ worked well and they outperformed conventional methods for NTCIR-7 data .

For similar language pairs , our method was comparable to conventional

Existing automatic evaluation metrics do not perform well on two distant language pairs where word order is important.

They propose to use rank correlation coefficients for the evaluation of distant language pairs which can penalize wrongly ordered outputs.

The proposed methods outperform existing methods for distant language pairs and are comparable to similar language pairs.


We address the problem of answering new questions in community forums , by selecting suitable answers to already asked questions .

We approach the task as an answer ranking problem , adopting a pairwise neural network architecture that selects which of two competing answers is better .

We focus on the utility of the three types of similarities occurring in the triangle formed by the original question , the related question , and an answer to the related comment , which we call relevance , relatedness , and appropriateness .

Our proposed neural network models the interactions among all input components using syntactic and semantic embeddings , lexical matching , and domain-specific features .

It achieves state-of-the-art results , showing that the three similarities are important and need to be modeled together .

Our experiments demonstrate that all feature types are relevant , but the most important ones are the lexical similarity features , the domain-specific features , and the syntactic and semantic embeddings .

In recent years , community Question Answering ( cQA ) forums , such as StackOverflow , Quora , Qatar Living , etc . , have gained a lot of popularity as a source of knowledge and information .

These forums typically organize their content in the form of multiple topic-oriented question-comment threads , where a question posed by a user is followed by a list of other users ' comments , which intend to answer the question .

Many of such on-line forums are not moderated , which often results in ( a ) noisy and ( b ) redundant content , as users tend to deviate from the question and start asking new questions or engage in conversations , fights , etc .

Web forums try to solve problem ( a ) in various ways , most often by allowing users to up / downvote answers according to their perceived usefulness , which makes it easier to retrieve useful answers in the future .

Unfortunately , this negatively penalizes recent comments , which might be the most relevant and updated ones .

This is due to the time it takes for a comment to accumulate votes .

Moreover , voting is prone to abuse by forum trolls ( Mihaylov et al . , 2015 ; Mihaylov and Nakov , 2016a ) .

Problem ( b ) is harder to solve , as it requires that users verify that their question has not been asked before , possibly in a slightly different way .

This search can be hard , especially for less experienced users as most sites only offer basic search , e.g. , a site search by Google .

Yet , solving problem ( b ) automatically is important both for site owners , as they want to prevent question duplication as much as possible , and for users , as finding an answer to their questions without posting means immediate satisfaction of their information needs .

In this paper , we address the general problem of finding good answers to a given new question ( referred to as original question ) in one such community-created forum .

More specifically , we use a pairwise deep neural network to rank comments retrieved from different question-comment threads according to their relevance as answers to the original question being asked .

A key feature of our approach is that we investigate the contribution of the edges in the triangle formed by the pairwise interactions between the original question , the related question , and the related comments to rank comments in a unified fashion .

Additionally , we use three different sets of features that capture such similarity : lexical , distributed ( semantics / syntax ) , and domain-specific knowledge .

The experimental results show that addressing the answer ranking task directly , i.e. , modelling only the similarity between the original question and the answer-candidate comments , yields very low results .

The other two edges of the triangle are needed to obtain good results , i.e. , the similarity between the original question and the related question and the similarity between the related question and the related comments .

Both aspects add significant and cumulative improvements to the overall performance .

Finally , we show that the full network , including the three pairs of similarities , outperforms the state-of-the-art on a benchmark dataset .

The rest of the paper is organized as follows : Section 2 discusses the similarity triangle in answer ranking for cQA , Section 3 presents our pairwise neural network model for answering new questions in community forums , which integrates multiple levels of interaction , Section 4 describes the features we used , Section 5 presents our evaluation setup , the experiments and the results , Section 6 discusses some related work , and Section 7 wraps up the paper with a brief summary of the contributions and some possible directions for future work .

2 The Similarity Triangle in cQA Figure 1 presents an example illustrating the similarity triangle that we use when solving the answer ranking problem in cQA .

In the figure , q stands for the new question , q is an existing related question , and c is a comment within the thread of question q .

The edge qc relates to the main cQA task addressed in this paper , i.e. , deciding whether a comment for a potentially related question is a good answer to the original question .

We will say that the relation captures the relevance of c for q.

The edge qq represents the similarity between the original and the related questions .

We will call this relation relatedness .

We presented a neural-based approach to a novel problem in cQA , where given a new question , the task is to rank comments from related questionthreads according to their relevance as answers to the original question .

We explored the utility of three types of similarities between the original question , the related question , and the related comment .

We adopted a pairwise feed-forward neural network architecture , which takes as input the original question and two comments together with their corresponding related questions .

This allowed us to study the impact and the interaction effects of the question-question relatedness and commentto-related question appropriateness relations when solving the primary cQA relevance task .

The large performance gains obtained from using relatedness features show that question-question similarity plays a crucial role in finding relevant comments ( +30 MAP points ) .

Yet , including appropriateness relations is needed to achieve state-of-the-art results ( +3.3 MAP ) on benchmark datasets .

We also studied the impact of several types of features , especially domain-specific features , but also lexical features and syntactic embeddings .

We observed that lexical similarity MTE features prove the most important , followed by domain-specific features , and syntactic and semantic embeddings .

Overall , they all showed to be necessary to achieve state-of-the-art results .

In future work , we plan to use the labels for subtasks A and B , which are provided in the datasets in order to pre-train the corresponding components of the full network for answer ranking .

We further want to apply a similar network to other semantic similarity problems , such as textual entailment .

Detecting duplicated questions in forums is difficult while it offers merits for owners to prevent them and users from finding answers to questions without asking.

They propose to feed related questions and the corresponding answers coupled with the original question to a pairwise model to approach an answer ranking problem.

The proposed model achieves the state-of-the-art and they find that lexical similarity contributes more to the final performance than other similarity measures.


Many statistical learning problems in NLP call for local model search methods .

But accuracy tends to suffer with current techniques , which often explore either too narrowly or too broadly : hill-climbers can get stuck in local optima , whereas samplers may be inefficient .

We propose to arrange individual local optimizers into organized networks .

Our building blocks are operators of two types : ( i ) transform , which suggests new places to search , via non-random restarts from already-found local optima ; and ( ii ) join , which merges candidate solutions to find better optima .

Experiments on grammar induction show that pursuing different transforms ( e.g. , discarding parts of a learned model or ignoring portions of training data ) results in improvements .

Groups of locally-optimal solutions can be further perturbed jointly , by constructing mixtures .

Using these tools , we designed several modular dependency grammar induction networks of increasing complexity .

Our complete system achieves 48.6 % accuracy ( directed dependency macro-average over all 19 languages in the 2006/7 CoNLL data ) -more than 5 % higher than the previous state-of-the-art .

Statistical methods for grammar induction often boil down to solving non-convex optimization problems .

Early work attempted to locally maximize the likelihood of a corpus , using EM to estimate probabilities of dependency arcs between word bigrams ( Paskin 2001a ; 2001b ) .

That parsing model has since been extended to make unsupervised learning more feasible ( Klein and Manning , 2004 ; Headden et al . , 2009 ; Spitkovsky et al . , 2012b ) .

But even the latest techniques can be quite error-prone and sensitive to initialization , because of approximate , local search .

In theory , global optima can be found by enumerating all parse forests that derive a corpus , though this is usually prohibitively expensive in practice .

A preferable brute force approach is sampling , as in Markov-chain Monte Carlo ( MCMC ) and random restarts ( Hu et al . , 1994 ) , which hit exact solutions eventually .

Restarts can be giant steps in a parameter space that undo all previous work .

At the other extreme , MCMC may cling to a neighborhood , rejecting most proposed moves that would escape a local attractor .

Sampling methods thus take unbounded time to solve a problem ( and ca n't certify optimality ) but are useful for finding approximate solutions to grammar induction ( Cohn et al . , 2011 ; Mareček and Žabokrtský , 2011 ; Naseem and Barzilay , 2011 ) .

We propose an alternative ( deterministic ) search heuristic that combines local optimization via EM with non-random restarts .

Its new starting places are informed by previously found solutions , unlike conventional restarts , but may not resemble their predecessors , unlike typical MCMC moves .

We show that one good way to construct such steps in a parameter space is by forgetting some aspects of a learned model .

Another is by merging promising solutions , since even simple interpolation ( Jelinek and Mercer , 1980 ) of local optima may be superior to all of the originals .

Informed restarts can make it possible to explore a combinatorial search space more rapidly and thoroughly than with traditional

alone .

We proposed several simple algorithms for combining grammars and showed their usefulness in merging the outputs of iterative and static grammar induction systems .

Unlike conventional system combination , e.g. , in machine translation ( Xiao et al . , 2010 ) , ours do not require incoming models to be of similar quality to make improvements .

We exploited these properties of the combiners to reconcile grammars induced by different views of data ( Blum and Mitchell , 1998 ) .

One such view retains just the simple sentences , making it easier to recognize root words .

Another splits text into many inter-punctuation fragments , helping learn word associations .

The induced dependency trees can themselves also be viewed not only as directed structures but also as skeleton parses , facilitating the recovery of correct polarities for unlabeled dependency arcs .

By reusing templates , as in dynamic Bayesian network ( DBN ) frameworks ( Koller and Friedman , 11 The so-called Yarowsky-cautious modification of the original algorithm for unsupervised word-sense disambiguation .

2009 , § 6.2.2 ) , we managed to specify relatively " deep " learning architectures without sacrificing ( too much ) clarity or simplicity .

On a still more speculative note , we see two ( admittedly , tenuous ) connections to human cognition .

First , the benefits of not normalizing probabilities , when symmetrizing , might be related to human language processing through the base-rate fallacy ( Bar-Hillel , 1980 ; Kahneman and Tversky , 1982 ) and the availability heuristic ( Chapman , 1967 ; Tversky and Kahneman , 1973 ) , since people are notoriously bad at probability ( Attneave , 1953 ; Kahneman and Tversky , 1972 ; Kahneman and Tversky , 1973 ) .

And second , intermittent " unlearning " -though perhaps not of the kind that takes place inside of our transformsis an adaptation that can be essential to cognitive development in general , as evidenced by neuronal pruning in mammals ( Craik and Bialystok , 2006 ; Low and Cheng , 2006 ) . " Forgetful EM " strategies that reset subsets of parameters may thus , possibly , be no less relevant to unsupervised learning than is " partial EM , " which only suppresses updates , other EM variants ( Neal and Hinton , 1999 ) , or " dropout training " ( Hinton et al . , 2012 ; Wang and Manning , 2013 ) , which is important in supervised settings .

Future parsing models , in grammar induction , may benefit by modeling head-dependent relations separately from direction .

As frequently employed in tasks like semantic role labeling ( Carreras and Màrquez , 2005 ) and relation extraction ( Sun et al . , 2011 ) , it may be easier to first establish existence , before trying to understand its nature .

Other key next steps may include exploring more intelligent ways of combining systems ( Surdeanu and Manning , 2010 ; Petrov , 2010 ) and automating the operator discovery process .

Furthermore , we are optimistic that both count transforms and model recombination could be usefully incorporated into sampling : although symmetrized models may have higher cross-entropies , hence prone to rejection in vanilla MCMC , they could work well as seeds in multi-chain designs ; existing algorithms , such as MCMCMC ( Geyer , 1991 ) , which switch contents of adjacent chains running at different temperatures , may also benefit from introducing the option to combine solutions , in addition to just swapping them .

Existing local model search methods tend to explore too narrowly or broadly because they are error-prone and initialization sensitive, and finding global optima is expensive.

They propose a search heuristic that combines local optimization via EM with non-random restarts which may not resemble their predecessors by using previously found solutions.

The proposed methods show improvements in grammar induction experiments and a newly designed system outperforms the previous state-of-the-art by more than 5%.


We propose an algorithm to automatically induce the morphology of inflectional languages using only text corpora and no human input .

Our algorithm combines cues from orthography , semantics , and syntactic distributions to induce morphological relationships in German , Dutch , and English .

Using CELEX as a gold standard for

, we show our algorithm to be an improvement over any knowledge-free algorithm yet proposed .

Many NLP tasks , such as building machine-readable dictionaries , are dependent on the results of morphological analysis .

While morphological analyzers have existed since the early 1960s , current algorithms require human labor to build rules for morphological structure .

In an attempt to avoid this labor-intensive process , recent work has focused on machine-learning approaches to induce morphological structure using large corpora .

In this paper , we propose a knowledge-free algorithm to automatically induce the morphology structures of a language .

Our algorithm takes as input a large corpus and produces as output a set of conflation sets indicating the various inflected and derived forms for each word in the language .

As an example , the conflation set of the word " abuse " would contain " abuse " , " abused " , " abuses " , " abusive " , " abusively " , and so forth .

Our algorithm extends earlier approaches to morphology induction by combining various induced information sources : the semantic relatedness of the affixed forms using a Latent Semantic Analysis approach to corpusbased semantics ( Schone and Jurafsky , 2000 ) , affix frequency , syntactic context , and transitive closure .

Using the hand-labeled CELEX lexicon ( Baayen , et al . , 1993 ) as our gold standard , the current version of our algorithm achieves an F-score of 88.1 % on the task of identifying conflation sets in English , outperforming earlier algorithms .

Our algorithm is also applied to German and Dutch and evaluated on its ability to find prefixes , suffixes , and circumfixes in these languages .

To our knowledge , this serves as the first of complete regular morphological induction of German or Dutch ( although researchers such as Nakisa and Hahn ( 1996 ) have evaluated induction algorithms on morphological sub-problems in German ) .

We have illustrated three extensions to our earlier morphology induction work ( Schone and Jurafsky ( 2000 ) ) .

In addition to induced semantics , we incorporated induced orthographic , syntactic , and transitive information resulting in almost a 20 % relative reduction in overall induction error .

We have also extended the work by illustrating performance in German and Dutch where , to our knowledge , complete morphology induction performance measures have not previously been obtained .

Lastly , we showed a mechanism whereby circumfixes as well as combinations of prefixing and suffixing can be induced in lieu of the suffixonly strategies prevailing in most previous research .

For the future , we expect improvements could be derived by coupling this work , which focuses primarily on inducing regular morphology , with that of Yarowsky and Wicentowski ( 2000 ) , who assume some information about regular morphology in order to induce irregular morphology .

We also believe that some findings of this work can benefit other areas of linguistic induction , such as part of speech .

To avoid human labor to build rules, machine-leaning approaches have been studied to induce morphological structures using large corpora.

They propose a knowledge-free algorithm that induces the morphology of inflectional languages by combining cues from orthography, semantics, and syntactic distributions.

Their method is applied to German, Dutch, and English, and improves over any knowledge-free algorithm using the CELEX lexicon as a gold standard in English.


Chatbot is increasingly thriving in different domains , however , because of unexpected discourse complexity and training data sparseness , its potential distrust hatches vital apprehension .

Recently , Machine-Human Chatting Handoff ( MHCH ) , predicting chatbot failure and enabling human-algorithm collaboration to enhance chatbot quality , has attracted increasing attention from industry and academia .

In this study , we propose a novel model , Role-Selected Sharing Network ( RSSN ) , which integrates both dialogue satisfaction estimation and handoff prediction in one multi-task learning framework .

Unlike prior efforts in dialog mining , by utilizing local user satisfaction as a bridge , global satisfaction detector and handoff predictor can effectively exchange critical information .

Specifically , we decouple the relation and interaction between the two tasks by the role information after the shared encoder .

Extensive

on two public datasets demonstrate the effectiveness of our model .

Chatbot , as one of the recent palpable AI excitements , has been widely adopted to reduce the cost of customer service ( Qiu et al . , 2017 ; Ram et al . , 2018 ; Zhou et al . , 2020 ) .

However , due to the complexity of human conversation , auto-chatbot can hardly meet all users ' needs , while its potential failure perceives skepticism .

AI-enabled customer service , for instance , may trigger unexpected business losses because of chatbot failures ( Radziwill and Benton , 2017 ; Rajendran et al . , 2019 ) .

Moreover , for chatbot adoption in sensitive areas , such as healthcare ( Chung and Park , 2019 ) and criminal justice ( Wang et al . , 2020a ) , any subtle statistical miscalculation may trigger serious health and legal * Corresponding authors .

What a business !

It has been a week ! utter 3 We will ship goods about a week after placing the order .

Please be patient . utter 2 Sorry , my dear customer .

We have pushed the warehouse to ship as soon as possible , and we will compensate the freight for you . utter 7 We will try our best to improve your shopping experience .

Thank you for your understanding and patience . consequences .

To address this problem , recently , scholars proposed new dialog mining tasks to autoassess dialogue satisfaction , a.k.a .

Service Satisfaction Analysis ( SSA ) at dialogue-level ( Song et al . , 2019 ) , and to predict potential chatbot failure via machine-human chatting handoff ( MHCH ) at utterance-level ( Huang et al . , 2018 ; Liu et al . , 2021 ) .

In a MHCH context , algorithm can transfer an ongoing auto-dialogue to the human agent when the current utterance is confusing .

Figure 1 depicts an exemplar dialogue of online customer service .

In this dialogue , the chatbot gives an unsatisfied answer about shipping , thus causing the customer 's complaint ( local dissatisfaction utter 2 and utter 3 ) .

Ideally , chatbot should be able to detect the negative ( local ) emotion ( utter 3 ) and tries to appease complaints , but this problem remains unresolved .

If chatbot continues , the customer may cancel the deal and give a negative rating ( dialogue global dissatisfaction ) .

With MHCH ( detects the risks of utter 2 and utter 3 ) , the dialogue can be transferred to the human agent , who is better at handling , compensating , and comforting the customer and enhance customer satisfaction .

This example illustrates the cross-impact between handoff and dialogue ( local+global ) satisfaction .

Intuitively , MHCH and SSA tasks can be compatible and complementary given a dialogue discourse , i.e. , the local satisfaction is related to the quality of the conversation ( Bodigutla et al . , 2019a ( Bodigutla et al . , , 2020 ) ) , which can support the handoff judgment and ultimately affect the overall satisfaction .

On the one hand , handoff labels of utterances are highly pertinent to local satisfaction , e.g. , one can utilize single handoff information to enhance local satisfaction prediction , which ultimately contributes to the overall satisfaction estimation .

On the other hand , the overall satisfaction is obtained by combining local satisfactions , which reflects the quality in terms of answer generation , language understanding , and emotion perception , and subsequently helps to facilitate handoff judgment .

In recent years , researchers ( Bodigutla et al . , 2019a , b ; Ultes , 2019 ; Bodigutla et al . , 2020 ) explore joint of turn and dialogue level qualities in spoken dialogue systems .

In terms of general dialogue system , to improve the efficiency of dialogue management , Qin et al .

( 2020 ) propose a co-interactive relation layer to explicitly examine the cross-impact and model the interaction between sentiment classification and dialog act recognition , which are relevant tasks at the same level ( utterancelevel ) .

However , MHCH ( utterance-level ) and SSA ( dialogue-level ) target satisfaction at different levels .

More importantly , handoff labels of utterances are more comprehensive and pertinent to local satisfaction than sentiment polarities .

Meanwhile , customer utterances have significant impacts on the overall satisfaction ( Song et al . , 2019 ) , which motivates us that the role information can be critical for knowledge transfer of these two tasks .

To address the aforementioned issues , we propose an innovative Role-Selected Sharing Network ( RSSN ) for handoff prediction and dialogue satisfaction estimation , which utilizes role information to selectively characterize complex relations and interactions between two tasks .

To the best of our knowledge , it is the pioneer investigation to leverage the multi-task learning approach for integrating MHCH and SSA .

In practice , we first adopt a shared encoder to obtain the shared representations of utterances .

Inspired by the co-attention mechanism ( Xiong et al . , 2016 ; Qin et al . , 2020 ) , the shared representations are then fed into the roleselected sharing module , which consists of two directional interactions : MHCH to SSA and SSA to MHCH .

This module is used to get the fusion of MHCH and SSA representations .

We propose the role-selected sharing module based on the hypothesis that the role information can benefit the tasks ' performances .

The satisfaction distributions of utterances from different roles ( agent and customer ) are different , and the effects for the tasks are also different .

Specifically , the satisfaction of agent is non-negative .

The utterances from agent can enrich the context of customer 's utterances and indirectly affect satisfaction polarity .

Thus , directly employing local satisfaction of agent into the interaction with handoff may introduce noise .

In the proposed role-selected sharing module , we adopt local satisfaction based on the role information : only the local satisfaction from customer can be adopted to interact with handoff information .

By this means , we can control knowledge transfer for both tasks and make our framework more explainable .

The final integrated outputs are then fed to separate decoders for handoff and satisfaction predictions .

To summarize , our contributions are mainly as follows : ( 1 ) We introduce a novel multi-task learning framework for combining machine-human chatting handoff and service satisfaction analysis .

( 2 ) We propose a Role-Selected Sharing Network for handoff prediction and satisfaction rating estimation , which can utilize different role information to control knowledge transfer for both tasks and enhance model performance and explainability .

( 3 ) The experimental results demonstrate that our model outperforms a series of baselines that consists of the state-of-the-art ( SOTA ) models on each task and multi-task learning models for both tasks .

To assist other scholars in reproducing the experiment outcomes , we release the codes and the annotated dataset1 .

In this paper , we propose an innovative multi-task framework for service satisfaction analysis and machine-human chatting handoff , which deliberately establishes the mutual interrelation for each other .

Specifically , we propose a Role-Selected Sharing Network for joint handoff prediction and satisfaction estimation , utilizing role and positional information to control knowledge transfer for both tasks .

Extensive and analyses reveal that explicitly modeling the interrelation between the two tasks can boost the performance mutually .

However , our model has not been calibrated to account for user preferences and biases , which we plan to address in future work .

Moreover , we will further explore how to adjust the handoff priority with the assistance of personalized information .

For chatbots to achieve satisfactory user experience, a system needs to model user satisfaction well but existing systems model local uttrance-level and global dialogue-level disjointly.

They propose a Role-Selected Sharing Network which models local dialogue satisfaction estimation and global handoff prediction in one multi-task learning framework.

Experiments with two datasets show the proposed model outperforms existing models for both tasks modeled in multi-task learning showing that interrelation helps performance improvements.