When Part-of-Speech annotated data is scarce , e.g. for under-resourced languages , one can turn to cross-lingual transfer and crawled dictionaries to collect partially supervised data .

We cast this problem in the framework of ambiguous learning and show how to learn an accurate history-based model .

Experiments on ten languages show significant improvements over prior state of the art performance .

In the past two decades , supervised Machine Learning techniques have established new performance standards for many NLP tasks .

Their success however crucially depends on the availability of annotated in-domain data , a not so common situation .

This means that for many application domains and/or less-resourced languages , alternative ML techniques need to be designed to accommodate unannotated or partially annotated data .

Several attempts have recently been made to mitigate the lack of annotated corpora using parallel data pairing a ( source ) text in a resource-rich language with its counterpart in a less-resourced language .

By transferring labels from the source to the target , it becomes possible to obtain noisy , yet useful , annotations that can be used to train a model for the target language in a weakly supervised manner .

This research trend was initiated by Yarowsky et al . ( 2001 ) , who consider the transfer of POS and other syntactic information , and further developed in ( Hwa et al . , 2005 ; Ganchev et al . , 2009 ) for syntactic dependencies , in ( Padó and Lapata , 2009 ; Kozhevnikov and Titov , 2013 ; van der Plas et al . , 2014 ) for semantic role labeling and in ( Kim et al . , 2012 ) for named-entity recognition , to name a few .

Assuming that labels can actually be projected across languages , these techniques face the issue of extending standard supervised techniques with partial and/or uncertain labels in the presence of alignment noise .

In comparison to the early approach of Yarowsky et al . ( 2001 ) in which POS are directly transferred , subject to heuristic filtering rules , recent works consider the integration of softer constraints using expectation regularization techniques ( Wang and Manning , 2014 ) , the combination of alignment-based POS transfer with additional information sources such as dictionaries ( Li et al . , 2012 ; Täckström et al . , 2013 ) ( Section 2 ) , or even the simultaneous use of both techniques ( Ganchev and Das , 2013 ) .

In this paper , we reproduce the weakly supervised setting of Täckström et al . ( 2013 ) .

By recasting this setting in the framework of ambiguous learning ( Bordes et al . , 2010 ; Cour et al . , 2011 ) ( Section 3 ) , we propose an alternative learning methodology and show that it improves the state of the art performance on a large array of languages ( Section 4 ) .

Our analysis of the remaining errors suggests that in cross-lingual settings , improvements of error rates can have multiple causes and should be looked at with great care ( Section 4.2 ) .

All tools and resources used in this study are available at http://perso.limsi.fr/ wisniews / ambiguous .

In this paper , we have presented a novel learning methodology to learn from ambiguous supervision information , and used it to train several POS taggers .

Using this method , we have been able to achieve performance that surpasses the best reported results , sometimes by a wide margin .

Further work will attempt to better analyse these results , which could be caused by several subtle differences between HBAL and the baseline system .

Nonetheless , these experiments confirm that cross-lingual projection of annotations have the potential to help in building very efficient POS taggers with very little monolingual supervision data .

Our analysis of these results also suggests that , for this task , additional gains might be more easily obtained by fixing systematic biases introduced by conflicting mappings between tags or by train / test domain mismatch than by designing more sophisticated weakly supervised learners .


Recognizing shallow linguistic patterns , such as basic syntactic relationships between words , is a common task in applied natural language and text processing .

The common practice for approaching this task is by tedious manual definition of possible pattern structures , often in the form of regular expressions or finite automata .

This paper presents a novel memory-based learning method that recognizes shallow patterns in new text based on a bracketed training corpus .

The training data are stored as-is , in efficient suffix-tree data structures .

Generalization is performed on-line at recognition time by comparing subsequences of the new text to positive and negative evidence in the corpus .

This way , no information in the training is lost , as can happen in other learning systems that construct a single generalized model at the time of training .

The paper presents experimental results for recognizing noun phrase , subject-verb and verb-object patterns in English .

Since the learning approach enables easy porting to new domains , we plan to apply it to syntactic patterns in other languages and to sub-language patterns for information extraction .

Identifying local patterns of syntactic sequences and relationships is a fundamental task in natural language processing ( NLP ) .

Such patterns may correspond to syntactic phrases , like noun phrases , or to pairs of words that participate in a syntactic relationship , like the heads of a verb-object relation .

Such patterns have been found useful in various application areas , including information extraction , text summarization , and bilingual alignment .

Syntactic patterns are useful also for many basic computational linguistic tasks , such as statistical word similarity and various disambiguation problems .

One approach for detecting syntactic patterns is to obtain a full parse of a sentence and then extract the required patterns .

However , obtaining a complete parse tree for a sentence is difficult in many cases , and may not be necessary at all for identifying most instances of local syntactic patterns .

An alternative approach is to avoid the complexity of full parsing and instead to rely only on local information .

A variety of methods have been developed within this framework , known as shallow parsing , chunking , local parsing etc .

( e.g. , ( Abney , 1991 ; Greffenstette , 1993 ) ) .

These works have shown that it is possible to identify most instances of local syntactic patterns by rules that examine only the pattern itself and its nearby context .

Often , the rules are applied to sentences that were tagged by partof-speech ( POS ) and are phrased by some form of regular expressions or finite state automata .

Manual writing of local syntactic rules has become a common practice for many applications .

However , writing rules is often tedious and time consuming .

Furthermore , extending the rules to different languages or sub-language domains can require substantial resources and expertise that are often not available .

As in many areas of NLP , a learning approach is appealing .

Surprisingly , though , rather little work has been devoted to learning local syntactic patterns , mostly noun phrases ( Ramshaw and Marcus , 1995 ; Vilain and Day , 1996 ) .

This paper presents a novel general learning approach for recognizing local sequential patterns , that may be perceived as falling within the memorybased learning paradigm .

The method utilizes a part-of-speech tagged training corpus in which all instances of the target pattern are marked ( bracketed ) .

The training data are stored as-is in suffix-tree data structures , which enable linear time searching for subsequences in the corpus .

The memory-based nature of the presented algorithm stems from its deduction strategy : a new instance of the target pattern is recognized by examining the raw training corpus , searching for positive and negative evidence with respect to the given test sequence .

No model is created for the training corpus , and the raw examples are not converted to any other representation .

Suppose we 1We use here the POS tags : DT ----determiner , ADJ = adjective , hDV = adverb , C0NJ = conjunction , VB = verb , PP = preposition , NN = singular noun , and NNP ----plural noun .

want to decide whether the candidate sequence DT ADJ ADJ NN NNP is a noun phrase ( NP ) by comparing it to the training corpus .

A good match would be if the entire sequence appears as-is several times in the corpus .

However , due to data sparseness , an exact match can not always be expected .

A somewhat weaker match may be obtained if we consider sub-parts of the candidate sequence ( called tiles ) .

For example , suppose the corpus contains noun phrase instances with the following structures :

The first structure provides positive evidence that the sequence " DT ADJ ADJ NN " is a possible NP prefix while the second structure provides evidence for " ADJ NN NNP " being an NP suffix .

Together , these two training instances provide positive evidence that covers the entire candidate .

Considering evidence for sub-parts of the pattern enables us to generalize over the exact structures that are present in the corpus .

Similarly , we also consider the negative evidence for such sub-parts by noting where they occur in the corpus without being a corresponding part of a target instance .

The proposed method , as described in detail in the next section , formalizes this type of reasoning .

It searches specialized data structures for both positive and negative evidence for sub-parts of the candidate structure , and considers additional factors such as context and evidence overlap .

Section 3 presents experimental results for three target syntactic patterns in English , and Section 4 describes related work .

We have presented a novel general schema and a particular instantiation of it for learning sequential patterns .

Applying the method to three syntactic patterns in English yielded positive results , suggesting its applicability for recognizing local linguistic patterns .

In future work we plan to investigate a datadriven approach for optimal selection and weighting of statistical features of candidate scores , as well as to apply the method to syntactic patterns of Hebrew and to domain-specific patterns for information extraction .


In this paper , we study the problem of automatic enrichment of a morphologically underspecified treebank for Arabic , a morphologically rich language .

We show that we can map from a tagset of size six to one with 485 tags at an accuracy rate of 94%-95 % .

We can also identify the unspecified lemmas in the treebank with an accuracy over 97 % .

Furthermore , we demonstrate that using our automatic annotations improves the performance of a state-of-the-art Arabic morphological tagger .

Our approach combines a variety of techniques from corpus-based statistical models to linguistic rules that target specific phenomena .

These results suggest that the cost of treebanking can be reduced by designing underspecified treebanks that can be subsequently enriched automatically .

Collections of manually-annotated morphological and syntactic analyses of sentences , or treebanks , are an important resource for building statistical parsing models or for syntax-aware approaches to applications such as machine translation .

Rich treebank annotations have also been used for a variety of natural language processing ( NLP ) applications such as tokenization , diacritization , part-of-speech ( POS ) tagging , morphological disambiguation , base phrase chunking , and semantic role labeling .

The development of a treebank with rich annotations is demanding in time and money , especially for morphologically complex languages .

Consequently , the richer the annotation , the slower the annotation process and the smaller the size of the treebank .

As such , a tradeoff is usually made between the size of the treebank and the richness of its annotations .

In this paper , we investigate the possibility of automatically enriching the morphologically underspecified Columbia Arabic Treebank ( CATiB ) ( Habash and Roth , 2009 ; Habash et al . , 2009 ) with the more complex POS tags and lemmas used in the Penn Arabic Treebank ( PATB ) ( Maamouri et al . , 2004 ) .

We employ a variety of techniques that range from corpus-based statistical models to handwritten rules based on linguistic observations .

Our best method reaches accuracy rates of 94%-95 % on full POS tag identification .

We can also identify the unspecified lemmas in CATiB with an accuracy over 97 % .

37 % of our POS tag errors are due to gold tree or gold POS errors .

A learning curve experiment to evaluate the dependence of our method on annotated data shows that while the quality of some components may reduce sharply with less data ( 12 % absolute reduction in accuracy when using 1 32 of the data or some 10 K annotated words ) , the overall effect is a lot smaller ( 2 % absolute drop ) .

The rest of this paper is structured as follows : Section 2 presents related work ; Section 3 details various language background facts about Arabic and its treebanking ; Section 4 explains our approach ; and Section 5 presents and discusses our results .

We have demonstrated that an underspecified version of an Arabic treebank can be fully specified for Arabic 's rich morphology automatically at an accuracy rate of 94%-95 % for POS tags and 97 % for lemmas .

Our approach combines a variety of techniques from corpus-based statistical models ( which require some rich annotations ) to linguistic rules that target specific phenomena .

Since the underspecified treebank is much faster to manually annotate than its fully specified version , these results suggest that the cost of treebanking can be reduced by designing underspecified treebanks that can be subsequently enriched automatically .

In the future , we plan to extend the automatic enrichment effort to include more complex features such as empty nodes and semantic labels .

We also plan to take the insights from this effort and apply them to treebanks of other languages .

A small portion of a treebank that is fully annotated in rich format will of course be needed before we can apply these insights to other languages .


This paper evaluates the utility of Rhetorical Structure Theory ( RST ) trees and relations in discourse coherence evaluation .

We show that incorporating silver-standard RST features can increase accuracy when classifying coherence .

We demonstrate this through our tree-recursive neural model , namely RST-Recursive , which takes advantage of the text 's RST features produced by a state of the art RST parser .

We evaluate our approach on the Grammarly Corpus for Discourse Coherence ( GCDC ) and show that when ensembled with the current state of the art , we can achieve the new state of the art accuracy on this benchmark .

Furthermore , when deployed alone , RST-Recursive achieves competitive accuracy while having 62 % fewer parameters .

Discourse coherence has been the subject of much research in Computational Linguistics thanks to its widespread applications ( Lai and Tetreault , 2018 ) .

Most current methods can be described as either stemming from explicit representations based on the Centering Theory ( Grosz et al . , 1994 ) , or deep learning approaches that learn without the use of hand-crafted linguistic features .

Our work explores a third research avenue based on the Rhetorical Structure Theory ( RST ) ( Mann and Thompson , 1988 ) .

We hypothesize that texts of low / high coherence tend to adhere to different discourse structures .

Thus , we pose that using even silver-standard RST features should help in separating coherent texts from incoherent ones .

This stems from the definition of the coherence itselfas the writer of a document needs to follow specific rules for building a clear narrative or argument structure in which the role of each constituent of the document should be appropriate with respect * to its local and global context , and even existing discourse parsers should be able to predict a plausible structure that is consistent across all coherent documents .

However , if a parser has difficulty interpreting a given document , it will be more likely to produce unrealistic trees with improbable patterns of discourse relations between constituents .

This idea was first explored by Feng et al . ( 2014 ) , who followed an approach similar to Barzilay and Lapata ( 2008 ) by estimating entity transition likelihoods , but instead using discourse relations ( predicted by a state of the art discourse parser ( Feng and Hirst , 2014 ) ) that entities participate in as opposed to their grammatical roles .

Their method achieved significant improvements in performance even when using silver-standard discourse trees , showing potential in the use of parsed RST features for classifying textual coherence .

In this paper , we explore the usefulness of silverstandard parsed RST features in neural coherence classification .

We propose two new methods , RST-Recursive and Ensemble .

The former achieves reasonably good performance , only 2 % short of state of the art , while more robust with 62 % fewer parameters .

The latter demonstrates the added advantage of RST features in improving classification accuracy of the existing state of the art methods by setting new state of the art performance with a modest but promising margin .

This signifies that the document 's rhetorical structure is an important aspect of its perceived clarity .

Naturally , this improvement in performance is bounded by the quality of parsed RST features and could increase as better discourse parsers are developed .

In the future , exploring other RST-based architectures for coherence classification , as well as better RST ensemble schemes and improving RST parsing can be avenues of potentially fruitful research .

Additional research on multipronged approaches that draw from Centering Theory , RST and deep learning all together can also be of value .


Morphological disambiguation proceeds in 2 stages : ( 1 ) an analyzer provides all possible analyses for a given token and ( 2 ) a stochastic disambiguation module picks the most likely analysis in context .

When the analyzer does not recognize a given token , we hit the problem of unknowns .

In large scale corpora , unknowns appear at a rate of 5 to 10 % ( depending on the genre and the maturity of the lexicon ) .

We address the task of computing the distribution p(t|w ) for unknown words for full morphological disambiguation in Hebrew .

We introduce a novel algorithm that is language independent : it exploits a maximum entropy letters model trained over the known words observed in the corpus and the distribution of the unknown words in known tag contexts , through iterative approximation .

The algorithm achieves 30 % error reduction on disambiguation of unknown words over a competitive baseline ( to a level of 70 % accurate full disambiguation of unknown words ) .

We have also verified that taking advantage of a strong language-specific model of morphological patterns provides the same level of disambiguation .

The algorithm we have developed exploits distributional information latent in a wide-coverage lexicon and large quantities of unlabeled data .

The term unknowns denotes tokens in a text that can not be resolved in a given lexicon .

For the task of full morphological analysis , the lexicon must provide all possible morphological analyses for any given token .

In this case , unknown tokens can be categorized into two classes of missing information : unknown tokens are not recognized at all by the lexicon , and unknown analyses , where the set of analyses for a lexeme does not contain the correct analysis for a given token .

Despite efforts on improving the underlying lexicon , unknowns typically represent 5 % to 10 % of the number of tokens in large-scale corpora .

The alternative to continuously investing manual effort in improving the lexicon is to design methods to learn possible analyses for unknowns from observable features : their letter structure and their context .

In this paper , we investigate the characteristics of Hebrew unknowns for full morphological analysis , and propose a new method for handling such unavoidable lack of information .

Our method generates a distribution of possible analyses for unknowns .

In our evaluation , these learned distributions include the correct analysis for unknown words in 85 % of the cases , contributing an error reduction of over 30 % over a competitive baseline for the overall task of full morphological analysis in Hebrew .

The task of a morphological analyzer is to produce all possible analyses for a given token .

In Hebrew , the analysis for each token is of the form lexeme-and-features 1 : lemma , affixes , lexical cate-gory ( POS ) , and a set of inflection properties ( according to the POS ) -gender , number , person , status and tense .

In this work , we refer to the morphological analyzer of MILA -the Knowledge Center for Processing Hebrew 2 ( hereafter KC analyzer ) .

It is a synthetic analyzer , composed of two data resources -a lexicon of about 2,400 lexemes , and a set of generation rules ( see ( Adler , 2007 , Section 4.2 ) ) .

In addition , we use an unlabeled text corpus , composed of stories taken from three Hebrew daily news papers ( Aruts 7 , Haaretz , The Marker ) , of 42 M tokens .

We observed 3,561 different composite tags ( e.g. , noun-sing-fem-prepPrefix : be ) over this corpus .

These 3,561 tags form the large tagset over which we train our learner .

On the one hand , this tagset is much larger than the largest tagset used in English ( from 17 tags in most unsupervised POS tagging experiments , to the 46 tags of the WSJ corpus and the about 150 tags of the LOB corpus ) .

On the other hand , our tagset is intrinsically factored as a set of dependent sub-features , which we explicitly represent .

The task we address in this paper is morphological disambiguation : given a sentence , obtain the list of all possible analyses for each word from the analyzer , and disambiguate each word in context .

On average , each token in the 42 M corpus is given 2.7 possible analyses by the analyzer ( much higher than the average 1.41 POS tag ambiguity reported in English ( Dermatas and Kokkinakis , 1995 ) ) .

In previous work , we report disambiguation rates of 89 % for full morphological disambiguation ( using an unsupervised EM-HMM model ) and 92.5 % for part of speech and segmentation ( without assigning all the inflectional features of the words ) .

In order to estimate the importance of unknowns in Hebrew , we analyze tokens in several aspects : ( 1 ) the number of unknown tokens , as observed on the corpus of 42 M tokens ; ( 2 ) a manual classification of a sample of 10 K unknown token types out of the 200 K unknown types identified in the corpus ; ( 3 ) the number of unknown analyses , based on an annotated corpus of 200 K tokens , and their classification .

About 4.5 % of the 42 M token instances in the Buckwalter 's Arabic analyzer ( 2004 ) , which looks for any legal combination of prefix-stem-suffix , but does not provide full morphological features such as gender , number , case etc . 2

http://mila.cs.technion.ac.il.html training corpus were unknown tokens ( 45 % of the 450 K token types ) .

For less edited text , such as random text sampled from the Web , the percentage is much higher -about 7.5 % .

In order to classify these unknown tokens , we sampled 10 K unknown token types and examined them manually .

The classification of these tokens with their distribution is shown in Table 1 3 .

As can be seen , there are two main classes of unknown token types : Neologisms ( 32 % ) and Proper nouns ( 48 % ) , which cover about 80 % of the unknown token instances .

The POS distribution of the unknown tokens of our annotated corpus is shown in Table 2 .

As expected , most unknowns are open class words : proper names , nouns or adjectives .

Regarding unknown analyses , in our annotated corpus , we found 3 % of the 100 K token instances were missing the correct analysis in the lexicon ( 3.65 % of the token types ) .

The POS distribution of the unknown analyses is listed in Table 2 .

The high rate of unknown analyses for prepositions at about 3 % is a specific phenomenon in Hebrew , where prepositions are often prefixes agglutinated to the first word of the noun phrase they head .

We observe the very low rate of unknown verbs ( 2 % ) -which are well marked morphologically in Hebrew , and where the rate of neologism introduction seems quite low .

This evidence illustrates the need for resolution of unknowns : The naive policy of selecting ' proper name ' for all unknowns will cover only half of the errors caused by unknown tokens , i.e. , 30 % of the whole unknown tokens and analyses .

The other 70 % of the unknowns ( 5.3 % of the words in the text in our experiments ) will be assigned a wrong tag .

As a result of this observation , our strategy is to focus on full morphological analysis for unknown tokens and apply a proper name classifier for unknown analyses and unknown tokens .

In this paper , we investigate various methods for achieving full morphological analysis distribution for unknown tokens .

The methods are not based on an annotated corpus , nor on hand-crafted rules , but instead exploit the distribution of words in an available lexicon and the letter similarity of the unknown words with known words .

We have addressed the task of computing the distribution p(t|w ) for unknown words for full morphological disambiguation in Hebrew .

The algorithm we have proposed is language independent : it exploits a maximum entropy letters model trained over the known words observed in the corpus and the distribution of the unknown words in known tag contexts , through iterative approximation .

We observe that the task of analyzing unknown tokens for POS in Hebrew remains challenging when compared with English ( 78 % vs. 85 % ) .

We hypothesize this is due to the highly ambiguous pattern of prefixation that occurs widely in Hebrew and are currently investigating syntagmatic models that exploit the specific nature of agglutinated prefixes in Hebrew .

Our method generates a distribution of possible analyses for unknowns .


This paper introduces the PhotoBook dataset , a large-scale collection of visually-grounded , task-oriented dialogues in English designed to investigate shared dialogue history accumulating during conversation .

Taking inspiration from seminal work on dialogue analysis , we propose a data-collection task formulated as a collaborative game prompting two online participants to refer to images utilising both their visual context as well as previously established referring expressions .

We provide a detailed description of the task setup and a thorough analysis of the 2,500 dialogues collected .

To further illustrate the novel features of the dataset , we propose a baseline model for reference resolution which uses a simple method to take into account shared information accumulated in a reference chain .

Our results show that this information is particularly important to resolve later descriptions and underline the need to develop more sophisticated models of common ground in dialogue interaction .

The past few years have seen an increasing interest in developing computational agents for visually grounded dialogue , the task of using natural language to communicate about visual content in a multi-agent setup .

The models developed for this task often focus on specific aspects such as image labelling ( Mao et al . , 2016 ; Vedantam et al . , 2017 ) , object reference ( Kazemzadeh et al . , 2014 ; De Vries et al . , 2017a ) , visual question answering ( Antol et al . , 2015 ) , and first attempts of visual dialogue proper ( Das et al . , 2017 ) , but fail to produce consistent outputs over a conversation .

We hypothesise that one of the main reasons for this shortcoming is the models ' inability to effectively utilise dialogue history .

Human interlocutors are known to collaboratively establish a shared repository of mutual information during a conversation ( Clark and Wilkes-Gibbs , 1986 ; Clark , 1996 ; Brennan and Clark , 1996 ) .

This common ground ( Stalnaker , 1978 ) then is used to optimise understanding and communication efficiency .

Equipping artificial dialogue agents with a similar representation of dialogue context thus is a pivotal next step in improving the quality of their dialogue output .

To facilitate progress towards more consistent and effective conversation models , we introduce the PhotoBook dataset : a large collection of 2,500 human-human goal-oriented English conversations between two participants , who are asked to identify shared images in their respective photo books by exchanging messages via written chat .

This setup takes inspiration from experimental paradigms extensively used within the psycholinguistics literature to investigate partnerspecific common ground ( for an overview , see Brown-Schmidt et al . , 2015 ) , adapting them to the requirements imposed by online crowdsourcing methods .

The task is formulated as a game consisting of five rounds .

Figure 1 shows an example of a participant 's display .

Over the five rounds of a game , a selection of previously displayed images will be visible again , prompting participants to re-refer to images utilising both their visual context as well as previously established referring expressions .

The resulting dialogue data therefore allows for tracking the common ground developing between dialogue participants .

We describe in detail the PhotoBook task and the data collection , and present a thorough analysis of the dialogues in the dataset .

In addition , to showcase how the new dataset may be exploited for computational modelling , we propose a reference resolution baseline model trained to identify target images being discussed in a given dialogue segment .

The model uses a simple method to take into account information accumulated in a reference chain .

Our results show that this information is particularly important to resolve later descriptions and highlight the importance of developing more sophisticated models of common ground in dialogue interaction .

The PhotoBook dataset , together with the data collection protocol , the automatically extracted reference chains , and the code used for our analyses and models are available at the following site : https://dmg-photobook.github.io .

We have presented the first large-scale dataset of goal-oriented , visually grounded dialogues for investigating shared linguistic history .

Through the data collection 's task setup , participants repeatedly refer to a controlled set of target images , which allows them to improve task efficiency if they utilise their developing common ground and establish conceptual pacts ( Brennan and Clark , 1996 ) on referring expressions .

The collected dialogues exhibit a significant shortening of utterances throughout a game , with final referring expressions starkly differing from both standard image captions and initial descriptions .

To illustrate the potential of the dataset , we trained a baseline reference resolution model and showed that information accumulated over a reference chain helps to resolve later descriptions .

Our results suggest that more sophisticated models are needed to fully exploit shared linguistic history .

The current paper showcases only some of the aspects of the PhotoBook dataset , which we hereby release to the public ( https:// dmg-photobook.github.io ) .

In future work , the data can be used to further investigate common ground and conceptual pacts ; be extended through manual annotations for a more thorough linguistic analysis of co-reference chains ; exploit the combination of vision and language to develop computational models for referring expression generation ; or use the PhotoBook task in the ParlAI framework for Turing-Test-like evaluation of dialogue agents .


In this paper , we present an innovative chart mining technique for improving parse coverage based on partial parse outputs from precision grammars .

The general approach of mining features from partial analyses is applicable to a range of lexical acquisition tasks , and is particularly suited to domain-specific lexical tuning and lexical acquisition using lowcoverage grammars .

As an illustration of the functionality of our proposed technique , we develop a lexical acquisition model for English verb particle constructions which operates over unlexicalised features mined from a partial parsing chart .

The proposed technique is shown to outperform a state-of-the-art parser over the target task , despite being based on relatively simplistic features .

Parsing with precision grammars is increasingly achieving broad coverage over open-domain texts for a range of constraint-based frameworks ( e.g. , TAG , LFG , HPSG and CCG ) , and is being used in real-world applications including information extraction , question answering , grammar checking and machine translation ( Uszkoreit , 2002 ; Oepen et al . , 2004 ; Frank et al . , 2006 ; Zhang and Kordoni , 2008 ; MacKinlay et al . , 2009 ) .

In this context , a " precision grammar " is a grammar which has been engineered to model grammaticality , and contrasts with a treebank-induced grammar , for example .

Inevitably , however , such applications demand complete parsing outputs , based on the assumption that the text under investigation will be completely analysable by the grammar .

As precision grammars generally make strong assumptions about complete lexical coverage and grammaticality of the input , their utility is limited over noisy or domain-specific data .

This lack of complete coverage can make parsing with precision grammars less attractive than parsing with shallower methods .

One technique that has been successfully applied to improve parser and grammar coverage over a given corpus is error mining ( van Noord , 2004 ; de Kok et al . , 2009 ) , whereby n-grams with low " parsability " are gathered from the large-scale output of a parser as an indication of parser or ( precision ) grammar errors .

However , error mining is very much oriented towards grammar engineering : its results are a mixture of different ( mistreated ) linguistic phenomena together with engineering errors for the grammar engineer to work through and act upon .

Additionally , it generally does not provide any insight into the cause of the parser failure , and it is difficult to identify specific language phenomena from the output .

In this paper , we instead propose a chart mining technique that works on intermediate parsing results from a parsing chart .

In essence , the method analyses the validity of different analyses for words or constructions based on the " lifetime " and probability of each within the chart , combining the constraints of the grammar with probabilities to evaluate the plausibility of each .

For purposes of exemplification of the proposed technique , we apply chart mining to a deep lexical acquisition ( DLA ) task , using a maximum entropybased prediction model trained over a seed lexicon and treebank .

The experimental set up is the following : given a set of sentences containing putative instances of English verb particle constructions , extract a list of non-compositional VPCs optionally with valence information .

For comparison , we parse the same sentence set using a state-of-the-art statistical parser , and extract the VPCs from the parser output .

Our results show that our chart mining method produces a model which is superior to the treebank parser .

To our knowledge , the only other work that has looked at partial parsing results of precision grammars as a means of linguistic error analysis is that of Kiefer et al . ( 1999 ) and Zhang et al . ( 2007a ) , where partial parsing models were proposed to select a set of passive edges that together cover the input sequence .

Compared to these approaches , our proposed chart mining technique is more general and can be adapted to specific tasks and domains .

While we experiment exclusively with an HPSG grammar in this paper , it is important to note that the proposed method can be applied to any grammar formalism which is compatible with chart parsing , and where it is possible to describe an unlexicalised lexical entry for the different categories of lexical item that are to be extracted ( see Section 3.2 for details ) .

The remainder of the paper is organised as follows .

Section 2 defines the task of VPC extraction .

Section 3 presents the chart mining technique and the feature extraction process for the VPC extraction task .

Section 4 evaluates the model performance with comparison to two competitor models over several different measures .

Section 5 further discusses the general applicability of chart mining .

Finally , Section 6 concludes the paper .

We have proposed a chart mining technique for lexical acquisition based on partial parsing with precision grammars .

We applied the proposed method to the task of extracting English verb particle constructions from a prescribed set of corpus instances .

Our results showed that simple unlexicalised features mined from the chart can be used to effectively extract VPCs , and that the model outperforms a probabilistic baseline and the Charniak parser at VPC extraction .


A variety of query systems have been developed for interrogating parsed corpora , or treebanks .

With the arrival of efficient , widecoverage parsers , it is feasible to create very large databases of trees .

However , existing approaches that use in-memory search , or relational or XML database technologies , do not scale up .

We describe a method for storage , indexing , and query of treebanks that uses an information retrieval engine .

Several experiments with a large treebank demonstrate excellent scaling characteristics for a wide range of query types .

This work facilitates the curation of much larger treebanks , and enables them to be used effectively in a variety of scientific and engineering tasks .

The problem of representing and querying linguistic annotations has been an active area of research for several years .

Much of the work has grown from efforts to curate large databases of annotated text such as treebanks , for use in developing and testing language technologies ( Marcus et al . , 1993 ;

Abeillé , 2003 ; Hockenmaier and Steedman , 2007 ) .

At least a dozen linguistic tree query languages have been developed for interrogating treebanks ( see § 2 ) .

While high quality syntactic parsers are able to efficiently annotate large quantities of English text ( Clark and Curran , 2007 ) , existing approaches to query do not work on the same scale .

Many existing systems load the entire corpus into memory and check a user-supplied query against every tree .

Others avoid the memory limitation , and use relational or XML database systems .

Although these have built-in support for indexes , they do not scale up either ( Ghodke and Bird , 2008 ; Zhang et al . , 2001 ) ) .

The ability to interrogate large collections of parsed text has important practical applications .

First , it opens the way to a new kind of information retrieval ( IR ) that is sensitive to syntactic information , permitting users to do more focussed search .

At the simplest level , an ambiguous query term like wind or park could be disambiguated with the help of a POS tag ( e.g. wind / N , park / V ) .

( Existing IR engines already support query with part-of-speech tags ( Chowdhury and McCabe , 1998 ) ) .

More complex queries could stipulate the syntactic category of apple is in subject position .

A second benefit of large scale tree query is for natural language processing .

For example , we might compute the likelihood that a given noun appears as the agent or patient of a verb , as a measure of animacy .

We can use features derived from syntactic trees in order to support semantic role labeling , language modeling , and information extraction ( Chen and Rambow , 2003 ; Collins et al . , 2005 ; Hakenberg et al . , 2009 ) .

A further benefit for natural language processing , though not yet realized , is for a treebank and query engine to provide the underlying storage and retrieval for a variety of linguistic applications .

Just as a relational database is present in most business applications , providing reliable and efficient access to relational data , such a system would act as a repository of annotated texts , and expose an expressive API to client applications .

A third benefit of large scale tree query is to support syntactic investigations , e.g. for develop-ing syntactic theories or preparing materials for language learners .

Published treebanks will usually not attest particular words in the context of some infrequent construction , to the detriment of syntactic studies that make predictions about such combinations , and language learners wanting to see instances of some construction involving words from some specialized topic .

A much larger treebank alleviates these problems .

To improve recall performance , multiple parses for a given sentence could be stored ( possibly derived from different parsers ) .

A fourth benefit for large scale tree query is to support the curation of treebanks , a major enterprise in its own right ( Abeillé , 2003 ) .

Manual selection and correction of automatically generated parse trees is a substantial part of the task of preparing a treebank .

At the point of making such decisions , it is often helpful for an annotator to view existing annotations of a given construction which have already been manually validated ( Hiroshi et al . , 2005 ) .

Occasionally , an earlier annotation decision may need to be reconsidered in the light of new examples , leading to further queries and to corrections that are spread across the whole corpus ( Wallis , 2003 ; Xue et al . , 2005 ) .

This paper explores a new methods for scaling up tree query using an IR engine .

In § 2 we describe existing tree query systems , elaborating on the design decisions , and on key aspects of their implementation and performance .

In § 3 we describe a method for indexing trees using an IR engine , and discuss the details of our open source implementation .

In § 4 we report results from a variety of experiments involving two data collections .

The first collection contains of 5.5 million parsed trees , two orders of magnitude larger than those used by existing tree query systems , while the second collection contains 26.5 million trees .

We have shown how an IR engine can be used to build a high performance tree query system .

It outperforms existing approaches using indexless inmemory search , or custom indexes , or relational database systems , or XML database systems .

We reported the results of a variety of experiments to demonstrate the efficiency of query for a variety of query types on two treebanks consisting of around 5 and 26 million sentences , more than two orders of magnitude larger than what existing systems support .

The approach is quite general , and not limited to particular treebank formats or query languages .

This work suggests that web-scale tree query may soon be feasible .

This opens the door to some interesting possibilities : augmenting web search with syntactic constraints , the ability discover rare examples of particular syntactic constructions , and as a technique for garnering better statistics and more sensitive features for the purpose of constructing language models .


We present a generative model to map natural language questions into SQL queries .

Existing neural network based approaches typically generate a SQL query wordby-word , however , a large portion of the generated results is incorrect or not executable due to the mismatch between question words and table contents .

Our approach addresses this problem by considering the structure of table and the syntax of SQL language .

The quality of the generated SQL query is significantly improved through ( 1 ) learning to replicate content from column names , cells or SQL keywords ; and ( 2 ) improving the generation of WHERE clause by leveraging the column-cell relation .

Experiments are conducted on WikiSQL , a recently released dataset with the largest question-SQL pairs .

Our approach significantly improves the state-of-the-art execution accuracy from 69.0 % to 74.4 % .

We focus on semantic parsing that maps natural language utterances to executable programs ( Zelle and Mooney , 1996 ; Wong and Mooney , 2007 ; Zettlemoyer and Collins , 2007 ; Kwiatkowski et al . , 2011 ; Pasupat and Liang , 2015 ; Iyer et al . , 2017 ; Iyyer et al . , 2017 ) .

In this work , we regard SQL as the programming language , which could be executed on a table or a relational database to obtain an outcome .

Datasets are the main driver of progress for statistical approaches in semantic parsing ( Liang , 2016 ) .

Recently , Zhong et al . ( 2017 ) release WikiSQL , the largest handannotated semantic parsing dataset which is an order of magnitude larger than other datasets in terms of both the number of logical forms and the number of tables .

Pointer network ( Vinyals et al . , 2015 ) based approach is developed , which generates a SQL query word-by-word through replicating from a word sequence consisting of question words , column names and SQL keywords .

However , a large portion of generated results is incorrect or not executable due to the mismatch between question words and column names ( or cells ) .

This also reflects the real scenario where users do not always use exactly the same column name or cell content to express the question .

To address the aforementioned problem , we present a generative semantic parser that considers the structure of table and the syntax of SQL language .

The approach is partly inspired by the success of structure / grammar driven neural network approaches in semantic parsing ( Xiao et al . , 2016 ; Krishnamurthy et al . , 2017 ) .

Our approach is based on pointer networks , which encodes the question into continuous vectors , and synthesizes the SQL query with three channels .

The model learns when to generate a column name , a cell or a SQL keyword .

We further incorporate columncell relation to mitigate the ill-formed outcomes .

Results show that our approach outperforms existing systems , improving state-of-the-art execution accuracy to 74.4 % and logical form accuracy to 60.7 % .

An extensive analysis reveals the advantages and limitations of our approach .

In this work , we develop STAMP , a Syntax-and Table-Aware seMantic Parser that automatically maps natural language questions to SQL queries , which could be executed on web table or relational dataset to get the answer .

STAMP has three channels , and it learns to switch to which channel at each time step .

STAMP considers cell information and the relation between cell and column name in the generation process .

Experiments are conducted on the WikiSQL dataset .

Results show that STAMP achieves the new state-of-the-art performance on WikiSQL .

We conduct extensive experiment analysis to show advantages and limitations of our approach , and where is the room for others to make further improvements .

SQL language has more complicated queries than the cases included in the WikiSQL dataset , including ( 1 ) querying over multiple relational databases , ( 2 ) nested SQL query as condition value , ( 3 ) more operations such as " group by " and " order by " , etc .

In this work , the STAMP model is not designed for the first and second cases , but it could be easily adapted to the third case through incorporating additional SQL keywords and of course the learning of which requires dataset of the same type .

In the future , we plan to improve the accuracy of the column prediction component .

We also plan to build a large-scale dataset that considers more sophisticated SQL queries .

We also plan to extend the approach to low-resource scenarios ( Feng et al . , 2018 ) .


Inclusions from other languages can be a significant source of errors for monolingual parsers .

We show this for English inclusions , which are sufficiently frequent to present a problem when parsing German .

We describe an annotation-free approach for accurately detecting such inclusions , and develop two methods for interfacing this approach with a state-of-the-art parser for German .

An evaluation on the TIGER corpus shows that our inclusion entity model achieves a performance gain of 4.3 points in F-score over a baseline of no inclusion detection , and even outperforms a parser with access to gold standard part-of-speech tags .

The status of English as a global language means that English words and phrases are frequently borrowed by other languages , especially in domains such as science and technology , commerce , advertising , and current affairs .

This is an instance of language mixing , whereby inclusions from other languages appear in an otherwise monolingual text .

While the processing of foreign inclusions has received some attention in the text-to-speech ( TTS ) literature ( see Section 2 ) , the natural language processing ( NLP ) community has paid little attention both to the problem of inclusion detection , and to potential applications thereof .

Also the extent to which inclusions pose a problem to existing NLP methods has not been investigated .

In this paper , we address this challenge .

We focus on English inclusions in German text .

Anglicisms and other borrowings from English form by far the most frequent foreign inclusions in German .

In specific domains , up to 6.4 % of the tokens of a German text can be English inclusions .

Even in regular newspaper text as used for many NLP applications , English inclusions can be found in up to 7.4 % of all sentences ( see Section 3 for both figures ) .

Virtually all existing NLP algorithms assume that the input is monolingual , and does not contain foreign inclusions .

It is possible that this is a safe assumption , and inclusions can be dealt with accurately by existing methods , without resorting to specialized mechanisms .

The alternative hypothesis , however , seems more plausible : foreign inclusions pose a problem for existing approaches , and sentences containing them are processed less accurately .

A parser , for example , is likely to have problems with inclusions -most of the time , they are unknown words , and as they originate from another language , standard methods for unknown words guessing ( suffix stripping , etc . )

Furthermore , the fact that inclusions are often multiword expressions ( e.g. , named entities ) means that simply part-of-speech ( POS ) tagging them accurately is not sufficient : if the parser posits a phrase boundary within an inclusion this is likely to severely decrease parsing accuracy .

In this paper , we focus on the impact of English inclusions on the parsing of German text .

We describe an annotation-free method that accurately recognizes English inclusions , and demonstrate that inclusion detection improves the performance of a state-of-the-art parser for German .

We show that the way of interfacing the inclusion detection and the parser is crucial , and propose a method for modifying the underlying probabilistic grammar in order to enable the parser to process inclusions accurately .

We review related work in Section 2 , and present the English inclusion classifier in Section 3 .

Section 4 describes our results on interfacing inclusion detection with parsing , and Section 5 presents an error analysis .

Discussion and conclusion follow in Section 6 .

This paper has argued that English inclusions in German text is an increasingly pervasive instance of language mixing .

Starting with the hypothesis that such inclusions can be a significant source of errors for monolingual parsers , we found evidence that an unmodified state-of-the-art parser for Ger- man performs substantially worse on a set of sentences with English inclusions compared to a set of length-matched sentences randomly sampled from the same corpus .

The lower performance on the inclusion set persisted even when the parser when given gold standard POS tags in the input .

To overcome the poor accuracy of parsing inclusions , we developed two methods for interfacing the parser with an existing annotation-free inclusion detection system .

The first method restricts the POS tags for inclusions that the parser can assign to those found in the data .

The second method applies tree transformations to ensure that inclusions are treated as phrases .

An evaluation on the TIGER corpus shows that the second method yields a performance gain of 4.3 in F-score over a baseline of no inclusion detection , and even outperforms a model involving perfect POS tagging of inclusions .

To summarize , we have shown that foreign inclusions present a problem for a monolingual parser .

We also demonstrated that it is insufficient to know where inclusions are or even what their parts of speech are .

Parsing performance only improves if the parser also has knowledge about the structure of the inclusions .

It is particularly important to know when adjacent foreign words are likely to be part of the same phrase .

As our error analysis showed , this prevents cascading errors further up in the parse tree .

Finally , our results indicate that future work could improve parsing performance for inclusions further : we found that parsing the inclusion set is still harder than parsing a randomly sampled test set , even for our best-performing model .

This provides an upper bound on the performance we can expect from a parser that uses inclusion detection .

Future work will also involve determining the English inclusion classifier 's merit when applied to rule-based parsing .


The brain is the locus of our language ability , and so brain images can be used to ground linguistic theories .

Here we introduce Brain-Bench , a lightweight system for testing distributional models of word semantics .

We compare the performance of several models , and show that the performance on brain-image tasks differs from the performance on behavioral tasks .

We release our benchmark test as part of a web service .

There is active debate over how we should test semantic models .

In fact , in 2016 there was an entire workshop dedicated to the testing of semantic representations ( RepEval , 2016 ) .

Several before us have argued for the usage of brain data to test semantic models ( Anderson et al . , 2013 ; Murphy et al . , 2012 ; Anderson et al . , 2015 ) , as a brain image represents a snapshot of one person 's own semantic representation .

Still , testing semantic models against brain imaging data is rarely done by those not intimately involved in psycholinguistics or neurolinguistics .

This may be due to a lack of familiarity with neuroimaging methods and publicly available datasets .

We present the first iteration of BrainBench , a new system that makes it easy to test semantic models using brain imaging data ( Available at http://www.langlearnlab.cs.uvic .

Our system has methodology that is similar to popular tests based on behavioral * Corresponding Author data ( see Section 2.2 ) , and has the additional benefit of being fast enough to offer as a web service .

We have presented our new system , BrainBench , which is a fast and lightweight alternative to previous methods for comparing DS models to brain images .

Our proposed methodology is more similar to well-known behavioral tasks , as BrainBench also uses the similarity of words as a proxy for meaning .

We hope that this contribution will bring brain imaging tests " to the masses " and encourage discussion around the testing of DS models against brain imaging data .


In recent years tree kernels have been proposed for the automatic learning of natural language applications .

Unfortunately , they show ( a ) an inherent super linear complexity and ( b ) a lower accuracy than traditional attribute / value methods .

In this paper , we show that tree kernels are very helpful in the processing of natural language as ( a ) we provide a simple algorithm to compute tree kernels in linear average running time and ( b ) our study on the classification properties of diverse tree kernels show that kernel combinations always improve the traditional methods .

Experiments with Support Vector Machines on the predicate argument classification task provide empirical support to our thesis .

In recent years tree kernels have been shown to be interesting approaches for the modeling of syntactic information in natural language tasks , e.g. syntactic parsing ( Collins and Duffy , 2002 ) , relation extraction ( Zelenko et al . , 2003 ) ,

Named Entity recognition ( Cumby and Roth , 2003 ;

Culotta and Sorensen , 2004 ) and Semantic Parsing ( Moschitti , 2004 ) .

The main tree kernel advantage is the possibility to generate a high number of syntactic features and let the learning algorithm to select those most relevant for a specific application .

In contrast , their major drawback are ( a ) the computational time complexity which is superlinear in the number of tree nodes and ( b ) the accuracy that they produce is often lower than the one provided by linear models on manually designed features .

To solve problem ( a ) , a linear complexity algorithm for the subtree ( ST ) kernel computation , was designed in ( Vishwanathan and Smola , 2002 ) .

Unfortunately , the ST set is rather poorer than the one generated by the subset tree ( SST ) kernel designed in ( Collins and Duffy , 2002 ) .

Intuitively , an ST rooted in a node n of the target tree always contains all n 's descendants until the leaves .

This does not hold for the SSTs whose leaves can be internal nodes .

To solve the problem ( b ) , a study on different tree substructure spaces should be carried out to derive the tree kernel that provide the highest accuracy .

On the one hand , SSTs provide learning algorithms with richer information which may be critical to capture syntactic properties of parse trees as shown , for example , in ( Zelenko et al . , 2003 ; Moschitti , 2004 ) .

On the other hand , if the SST space contains too many irrelevant features , overfitting may occur and decrease the classification accuracy ( Cumby and Roth , 2003 ) .

As a consequence , the fewer features of the ST approach may be more appropriate .

In this paper , we aim to solve the above problems .

We present ( a ) an algorithm for the evaluation of the ST and SST kernels which runs in linear average time and ( b ) a study of the impact of diverse tree kernels on the accuracy of Support Vector Machines ( SVMs ) .

Our fast algorithm computes the kernels between two syntactic parse trees in O(m

+ n ) average time , where m and n are the number of nodes in the two trees .

This low complexity allows SVMs to carry out experiments on hundreds of thousands of training instances since it is not higher than the complexity of the polynomial ker-nel , widely used on large experimentation e.g. ( Pradhan et al . , 2004 ) .

To confirm such hypothesis , we measured the impact of the algorithm on the time required by SVMs for the learning of about 122,774 predicate argument examples annotated in PropBank ( Kingsbury and Palmer , 2002 ) and 37,948 instances annotated in FrameNet ( Fillmore , 1982 ) .

Regarding the classification properties , we studied the argument labeling accuracy of ST and SST kernels and their combinations with the standard features ( Gildea and Jurafsky , 2002 ) .

The results show that , on both PropBank and FrameNet datasets , the SST-based kernel , i.e. the richest in terms of substructures , produces the highest SVM accuracy .

When SSTs are combined with the manual designed features , we always obtain the best figure classifier .

This suggests that the many fragments included in the SST space are relevant and , since their manual design may be problematic ( requiring a higher programming effort and deeper knowledge of the linguistic phenomenon ) , tree kernels provide a remarkable help in feature engineering .

In the remainder of this paper , Section 2 describes the parse tree kernels and our fast algorithm .

Section 3 introduces the predicate argument classification problem and its solution .

Section 4 shows the comparative performance in term of the execution time and accuracy .

Finally , Section 5 discusses the related work whereas Section 6 summarizes the conclusions .

In this paper , we have shown that tree kernels can effectively be adopted in practical natural language applications .

The main arguments against their use are their efficiency and accuracy lower than traditional feature based approaches .

We have shown that a fast algorithm ( FTK ) can evaluate tree kernels in a linear average running time and also that the overall converging time required by SVMs is compatible with very large data sets .

Regarding the accuracy , the experiments with Support Vector Machines on the PropBank and FrameNet predicate argument structures show that : ( a ) the richer the kernel is in term of substructures ( e.g. SST ) , the higher the accuracy is , ( b ) tree kernels are effective also in case of automatic parse trees and ( c ) as kernel combinations always improve traditional feature models , the best approach is to combine scalar-based and structured based kernels .


Accurate , automatic evaluation of machine translation is critical for system tuning , and evaluating progress in the field .

We proposed a simple unsupervised metric , and additional supervised metrics which rely on contextual word embeddings to encode the translation and reference sentences .

We find that these models rival or surpass all existing metrics in the WMT 2017 sentence-level and systemlevel tracks , and our trained model has a substantially higher correlation with human judgements than all existing metrics on the WMT 2017 to-English sentence level dataset .

Evaluation metrics are a fundamental component of machine translation ( MT ) and other language generation tasks .

The problem of assessing whether a translation is both adequate and coherent is a challenging text analysis problem , which is still unsolved , despite many years of effort by the research community .

Shallow surfacelevel metrics , such as BLEU and TER ( Papineni et al . , 2002 ; Snover et al . , 2006 ) , still predominate in practice , due in part to their reasonable correlation to human judgements , and their being parameter free , making them easily portable to new languages .

In contrast , trained metrics ( Song and Cohn , 2011 ; Stanojevic and Sima'an , 2014 ; Ma et al . , 2017 ; Shimanaka et al . , 2018 ) , which are learned to match human evaluation data , have been shown to result in a large boost in performance .

This paper aims to improve over existing MT evaluation methods , through developing a series of new metrics based on contextual word embeddings ( Peters et al . , 2018 ; Devlin et al . , 2019 ) , a technique which captures rich and portable representations of words in context , which have been shown to provide important signal to many other NLP tasks ( Rajpurkar et al . , 2018 ) .

We propose a simple untrained model that uses off-the-shelf contextual embeddings to compute approximate recall , when comparing a reference to an automatic translation , as well as trained models , including : a recurrent model over reference and translation sequences , incorporating attention ; and the adaptation of an NLI method ( Chen et al . , 2017 ) to MT evaluation .

These approaches , though simple in formulation , are highly effective , and rival or surpass the best approaches from WMT 2017 .

Moreover , we show further improvements in performance when our trained models are learned using noisy crowd-sourced data , i.e. , having single annotations for more instances is better than collecting and aggregating multiple annotations for single instances .

The net result is an approach that is more data efficient than existing methods , while producing substantially better human correlations.1

We show that contextual embeddings are very useful for evaluation , even in simple untrained models , as well as in deeper attention based methods .

When trained on a larger , much noisier range of instances , we demonstrate a substantial improvement over the state of the art .

In future work , we plan to extend these models by using cross-lingual embeddings , and combine information from translation-source interactions as well as translation-reference interactions .

There are also direct applications to Quality Estimation , by using the source instead of the reference .

B System-level results for WMT 17 news and WMT 2016 IT domain


Existing approaches for Chinese zero pronoun resolution typically utilize only syntactical and lexical features while ignoring semantic information .

The fundamental reason is that zero pronouns have no descriptive information , which brings difficulty in explicitly capturing their semantic similarities with antecedents .

Meanwhile , representing zero pronouns is challenging since they are merely gaps that convey no actual content .

In this paper , we address this issue by building a deep memory network that is capable of encoding zero pronouns into vector representations with information obtained from their contexts and potential antecedents .

Consequently , our resolver takes advantage of semantic information by using these continuous distributed representations .

Experiments on the OntoNotes 5.0 dataset show that the proposed memory network could substantially outperform the state-of-the-art systems in various experimental settings .

A zero pronoun ( ZP ) is a gap in a sentence , which refers to an entity that supplies the necessary information for interpreting the gap ( Zhao and Ng , 2007 ) .

A ZP can be either anaphoric if it corefers to one or more preceding noun phrases ( antecedents ) in the associated text , or non-anaphoric if there are no such noun phrases .

Below is an example of ZPs and their antecedents , where " φ " denotes the ZP .

[ 警方 ] 表示 他们 自杀

的 可能性 很高 ， 不 过 φ 1 也 不

排除 φ 2 有 他杀 的 可能 。 *

( [ The police ] said that they are more likely to commit suicide , but φ 1 could not rule out φ 2 the possibility of homicide . )

In this example , the ZP " φ 1 " is an anaphoric ZP that refers to the antecedent " 警方 / The police " while the ZP " φ 2 " is non-anaphoric .

Unlike overt pronouns , ZPs lack grammatical attributes such as gender and number that have been proven to be essential in pronoun resolution ( Chen and Ng , 2014a ) , which makes ZP resolution a more challenging task than overt pronoun resolution .

Automatic Chinese ZP resolution is typically composed of two steps , i.e. , anaphoric zero pronoun ( AZP ) identification that identifies whether a ZP is anaphoric ; and AZP resolution , which determines antecedents for AZPs .

For AZP identification , state-of-the-art resolvers use machine learning algorithms to build AZP classifiers in a supervised manner ( Chen and Ng , 2013 , 2016 ) .

For AZP resolution , literature approaches include unsupervised methods ( Chen and Ng , 2014b , 2015 ) , feature-based supervised models ( Zhao and Ng , 2007 ; Kong and Zhou , 2010 ) , and neural network models ( Chen and Ng , 2016 ) .

Neural network models for AZP resolution are of growing interest for their capacity to learn task-specific representations without extensive feature engineering and to effectively exploit lexical information for ZPs and their candidate antecedents in a more scalable manner than feature-based models .

Despite these advantages , existing supervised approaches ( Zhao and Ng , 2007 ; Chen and Ng , 2013 , 2016 ) for AZP resolution typically utilize only syntactical and lexical information through features .

They overlook semantic information that is regarded as an important factor in the resolution of common noun phrases ( Ng , 2007 ) .

The fundamental reason is that ZPs have no descriptive information , which results in difficulty in calculating semantic similarities and relatedness scores between the ZPs and their antecedents .

Therefore , the proper representations of ZPs are required so as to take advantage of semantic information when resolving ZPs .

However , representing ZPs is challenging because they are merely gaps that convey no actual content .

One straightforward method to address this issue is to represent ZPs with supplemental information provided by some available components , such as contexts and candidate antecedents .

Motivated by Chen and Ng ( 2016 ) who encode a ZP 's lexical contexts by utilizing its preceding word and governing verb , we notice that a ZP 's context can help to describe the ZP itself .

As an example of its usefulness , given the sentence " φ taste spicy " , people may resolve the ZP " φ " to the candidate antecedent " red peppers " , but can hardly regard " my shoes " as its antecedent , because they naturally look at the ZP 's context " taste spicy " to resolve it ( " my shoes " can not " taste spicy " ) .

Meanwhile , considering that the antecedents of a ZP provide the necessary information for interpreting the gap ( ZP ) , it is a natural way to express a ZP by its potential antecedents .

However , only some subsets of candidate antecedents are needed to represent a ZP1 .

To achieve this goal , a desirable solution should be capable of explicitly capturing the importance of each candidate antecedent and using them to build up the representation for the ZP .

In this paper , inspired by the recent success of computational models with attention mechanism and explicit memory ( Sukhbaatar et al . , 2015 ; Tang et al . , 2016 ; Kumar et al . , 2015 ) , we focus on AZP resolution , proposing the zero pronounspecific memory network ( ZPMN ) that is competent for representing a ZP with information obtained from its contexts and candidate antecedents .

These representations provide our system with an ability to take advantage of semantic information when resolving ZPs .

Our ZPMN consists of multiple computational layers with shared parameters .

With the underlying intuition that not all candidate antecedents are equally relevant for representing the ZP , we develop each computational layer as an attention-based model , which first learns the importance of each candidate antecedent and then utilizes this information to calculate the continu-ous distributed representation of the ZP .

The attention weights over candidate antecedents with respect to the ZP 's representation obtained by the last layer are regarded as the ZP coreference classification result .

Given that every component is differentiable , the entire model could be efficiently trained end-to-end with gradient descent .

We evaluate our method on the Chinese portions of the OntoNotes 5.0 corpus by comparing with the baseline systems in different experimental settings .

Results show that our approach significantly outperforms the baseline algorithms and achieves state-of-the-art performance .

In this study , we propose a novel zero pronounspecific memory network that is capable of encoding zero pronouns into the vector representations with supplemental information obtained from their contexts and candidate antecedents .

Consequently , these continuous distributed vectors provide our model with an ability to take advantage of the semantic information when resolving zero pronouns .

We evaluate our method on the Chinese portion of OntoNotes 5.0 dataset and report substantial improvements over the state-ofthe-art systems in various experimental settings .


We present a technique which complements Hidden Markov Models by incorporating some lexicalized states representing syntactically uncommon words .

' Our approach examines the distribution of transitions , selects the uncommon words , and makes lexicalized states for the words .

We perfor'med a part-of-speech tagging experiment on the Brown corpus to evaluate the resultant language model and discovered that this technique improved the tagging accuracy by 0.21 % at the 95 % level of confidence .

Hidden Markov ' Models are widely used for statistical language modelling in various fields , e.g. , part-of-speech tagging or speech recognition ( Rabiner and Juang , 1986 ) .

The models are based on Markov assumptions , which make it possible to view the language prediction as a Markov process .

' In general , we make the firstorder Markov ass'umptions that the current tag is only dependant on the previous tag and that the current word is only dependant on the current tag .

These are very ' strong ' assumptions , so that the first-order Hidden Markov Models have the advantage of drastically reducing the number of its parameters .

On the other hand , the assumptions restrict the model from utilizing enough constraints provided by the local context and the resultant model consults only a single category ' as the contex .

A lot of effort has been devoted in the past to make up for the insufficient contextual information of the first-order probabilistic model .

The second order Hidden Markov Models with " The research underlying this paper was supported t ) 3 " research grants fl'om Korea Science and Engineering Foundation .

appropriate smoothing techniques show better performance than the first order models and is considered a state-of-the-art technique ( Merialdo , 1994 ; Brants , 1996 ) .

The complexity of the model is however relatively very high considering the small improvement of the performance .

Garside describes IDIOMTAG ( Garside et al . , 1987 ) which is a component of a part-ofspeech tagging system named CLAWS .

ID-IOMTAG serves as a front-end to the tagger and modifies some initially assigned tags in order to reduce the amount of ambiguity to be dealt with by the tagger .

IDIOMTAG can look at any combination of words and tags , with or without intervening words .

By using the IDIOMTAG , CLAWS system improved tagging accuracy from 94 % to 96 - 97 % .

However , the manual-intensive process of producing idiom tags is very expensive although IDIOMTAG proved fruitful .

Kupiec ( Kupiec , 1992 ) describes a technique of augmenting the Hidden Markov Models for part-of-speech tagging by the use of networks .

Besides the original states representing each part-of-speech , the network contains additional states to reduce the noun / adjective confusion , and to extend the context for predicting past participles from preceding auxiliary verbs when they are separated by adverbs .

By using these additional states , the tagging system improved the accuracy from 95.7 % to 96.0 % .

However , the additional context is chosen by analyzing the tagging errors manually .

An automatic refining technique for Hidden Markov Models has been proposed by Brants ( Brants , 1996 ) .

It starts with some initial first order Markov Model .

Some states of the model are selected to be split or merged to take into account their predecessors .

As a result , each of new states represents a extended context .

With this technique , Brants reported a performance cquivalent to the second order Hidden Markov Models .

In this paper , we present an automatic refining technique for statistical language models .

First , we examine the distribution of transitions of lexicalized categories .

Next , we break out the uncommon ones from their categories and make new states for them .

All processes are automated and the user has only to determine the extent of the breaking-out .

In this paper , we present a method for complementing the Hidden Markov Models .

With this method , we lexicalize the Hidden Markov Model seletively and automatically by examining the transition distribution of each state relating to certain words .

Experimental results showed that the selective lexicalization improved the tagging accurary from about 95.79 % to about 96.00 % .

Using normal tests for statistical significance we found that the improvement is significant at the 95 % level of confidence .

Tile cost for this imt~rovenmnt is minimal .

The resulting network contains 210 additional lexicalized states which are found automatically .

Moreover , the lexicalization will not decrease the tagging speed 2 , because the lexicalized states and their corresponding original states are exclusive in our lexicalized network , and thus the rate of ambiguity is not increased even if the lexicalized states are included .

Our approach leaves much room for improvement .

We have so far considered only the outgoing transitions from the target states .

As a result , we have discriminated only the words with right-associativity .

We could also discriminate the words with left-associativity by examining the incoming transitions to the state .

Furthermore , we could extend the context by using the second-order context as represented in Figure l(c ) .

We believe that the same technique presented in this paper could be applied to the proposed extensions .


An ideal summarization system should produce summaries that have high content coverage and linguistic quality .

Many state-ofthe-art summarization systems focus on content coverage by extracting content-dense sentences from source articles .

A current research focus is to process these sentences so that they read fluently as a whole .

The current AE-SOP task encourages research on evaluating summaries on content , readability , and overall responsiveness .

In this work , we adapt a machine translation metric to measure content coverage , apply an enhanced discourse coherence model to evaluate summary readability , and combine both in a trained regression model to evaluate overall responsiveness .

The results show significantly improved performance over AESOP 2011 submitted metrics .

Research and development on automatic and manual evaluation of summarization systems have been mainly focused on content coverage ( Lin and Hovy , 2003 ; Nenkova and Passonneau , 2004 ; Hovy et al . , 2006 ; Zhou et al . , 2006 ) .

However , users may still find it difficult to read such high-content coverage summaries as they lack fluency .

To promote research on automatic evaluation of summary readability , the Text Analysis Conference ( TAC ) ( Owczarzak and Dang , 2011 ) introduced a new subtask on readability to its Automatically Evaluating Summaries of Peers ( AESOP ) task .

Most of the state-of-the-art summarization systems ( Ng et al . , 2011 ; Zhang et al . , 2011 ; Conroy et al . , 2011 ) are extraction-based .

They extract the most content-dense sentences from source articles .

If no post-processing is performed to the generated summaries , the presentation of the extracted sentences may confuse readers .

Knott ( 1996 ) argued that when the sentences of a text are randomly ordered , the text becomes difficult to understand , as its discourse structure is disturbed .

Lin et al . ( 2011 ) validated this argument by using a trained model to differentiate an original text from a randomlyordered permutation of its sentences by looking at their discourse structures .

This prior work leads us to believe that we can apply such discourse models to evaluate the readability of extract-based summaries .

We will discuss the application of Lin et al . 's discourse coherence model to evaluate readability of machine generated summaries .

We also introduce two new feature sources to enhance the model with hierarchical and Explicit / Non-Explicit information , and demonstrate that they improve the original model .

There are parallels between evaluations of machine translation ( MT ) and summarization with respect to textual content .

For instance , the widely used ROUGE ( Lin and Hovy , 2003 ) metrics are influenced by BLEU ( Papineni et al . , 2002 ) : both look at surface n-gram overlap for content coverage .

Motivated by this , we will adapt a state-of-theart , linear programming-based MT evaluation metric , TESLA ( Liu et al . , 2010 ) , to evaluate the content coverage of summaries .

TAC 's overall responsiveness metric evaluates the quality of a summary with regard to both its content and readability .

Given this , we combine our two component coherence and content models into an SVM-trained regression model as our surrogate to overall responsiveness .

Our experiments show that the coherence model significantly outperforms all AESOP 2011 submissions on both initial and update tasks , while the adapted MT evaluation metric and the combined model significantly outperform all submissions on the initial task .

To the best of our knowledge , this is the first work that applies a discourse coherence model to measure the readability of summaries in the AESOP task .

2 Related Work Nenkova and Passonneau ( 2004 ) proposed a manual evaluation method that was based on the idea that there is no single best model summary for a collection of documents .

Human annotators construct a pyramid to capture important Summarization Content Units ( SCUs ) and their weights , which is used to evaluate machine generated summaries .

Lin and Hovy ( 2003 ) introduced an automatic summarization evaluation metric , called ROUGE , which was motivated by the MT evaluation metric , BLEU ( Papineni et al . , 2002 ) .

It automatically determines the content quality of a summary by comparing it to the model summaries and counting the overlapping n-gram units .

Two configurations -ROUGE-2 , which counts bigram overlaps , and ROUGE-SU4 , which counts unigram and bigram overlaps in a word window of four -have been found to correlate well with human evaluations .

Hovy et al . ( 2006 ) pointed out that automated methods such as ROUGE , which match fixed length n-grams , face two problems of tuning the appropriate fragment lengths and matching them properly .

They introduced an evaluation method that makes use of small units of content , called Basic Elements ( BEs ) .

Their method automatically segments a text into BEs , matches similar BEs , and finally scores them .

Both ROUGE and BE have been implemented and included in the ROUGE / BE evaluation toolkit 1 , which has been used as the default evaluation tool in the summarization track in the Document Un-1 http://berouge.com / default.aspx derstanding Conference ( DUC ) and Text Analysis Conference ( TAC ) .

DUC and TAC also manually evaluated machine generated summaries by adopting the Pyramid method .

Besides evaluating with ROUGE / BE and Pyramid , DUC and TAC also asked human judges to score every candidate summary with regard to its content , readability , and overall responsiveness .

DUC and TAC defined linguistic quality to cover several aspects : grammaticality , non-redundancy , referential clarity , focus , and structure / coherence .

Recently , Pitler et al . ( 2010 ) conducted experiments on various metrics designed to capture these aspects .

Their experimental results on DUC 2006 and 2007 show that grammaticality can be measured by a set of syntactic features , while the last three aspects are best evaluated by local coherence .

Conroy and Dang ( 2008 ) combined two manual linguistic scores -grammaticality and focus -with various ROUGE / BE metrics , and showed this helps better predict the responsiveness of the summarizers .

Since 2009 , TAC introduced the task of Automatically Evaluating Summaries of Peers ( AESOP ) .

AESOP 2009 and 2010 focused on two summary qualities : content and overall responsiveness .

Summary content is measured by comparing the output of an automatic metric with the manual Pyramid score .

Overall responsiveness measures a combination of content and linguistic quality .

In AESOP 2011 ( Owczarzak and Dang , 2011 ) , automatic metrics are also evaluated for their ability to assess summary readability , i.e. , to measure how linguistically readable a machine generated summary is .

Submitted metrics that perform consistently well on the three aspects include Giannakopoulos and Karkaletsis ( 2011 ) , Conroy et al . ( 2011 ) , and de Oliveira ( 2011 ) .

Giannakopoulos and Karkaletsis ( 2011 ) created two character-based n-gram graph representations for both the model and candidate summaries , and applied graph matching algorithm to assess their similarity .

Conroy et al . ( 2011 ) extended the model in ( Conroy and Dang , 2008 ) to include shallow linguistic features such as term overlap , redundancy , and term and sentence entropy .

de Oliveira ( 2011 ) modeled the similarity between the model and candidate summaries as a maximum bipartite matching problem , where the two summaries are represented as two sets of nodes and precision and recall are cal- culated from the matched edges .

However , none of the AESOP metrics currently apply deep linguistic analysis , which includes discourse analysis .

Motivated by the parallels between summarization and MT evaluation , we will adapt a state-ofthe-art MT evaluation metric to measure summary content quality .

To apply deep linguistic analysis , we also enhance an existing discourse coherence model to evaluate summary readability .

We focus on metrics that measure the average quality of machine summarizers , i.e. , metrics that can rank a set of machine summarizers correctly ( human summarizers are not included in the list ) .

We proposed TESLA-S by adapting an MT evaluation metric to measure summary content coverage , and introduced DICOMER by applying a dis- course coherence model with newly introduced features to evaluate summary readability .

We combined these two metrics in the CREMER metric -an SVM-trained regression model -for automatic summarization overall responsiveness evaluation .

Experimental results on AESOP 2011 show that DICOMER significantly outperforms all submitted metrics on both initial and update tasks with large gaps , while TESLA-S and CREMER significantly outperform all metrics on the initial task.3


In this paper , we define the task of gender rewriting in contexts involving two users ( I and/or You ) -first and second grammatical persons with independent grammatical gender preferences .

We focus on Arabic , a gendermarking morphologically rich language .

We develop a multi-step system that combines the positive aspects of both rule-based and neural rewriting models .

Our results successfully demonstrate the viability of this approach on a recently created corpus for Arabic gender rewriting , achieving 88.42 M 2 F 0.5 on a blind test set .

Our proposed system improves over previous work on the first-person-only version of this task , by 3.05 absolute increase in M 2 F 0.5 .

We demonstrate a use case of our gender rewriting system by using it to post-edit the output of a commercial MT system to provide personalized outputs based on the users ' grammatical gender preferences .

We make our code , data , and pretrained models publicly available .

Gender bias is a fundamental problem in natural language processing ( NLP ) and it has been receiving an increasing attention across a variety of core tasks such as machine translation ( MT ) , co-reference resolution , and dialogue systems .

Research has shown that NLP systems have the ability to embed and amplify gender bias ( Sun et al . , 2019 ) , which not only degrades users ' experiences but also creates representational harm ( Blodgett et al . , 2020 ) .

The embedded bias within NLP systems is usually attributed to training models on biased data that reflects the social inequalities of the world we live in .

However , even the most balanced of models can still exhibit and amplify bias if they are designed to produce a single text output without taking their users ' gender preferences into consideration .

Therefore , to provide the correct user-aware output , NLP systems should be designed to produce outputs that are as gender-specific as the users information they have access to .

Users information could be either embedded as part of the input or provided externally by the users themselves .

In cases where this information is unavailable to the system , generating all gender-specific forms or a gender-neutral form is more appropriate .

Producing user-aware outputs becomes more challenging for systems targeting multi-user contexts ( first and second persons , with independent grammatical gender preferences ) , particularly when dealing with gender-marking morphologically rich languages .

In this paper , we define the task of gender rewriting in contexts involving two users ( I and/or You ) -first and second grammatical persons with independent grammatical gender preferences and we focus on Arabic , a gender-marking morphologically rich language .

The main contributions of our work are as follows :

We introduce a multi-step gender rewriting system that combines the positive aspects of rule-based and neural models .

We demonstrate our approach 's effectiveness by establishing a strong benchmark on a publicly available multi-user Arabic gender rewriting corpus .

We show that our best system yields state-ofthe-art results on the first-person-only version of this task , beating previous work .

We demonstrate a use case of our system by post-editing the output of an MT system to match users ' grammatical gender preferences .

We first discuss related work ( § 2 ) as well as relevant Arabic linguistic facts ( § 3 ) .

We then define the gender rewriting task in § 4 and describe the data we use and the gender rewriting model we build in § 5 and § 6 .

Lastly , we present our experimental setup ( § 7 ) and results ( § 8) and conclude in § 9 .

We defined the task of gender rewriting in contexts involving two users ( I and/or You ) , and developed a multi-step system that combines the positive aspects of both rule-based and neural rewriting models .

Our best models establish the benchmark for this newly defined task and the SOTA for a previously defined first person version of it .

We further demonstrated a use case of our gender rewriting system by post-editing the output of a commercial MT system to provide personalized outputs based on the users ' grammatical gender preferences .

In future work , we plan to explore the use of other pretrained models , and to work on the problem of gender rewriting in other languages and dialectal varieties .


The de-facto standard decoding method for semantic parsing in recent years has been to autoregressively decode the abstract syntax tree of the target program using a top-down depthfirst traversal .

In this work , we propose an alternative approach : a Semi-autoregressive Bottom-up Parser ( SMBOP ) that constructs at decoding step t the top-K sub-trees of height ≤ t. Our parser enjoys several benefits compared to top-down autoregressive parsing .

From an efficiency perspective , bottom-up parsing allows to decode all sub-trees of a certain height in parallel , leading to logarithmic runtime complexity rather than linear .

From a modeling perspective , a bottom-up parser learns representations for meaningful semantic sub-programs at each step , rather than for semantically-vacuous partial trees .

We apply SMBOP on SPIDER , a challenging zero-shot semantic parsing benchmark , and show that SMBOP leads to a 2.2x speed-up in decoding time and a ∼5x speed-up in training time , compared to a semantic parser that uses autoregressive decoding .

SMBOP obtains 71.1 denotation accuracy on SPIDER , establishing a new state-of-the-art , and 69.5 exact match , comparable to the 69.6 exact match of the autoregressive RAT-SQL+GRAPPA .

Semantic parsing , the task of mapping natural language utterances into programs ( Zelle and Mooney , 1996 ; Zettlemoyer and Collins , 2005 ; Clarke et al . ; Liang et al . , 2011 ) , has converged in recent years on a standard encoder-decoder architecture .

Recently , meaningful advances emerged on the encoder side , including developments in Transformer-based architectures ( Wang et al . , 2020a ) and new pretraining techniques ( Yin et al . , 2020 ; Herzig et al . , 2020 ; Yu et al . , 2020 ; Deng et al . , 2020 ; Shi et al . , 2021 ) .

Conversely , the decoder has remained roughly constant for years , where the abstract syntax tree of the target program is autoregressively decoded in a top-down manner ( Yin and Neubig , 2017 ; Krishnamurthy et al . , 2017 ; Rabinovich et al . , 2017 ) .

Bottom-up decoding in semantic parsing has received little attention ( Cheng et al . , 2019 ; Odena et al . , 2020 ) .

In this work , we propose a bottom-up semantic parser , and demonstrate that equipped with recent developments in Transformer-based ( Vaswani et al . , 2017 ) architectures , it offers several advantages .

From an efficiency perspective , bottom-up parsing can naturally be done semiautoregressively : at each decoding step t , the parser generates in parallel the top-K program sub-trees of depth ≤ t ( akin to beam search ) .

This leads to runtime complexity that is logarithmic in the tree size , rather than linear , contributing to the rocketing interest in efficient and greener artificial intelligence technologies ( Schwartz et al . , 2020 ) .

From a modeling perspective , neural bottom-up parsing provides learned representations for meaningful ( and executable ) sub-programs , which are sub-trees computed during the search procedure , in contrast to top-down parsing , where hidden states represent partial trees without clear semantics .

Figure 1 illustrates a single decoding step of our parser .

Given a beam Z t with K = 4 trees of height t ( blue vectors ) , we use cross-attention to contextualize the trees with information from the input question ( orange ) .

Then , we score the frontier , that is , the set of all trees of height t + 1 that can be constructed using a grammar from the current beam , and the top-K trees are kept ( purple ) .

Last , a representation for each of the new K trees is generated and placed in the new beam Z t+1 .

After T decoding steps , the parser returns the highest-scoring tree in Z T that corresponds to a full program .

Because we have gold trees at training time , the entire model is trained jointly using maximum likelihood .

We evaluate our model , SMBOP1 ( SeMiautoregressive Bottom-up semantic Parser ) , on SPI- What are the names of actors over 60 ?

In this work we present the first semiautoregressive bottom-up semantic parser that enjoys logarithmic theoretical runtime , and show that it leads to a 2.2x speed-up in decoding and ∼5x faster taining , while maintaining state-of-the-art performance .

Our work shows that bottom-up parsing , where the model learns representations for semantically meaningful sub-trees is a promising research direction , that can contribute in the future to setups such as contextual semantic parsing , where sub-trees often repeat , and can enjoy the benefits of execution at training time .

Future work can also leverage work on learning tree representations ( Shiv and Quirk , 2019 ) to further improve parser performance .


Recent advances in language and vision push forward the research of captioning a single image to describing visual differences between image pairs .

and I 2 , and the task is to generate a description W 1,2 comparing them , existing methods directly model ⟨I 1 , I 2 ⟩ → W 1,2 mapping without the semantic understanding of individuals .

In this paper , we introduce a Learningto-Compare ( L2C ) model , which learns to understand the semantic structures of these two images and compare them while learning to describe each one .

We demonstrate that L2C benefits from a comparison between explicit semantic representations and singleimage captions , and generalizes better on the new testing image pairs .

It outperforms the baseline on both automatic evaluation and human evaluation for the Birds-to-Words dataset .

The task of generating textual descriptions of images tests a machine 's ability to understand visual data and interpret it in natural language .

It is a fundamental research problem lying at the intersection of natural language processing , computer vision , and cognitive science .

For example , single-image captioning ( Farhadi et al . , 2010 ; Kulkarni et al . , 2013 ; Vinyals et al . , 2015 ; Xu et al . , 2015 ) has been extensively studied .

Recently , a new intriguing task , visual comparison , along with several benchmarks ( Jhamtani and Berg-Kirkpatrick , 2018 ; Tan et al . , 2019 ; Park et al . , 2019 ; Forbes et al . , 2019 ) has drawn increasing attention in the community .

To complete the task and generate comparative descriptions , a machine should understand the visual differences between a pair of images ( see Figure 1 ) .

Previous methods ( Jhamtani and Berg-Kirkpatrick , 2018 ) as the ResNet features ( He et al . , 2016 ) as a whole , and build end-to-end neural networks to predict the description of visual comparison directly .

In contrast , humans can easily reason about the visual components of a single image and describe the visual differences between two images based on their semantic understanding of each one .

Humans do not need to look at thousands of image pairs to describe the difference of new image pairs , as they can leverage their understanding of single images for visual comparison .

Therefore , we believe that visual differences should be learned by understanding and comparing every single image 's semantic representation .

A most recent work ( Zhang et al . , 2020 ) conceptually supports this argument , where they show that low-level ResNet visual features lead to poor generalization in vision-and-language navigation , and high-level semantic segmentation helps the agent Image I 1

Image I 2 V 1 V 2 Relation- enhanced features V g 1 V g 2 LSTM h t

In this paper , we present a learning-to-compare framework for generating visual comparisons .

Our segmentation encoder with semantic pooling and graph reasoning could construct structured image representations .

We also show that learning to describe visual differences benefits from understanding the semantics of each image .