We present a game-theoretic model of bargaining over a metaphor in the context of political communication , find its equilibrium , and use it to rationalize observed linguistic behavior .

We argue that game theory is well suited for modeling discourse as a dynamic resulting from a number of conflicting pressures , and suggest applications of interest to computational linguists .

A 13 Dec 1992 article in The Times starts thus :

The European train chugged out of the station last night ; for most of the day it looked as if it might be stalled there for some time .

It managed to pull away at around 10:30 pm only after the Spanish prime minister , Felipe Gonzalez , forced the passengers in the first class carriages into a last minute whip round to sweeten the trip for the European Community 's poor four : Spain , Portugal , Greece and Ireland .

The fat controller , Helmut Kohl , beamed with satisfaction as the deal was done .

The elegantlysuited Francois Mitterrand was equally satisfied .

But nobody was as pleased as John Major , stationmaster for the UK presidency , for whom the agreement marked a scarce high point in a battered premiership .

The departure had actually been delayed by seven months by Danes on the line .

Just when that problem was solved , there was the voluble outbreak , orchestrated by Spain , from the poor four passengers demanding that they should travel free and be given spending money , too .

The coupling of the carriages may not be reliably secure but the pan-European express is in motion .

That few seem to agree the destination suggests that future arguments are inevitable at every set of points .

Apart from an entertaining read , the extended metaphor provides an elaborate conceptual correspondence between a familiar domain of train journeys and the unfolding process of European integration .

Carriages are likened to nation states ; passengers to their peoples ; treaties to stations ; politicians to responsible rail company employees .

In a compact form , the metaphor gives expression to both the small and the large scale of the process .

Denmark 's failure to ratify the 1992 Maastricht treaty until opt-outs were negotiated later that year is compared to dissenters sabotaging the journey by laying on the tracks ( Danes on the line ) ; negotiations over the Cohesion Fund that would provide less developed regions with financial aid to help them comply with convergence criteria are likened to second class carriages with poor passengers for whom the journey had to be subsidized .

At a more general level , the European integration is a purposeful movement towards some destination according to a worked out plan , getting safely through negotiation and implementation from one treaty to another , as a train moving on its rails through subsequent stations , with each nation being separate yet tied with everyone else .

Numerous inferences regarding speed , timetables , stations , passengers , different classes of tickets , temporary obstacles on the tracks , and so on can be made by the reader based on the knowledge of train journeys , giving him or her a feeling of an enhanced understanding1 of the highly complex process of European integration .

So apt was the metaphor that political fights were waged over its details ( Musolff , 2000 ) .

Worries about destination were given an eloquent expression by Margaret Thatcher ( Sunday Times , 20 Sept 1992 ): She warned EC leaders to stop their endless round of summits and take notice of their own people .

" There is a fear that the European train will thunder forward , laden with its customary cargo of gravy , towards a destination neither wished for nor understood by electorates .

But the train can be stopped , " she said .

The metaphor proved flexible enough for further elaboration .

John Major , a Conservative PM of Britain , spoke on June 1st , 1994 about his vision of the decision making at the EU level , saying that he had never believed that Europe must act as one on every issue , and advocating " a sensible new approach , varying when it needs to , multitrack , multi-speed , multi-layered . "

He attempted to turn a largely negative Conservative take on the European train ( see Thatcher above ) into a tenable positive vision -each nation-carriage is now presumably a rather autonomous entity , waiting on a side track for the right locomotive , in a huge yet smoothly operating railroad system .

Major 's political opponents offered their counter-frames .

In both cases , the imagery of a large transportation system was taken up , yet turned around to suggest that " multi , for everyone " amounts to Britain being in " the slow lane , " and a different image was suggested that makes the negative evaluation of Britain 's opt-outs more poignant -a football metaphor , where relegation to the second division is a sign of a weak performance , and a school metaphor , where Britain is portrayed as an under-achiever :

John Cunningham , Labour He has admitted that his Government would let Britain fall behind in Europe .

He is apparently willing to offer voluntary relegation to the second division in Europe , and he is n't even prepared to put up a fight .

I believe that in any two-speed Europe , Britain must be up with those in the fast lane .

Paddy Ashdown , Liberal Democrat Are you really saying that the best that Britain can hope for under your leadership is ... the slow lane of a two-speed Europe ?

Most people in this country will want to aim higher , and will reject your view of a ' drop-out ' Britain .

The pro-European camp rallied around the " Britain in the slow lane " version as a critical stance towards the government 's European policy .

Of the alternative metaphors , the school metaphor has some traction in the Euro discourse , where the European ( mainly German ) financial officers are compared to school authorities , and governments struggling to meet the strict convergence criteria to enter the Euro are compared to pupils that barely make the grade with Britain as a ' drop-out ' who gave up even trying ( Musolff , 2000 ) .

The fact that European policy is being communicated and negotiated via a metaphor is not surprising ; after all , " there is always someone willing to help us think by providing us with a metaphor that accords with HIS views .

From the point of view of the dynamics of political discourse , the puzzle is rather the apparent tendency of politicians to be compelled by the rival 's metaphorical framework .

Thatcher tries to turn the train metaphor used by the pro-EU camp around .

Yet , assuming metaphors are matters of choice , why should Thatcher feel constrained by her rival 's choice , why does n't she ignore it and merely suggest a new metaphor of her own design ?

As the evidence above suggests , this is not Thatcher 's idiosyncrasy , as Major and his rivals acted similarly .

In this article , we use the explanatory framework of game theory , seeking to rationalize the observed behavior by designing a game that would produce , at equilibrium , the observed dynamics .

Specifically , we formalize the notion that the price of " locking " the public into a metaphorical frame of reference is that a politician is coerced into staying within the metaphor as well , even if he or she is at the receiving end of a rival 's rhetorical move .

Since the use of game theory is not common in computational linguistics , we first explain its main attributes , justify our decision to make use of it , and draw connections to research questions that can benefit from its application ( section 2 ) .

Next , we design the game of bargaining over a metaphor , and find its equilibrium ( section 3 ) , followed by a discussion ( section 4 ) .

This article addressed a specific communicative setting ( rival politicians trying to " sell " to the public their versions of the unfolding realities and necessary policies ) and a specific linguistic tool ( an extended metaphor ) , showing that the particular use made of metaphor in such setting can be rationalized based on the characteristics of the setting .

Given the central role played by the public gratification constraint in our model , would conversational situations without the need to persuade the public , such as meetings of small groups of peers or phone conversations between friends , tend less to the use of extended metaphor ?

Conversely , does the use of extended metaphor in other settings testify to the existence of presumed onlookers who need to be " captured " in a particular version of reality -as in pedagogic or poetic context ?

Considerations of the participants ' agendas and their impact on the ensuing dynamics of the exchange would we believe lead to further interest in game theoretic models when addressing complex social dynamics in situations like collaborative authoring , debates , or dating , and will augment the existing mostly statistical approaches with a broader picture of the relevant communication .


We examine a methodology using neural language models ( LMs ) for analyzing the word order of language .

This LM-based method has the potential to overcome the difficulties existing methods face , such as the propagation of preprocessor errors in count-based methods .

In this study , we explore whether the LMbased method is valid for analyzing the word order .

As a case study , this study focuses on Japanese due to its complex and flexible word order .

To validate the LM-based method , we test ( i ) parallels between LMs and human word order preference , and ( ii ) consistency of the results obtained using the LM-based method with previous linguistic studies .

Through our experiments , we tentatively conclude that LMs display sufficient word order knowledge for usage as an analysis tool .

Finally , using the LMbased method , we demonstrate the relationship between the canonical word order and topicalization , which had yet to be analyzed by largescale experiments .

Model Verbs whose type-A examples prefer the ACC-DAT order

Verbs whose type-B examples prefer the ACC-DAT order CLM " 預ける " ( deposit ) , " 置く " ( place ) , " 持つ " ( have ) , " 入 れる " ( put in ) , " 納める " ( pay ) , " 郵送 " ( mail ) , " 供 給 " ( supply ) , " 出す " ( put out ) , " 運ぶ " ( transport ) , " 流 す " ( shed ) , " 掛ける " ( hang ) , " 飾る " ( decorate ) , " 広 げる " ( spread ) , " 移す " ( transfer ) , " 残す " ( leave ) , " 配 送 " ( deliver ) , " 送る " ( send ) , " 投げる " ( throw ) , " 送 付 " ( send ) , " 返却 " ( return ) , " 届ける " ( send ) , " 戻す " ( return ) , " 着ける " ( wear ) , " 上げる " ( increase ) , " 落と す " ( drop ) , " 載せる " ( publish ) , " 変更 " ( change ) , " 納 入 " ( deliver ) , " 卸す " ( unload ) , " 掲載 " ( publish ) , " 通 す " ( get X through )

" 配 布 " ( distribute ) , " 渡 す " ( pass ) , " プ レ ゼ ン ト " ( present ) , " 合わせる " ( match ) , " 見せる " ( show ) , " 提 供 " ( offer ) , " 与える " ( give ) , " 当てる " ( hit ) , " 回す " ( turn ) , " 追加 " ( add ) , " 貸す " ( lend ) , " 展示 " ( exhibit ) , " 据える " ( lay ) , " 依頼 " ( request ) , " 挿入 " ( insert ) , " 纏 める " ( collect ) , " 請求 " ( claim ) SLM " 預ける " ( deposit ) , " 置く " ( place ) , " 頼む " ( ask ) , " 入 れる " ( put in ) , " 納める " ( pay ) , " 郵送 " ( mail ) , " 出 す " ( put out ) , " 運ぶ " ( transport ) , " 流す " ( shed ) , " 掛け る " ( hang ) , " 広げる " ( spread ) , " 移す " ( transfer ) , " 残 す " ( leave ) , " リクエスト " ( request ) , " 配送 " ( deliver ) , " 送る " ( send ) , " 投げる " ( throw ) , " 送付 " ( send ) , " 求 める " ( ask ) , " 提出 " ( submit ) , " 届ける " ( deliver ) , " 要 求 " ( request ) , " 戻す " ( return ) , " 寄付 " ( donate ) , " 寄贈 " ( donation ) , " 着ける " ( wear ) , " 乗せる " ( place ) , " 上げ る " ( increase ) , " 落とす " ( drop ) , " 貼る " ( stick ) , " 分け る " ( divide ) , " ばらまく " ( scatter ) , " はめる " ( fit ) , " 支 払う " ( pay ) , " 配達 " ( deliver ) , " 卸す " ( unload ) , " 纏め る " ( collect ) , " 通す " ( get X through )

" プレゼント " ( present ) , " 持つ " ( have ) , " 合わせる " ( match ) , " 見せる " ( show ) , " 向ける " ( point ) , " 提供 " ( offer ) , " 装備 " ( equip ) , " 追加 " ( add ) , " 展示 " ( exhibit ) , " 据える " ( lay ) , " 採用 " ( adopt )

Speakers sometimes have a range of options for word order in conveying a similar meaning .

A typical case in English is dative alternation :

Even for such a particular alternation , several studies ( Bresnan et al . , 2007 ; Hovav and Levin , 2008 ; Colleman , 2009 ) investigated the factors determining this word order and found that the choice is not random .

For analyzing such linguistic phenomena , linguists repeat the cycle of constructing hypotheses and testing their validity , usually through psychological experiments or count-based methods .

However , these approaches sometimes face difficulties , such as scalability issues in psychological experiments and the propagation of preprocessor errors in count-based methods .

Compared to the typical approaches for evaluating linguistic hypotheses , approaches using LMs have potential advantages ( Section 3.2 ) .

In this study , we examine the methodology of using LMs for analyzing word order ( Figure 1 ) .

To validate the LM-based method , we first examine if there is a parallel between canonical word order and generation probability of LMs for each word order .

Futrell and Levy ( 2019 ) reported that English LMs have human-like word order preferences , which can be one piece of evidence for validating the LM-based method .

However , it is not clear whether the above assumption is valid even in languages with more flexible word order .

In this study , we specifically focus on the Japanese language due to its complex and flexible word order .

There are many claims on the canonical word order of Japanese , and it has attracted considerable attention from linguists and natural language processing ( NLP ) researchers for decades ( Hoji , 1985 ; Saeki , 1998 ; Miyamoto , 2002 ; Matsuoka , 2003 ; Koizumi and Tamaoka , 2004 ; Nakamoto et al . , 2006 ; Shigenaga , 2014 ; Sasano and Okumura , 2016 ; Orita , 2017 ; Asahara et al . , 2018 ) .

We investigated the validity of using Japanese LMs for canonical word order analysis by conducting two sets of experiments : ( i ) comparing word order preference in LMs to that in Japanese speakers ( Section 4 ) , and ( ii ) checking the consistency Topic Time Location Subject ( Adverb ) Indirect object Direct object Verb Notation TOP TIM LOC NOM -DAT ACC -Typical particle " は " ( wa ) " に " ( ni ) " で " ( de ) " が " ( ga ) -"に " ( ni ) " を " ( o ) -Related section 6 5.2 5.2 5.2 5.3 5.1 5.1 5.1

Table 1 : Overview of the typical cases in Japanese , their typical particles , and the sections where the corresponding case is analyzed .

The well-known canonical word order of Japanese is listed from left to right .

between the preference of LMs with previous linguistic studies ( Section 5 ) .

From our experiments , we tentatively conclude that LMs display sufficient word order knowledge for usage as an analysis tool , and further explore potential applications .

Finally , we analyzed the relationship between topicalization and word order of Japanese by taking advantage of the LM-based method ( Section 6 ) .

• Discuss and validate the use of LMs as a tool for word order analysis as well as investigate the sensitivity of LMs against different word orders in non-European language ( Section 3 ) ; • Find encouraging parallels between the results obtained with the LM-based method and those with the previously established method on various hypotheses of canonical word order of Japanese ( Sections 4 and 5 ) ; and • Showcase the advantages of an LM-based method through analyzing linguistic phenomena that is difficult to explore with the previous data-driven methods ( Section 6 ) .

We have proposed to use LMs as a tool for analyzing word order in Japanese .

Our experimental results support the validity of using Japanese LMs for canonical word order analysis , which has the potential to broaden the possibilities of linguistic research .

From an engineering view , this study supports the use of LMs for scoring Japanese word order automatically .

From the viewpoint of the linguistic field , we provide additional empirical evidence to various word order hypotheses as well as demonstrate the validity of the LM-based method .

We plan to further explore the capability of LMs on other linguistic phenomena related to word order , such as " given new ordering " ( Nakagawa , 2016 ; Asahara et al . , 2018 ) .

Since LMs are language-agnostic , analyzing word order in another language with the LM-based method would also be an interesting direction to investigate .

Furthermore , we would like to extend a comparison between machine and human language processing beyond the perspective of word order .


A conventional approach to improving the performance of end-to-end speech translation ( E2E-ST ) models is to leverage the source transcription via pre-training and joint training with automatic speech recognition ( ASR ) and neural machine translation ( NMT ) tasks .

However , since the input modalities are different , it is difficult to leverage source language text successfully .

In this work , we focus on sequencelevel knowledge distillation ( SeqKD ) from external text-based NMT models .

To leverage the full potential of the source language information , we propose backward SeqKD , SeqKD from a target-to-source backward NMT model .

To this end , we train a bilingual E2E-ST model to predict paraphrased transcriptions as an auxiliary task with a single decoder .

The paraphrases are generated from the translations in bitext via back-translation .

We further propose bidirectional SeqKD in which SeqKD from both forward and backward NMT models is combined .

Experimental evaluations on both autoregressive and non-autoregressive models show that SeqKD in each direction consistently improves the translation performance , and the effectiveness is complementary regardless of the model capacity .

End-to-end speech translation ( E2E-ST ) ( Bérard et al . , 2016 ) , which aims to convert source speech to text in another language directly , is an active research area .

Because direct ST is a more difficult task than automatic speech recognition ( ASR ) and machine translation ( MT ) , various techniques have been proposed to ease the training process by using source transcription .

Examples include pretraining ( Bérard et al . , 2018 ; Wang et al . , 2020c ; Bansal et al . , 2019 ; Wang et al . , 2020d ) , multi-task learning ( Weiss et al . , 2017 ; Bérard et al . , 2018 ; Bahar et al . , 2019 ) , knowledge distillation ( Liu et al . , 2019 ) , meta-learning ( Indurthi et al . , 2020 ) , twopass decoding ( Anastasopoulos and Chiang , 2018 ; Sperber et al . , 2019 ) , and interactive decoding ( Liu et al . , 2020 ; Le et al . , 2020 ) .

However , as input modalities between ST and MT tasks are different , an auxiliary MT task is not always helpful , especially when additional bitext is not available ( Bahar et al . , 2019 ) .

Moreover , because monotonic speech-to-transcription alignments encourage the ASR task to see surface-level local information , an auxiliary ASR task helps the E2E-ST model to extract acoustic representations , not semantic ones , from speech .

Sequence-level knowledge distillation ( Se-qKD ) ( Kim and Rush , 2016 ) is another approach to transferring knowledge from one model to another .

Recent studies have shown that SeqKD has the effect of reducing the complexity of training data and thus eases the training of student models , e.g. , non-autoregressive ( NAR ) models ( Gu et al . , 2018 ; Zhou et al . , 2019a ; Ren et al . , 2020 ) .

Paraphrasing , which represents text in a different form but with the same meaning , can also be regarded as SeqKD when using neural paraphrasing via back-translation ( Mallinson et al . , 2017 ; Wieting et al . , 2017 ; Federmann et al . , 2019 ) .

It has been studied to improve the reference diversity for MT system evaluations ( Thompson and Post , 2020 ; Bawden et al . , 2020a , b ) and the performance of low-resource neural MT ( NMT ) models ( Zhou et al . , 2019b ; Khayrallah et al . , 2020 ) .

In this work , due to its simplicity and effectiveness , we focus on SeqKD from text-based NMT models to improve the performance of a bilingual E2E-ST model .

In order to fully leverage source language information , we propose backward Se-qKD , which targets paraphrased source transcriptions generated from a target-to-source backward NMT model as an auxiliary task .

Then , a single ST decoder is trained to predict both source and target language text as in a multilingual setting ( Inaguma et al . , 2019 ) .

This way , the decoder is biased to capture semantic representations from speech , un-like joint training with an auxiliary ASR task .

We also propose bidirectional SeqKD , which combines SeqKD from two NMT models in both language directions .

Therefore , the E2E-ST models can fully exploit the knowledge embedded in both forward and backward NMT models .

Experimental evaluations demonstrate that Se-qKD from each direction consistently improves the translation performance of both autoregressive and non-autoregressive E2E-ST models .

We also confirm that bidirectional SeqKD outperforms unidirectional SeqKD and that the effectiveness is maintained in large models .

To fully leverage knowledge in both source and target language directions for bilingual E2E-ST models , we have proposed bidirectional SeqKD , in which both forward SeqKD from a source-to-target NMT model and backward SeqKD from a target-tosource NMT model are combined .

Backward Se-qKD is performed by targeting source paraphrases generated via back-translation from the original translations in bitext .

Then , the E2E-ST model is enhanced by training to generate both source and target language text with a single decoder .

We experimentally confirmed that SeqKD from each direction boosted the translation performance of both autoregressive and non-autoregressive E2E-ST models , and the effectiveness was additive .

Multi-referenced training with the original and distilled text gave further gains .

We also showed that bidirectional SeqKD was effective regardless of model sizes .


This paper explores the problem of computing text similarity between verb phrases describing skilled human behavior for the purpose of finding approximate matches .

Four parsers are evaluated on a large corpus of skill statements extracted from an enterprise-wide expertise taxonomy .

A similarity measure utilizing common semantic role features extracted from parse trees was found superior to an information-theoretic measure of similarity and comparable to the level of human agreement .

Knowledge-intensive industries need to become more efficient at deploying the right expertise as quickly and smoothly as possible , thus it is desired to have systems that can quickly match and deploy skilled individuals to meet customer needs .

The searches in most of the current matching systems are based on exact matches between skill statements .

However , exact matching is very likely to miss individuals who are very good matches to the job but did n't select the exact skills that appeared in the open job description .

It is always hard for individuals to find the perfect skills to describe their skill sets .

For example , an individual might not know whether to choose a skill stating that refers to " maintaining " a given product or " supporting " it or whether to choose a skill about maintaining a " database " or about maintaining " DB2 " .

Thus , it is desirable for the job search system to be able to find approximate matches , instead of only exact matches , between available individuals and open job positions .

More specifically , a skill similarity computation is needed to allow searches to be expanded to related skills , and return more potential matches .

In this paper , we present our work on developing a skill similarity computation based upon semantic commonalities between skill statements .

Although there has been much work on text similarity metrics ( Lin , 1998a ; Corley and Mihalcea , 2005 ) , most approaches treat texts as a bag of words and try to find shared words with certain statistical properties based on corpus frequencies .

As a result , the structural information in the text is ignored in these approaches .

We will describe a new semantic approach that takes the structural information of the text into consideration and matches skill statements on corresponding semantic roles .

We will demonstrate that it can outperform standard statistical text similarity techniques , and reach the level of human agreement .

In Section 2 , we first describe the skill statements we extracted from an enterprise-wide expertise taxonomy .

In Section 3 , we describe the performance of a standard statistical approach on this task .

This motivates our semantic approach of matching skill statements on corresponding semantic roles .

We also compare and evaluate the performance of four natural language parsers ( the Charniak parser , the Stanford parser , the ESG parser , and MINIPAR ) for the purpose of our task .

An inter-rater agreement study and evaluation of our approach will be presented in Section 4 .

We end with a discussion and conclusion .

In this paper , we have presented our work on a semantic similarity computation for skill statements in natural language .

We compared and evaluated four different natural language parsers for our task , and matched skills on their corresponding semantic roles extracted from the parse trees generated by one of these parsers .

The evaluation results showed that the skill similarity computation based on semantic role matching can outperform a standard statistical approach and reach the level of human agreement .

The extracted semantic role information can also be incorporated into the standard statistical approaches as additional features .

One way is to give higher weights to those semantic role features deemed most important .

This approach has achieved a high performance for a text categorization task when combining extracted keywords with the full text ( Hulth and Megyesi , 2006 ) .

We have shown that good results can be achieved for a domain-specific text matching task by performing a simple word-based feature comparison on corresponding structural elements of texts .

We have shown that the structural elements of importance can be identified by domain-specific pattern analysis of corresponding parse trees .

We believe this approach can generalize to other domains where phrases , sentences , or other short texts need to be compared .


Continual relation learning aims to continually train a model on new data to learn incessantly emerging novel relations while avoiding catastrophically forgetting old relations .

Some pioneering work has proved that storing a handful of historical relation examples in episodic memory and replaying them in subsequent training is an effective solution for such a challenging problem .

However , these memorybased methods usually suffer from overfitting the few memorized examples of old relations , which may gradually cause inevitable confusion among existing relations .

Inspired by the mechanism in human long-term memory formation , we introduce episodic memory activation and reconsolidation ( EMAR ) to continual relation learning .

Every time neural models are activated to learn both new and memorized data , EMAR utilizes relation prototypes for memory reconsolidation exercise to keep a stable understanding of old relations .

The experimental results show that EMAR could get rid of catastrophically forgetting old relations and outperform the state-of-the-art continual learning models .

The code and datasets are released on https://github.com / thunlp/ ContinualRE .

Relation extraction aims at detecting relations between entities from text , e.g. , extracting the relation " the president of " from the given sentence " Newton served as the president of the Royal Society " , which could serve as external resource for various downstream applications ( Dong et al . , 2015 ; Xiong et al . , 2017 ; Schlichtkrull et al . , 2018 ) .

The conventional RE methods ( Riedel et al . , 2013 ; Zeng et al . , 2014 ; Lin et al . , 2016 ) mostly focus on recognizing relations for a fixed pre-defined relation set , and can not handle rapidly emerging novel relations in the real world .

Some researchers therefore explore to detect and learn incessantly emerging relations in an open scenario .

As shown in Figure 1 , their efforts can be formulated into a two-step pipeline :

( 1 ) Open Relation Learning extracts phrases and arguments to construct patterns of specific relations , and then discovers unseen relation types by clustering patterns , and finally expands sufficient examples of new relation types from large-scale textual corpora ; ( 2 ) Continual Relation Learning continually uses those expanded examples of new relations to train an effective classifier .

The classifier is trained on a sequence of tasks for handling both existing and novel relations , where each task has its own relation set .

Although continual relation learning is vital for learning emerging relations , there are rare explorations for this field .

A straightforward solution is to store all historical data and re-train models every time new relations and examples come in .

Nevertheless , it is computationally expensive since relations are in sustainable growth .

Moreover , the huge example number of each relation makes frequently mixing new and old examples become infeasible in the real world .

Therefore , storing all data is not practical in continual relation learning .

In view of this , the recent preliminary work ( Wang et al . , 2019 ) indicates that the main challenge of continual relation learning is the catastrophic forgetting problem , i.e. , it is hard to learn new relations and meanwhile avoid forgetting old relations , considering memorizing all the data is almost impossible .

Recent work ( Shin et al . , 2017 ; Kemker and Kanan , 2018 ; Chaudhry et al . , 2019 ) has shown that the memory-based approaches , maintaining episodic memory to save a few training examples in old tasks and re-training memorized examples during training new tasks , are one of the most effective solutions to the catastrophic forgetting problem , especially for continual learning in NLP scenarios ( Wang et al . , 2019 ;

However , existing memory-based models still suffer from an overfitting problem : when adapting them for continual relation learning , they may frequently change feature distribution of old relations , gradually overfit a few examples in memory , and finally become confused among old relations after long-term training .

In fact , these memory-based methods are similar to long-term memory model of mammalian memory in neuroscience ( McClelland et al . , 1995 ; Bontempi et al . , 1999 ) .

Although researchers in neuroscience are not clear about secrets inside the human brain , they reach a consensus that the formation of long-term memory relies on continually replaying and consolidating information ( Tononi and Cirelli , 2006 ; Boyce et al . , 2016 ; Yang et al . , 2014 ) , corresponding to the episodic memory and memory replay in continual learning models .

Yet later work ( Nader et al . , 2000 ; Lee et al . , 2004 ; Alberini , 2005 ) in neuroscience indicates that reactivation of consolidated memory triggers a reconsolidation stage to continually maintain memory , and memory is easy to be changed or erased in this stage .

To apply some reconsolidation exercises can help memory go through this stage and keep long-term memory stable .

Intuitively , the ex-isting memory-based models seem like continual memory activation without reconsolidation exercises , and thus become sensitive and volatile .

Inspired by the reconsolidation mechanism in human long-term memory formation , we introduce episodic memory activation and reconsolidation ( EMAR ) to continual relation learning in this paper .

More specifically , when training models on new relations and their examples , we first adopt memory replay to activate neural models on examples of both new relations and memory , and then utilize a special reconsolidation module to let models avoid excessively changing and erasing feature distribution of old relations .

As the core of relation learning is to grasp relation prototypes rather than rote memorization of relation examples , our reconsolidation module requires models to be able to distinguish old relation prototypes after each time memory is replayed and activated .

As compared with pioneering explorations to improve episodic memory replay ( Chaudhry et al . , 2019 ; Wang et al . , 2019 ) , with toughly keeping feature distribution of old relations invariant , EMAR is more flexible in feature spaces and powerful in remembering relation prototypes .

We conduct sufficient experiments on several RE datasets , and the results show that EMAR effectively alleviates the catastrophic forgetting problem and significantly outperforms the stateof-the-art continual learning models .

Further experiments and analyses indicate the reasons for the effectiveness of EMAR , proving that it can utilize a few examples in old tasks to reconsolidate old relation prototypes and keep better distinction among old relations after long-term training .

To alleviate catastrophically forgetting old relations in continual relation learning , we introduce episodic memory activation and reconsolidation ( EMAR ) , inspired by the mechanism in human long-term memory formation .

Compared with existing memory-based methods , EMAR requires models to understand the prototypes of old relations rather than to overfit a few specific memorized examples , which can keep better distinction among relations after long-term training .

We conduct experiments on three benchmarks in relation extraction and carry out extensive experimental results as well as empirical analyses , showing the effectiveness of EMAR on utilizing memorized examples .

For future work , how to combine open relation learning and continual relation learning together to complete the pipeline for emerging relations still remains a problem , and we will continue to work on it .


This paper proposes to study a fine-grained semantic novelty detection task , which can be illustrated with the following example .

It is normal that a person walks a dog in the park , but if someone says " A man is walking a chicken in the park , " it is novel .

Given a set of natural language descriptions of normal scenes , we want to identify descriptions of novel scenes .

We are not aware of any existing work that solves the problem .

Although existing novelty or anomaly detection algorithms are applicable , since they are usually topic-based , they perform poorly on our fine-grained semantic novelty detection task .

This paper proposes an effective model ( called GAT-MA ) to solve the problem and also contributes a new dataset .

Experimental evaluation shows that GAT-MA outperforms 11 baselines by large margins .

Novelty or anomaly detection has been an important research topic since 1970s ( Barnett and Lewis , 1994 ) due to numerous applications ( Chalapathy et al . , 2018 ; Pang et al . , 2021 ) .

Recently , it has also become important for natural language processing ( NLP ) .

Many researchers have studied the problem in the text classification setting ( Fei and Liu , 2016 ; Shu et al . , 2017 ; Xu et al . , 2019 ; Lin and Xu , 2019 ;

However , these text novelty classifiers are mainly coarse-grained , working at the document or topic level .

Given a text document , their goal is to detect whether the text belongs to a known class or unknown class .

This paper introduces a new text novelty detection problem -fine-grained semantic novelty detection .

Specifically , given a text description d , we detect whether d represents a semantically novel fact or not .

This work considers text data that describe scenes of real-world phenomena in natural language ( NL ) .

In our daily lives , we observe different real-world phenomena ( events , activities , situations , etc . ) and often describe these observations ( referred as " scenes " onwards ) in NL to others or write about them .

It is quite natural to observe scenes that we have not seen before ( i.e. , novel scenes ) .

For example , it is a common scene that " A person walks a dog in the park " , but if someone says " A man is walking a chicken in the park " , it is quite unexpected and novel .

Detecting such semantic novelty requires complex conceptual and semantic reasoning over text and thus , is a challenging NLP problem .

Note that conceptually , the judgement of the novelty of a scene is subjective and might differ from person to person .

However , there are some scenes for which a majority of people have agreement about their novelty .

A good example of such majority-view of novelty is the widely-spread meme pictures on social media , which contain novel interactions between objects .

In this work , we restrict our research to this majority-based view of novelty and leave the personalized novelty view angle for the future work .

In this work , we leverage the captions of images from popular datasets like COCO , Flickr , etc . , to build a semantic novelty detection dataset ( Sec . 3 ) , 1where we consider an image as a scene and the corresponding image captions as different NL descriptions of the scene .

Detecting text describing semantically novel observations have many applications , e.g. , recommending novel news , novel images & videos ( based on their text descriptions ) , social media posts and conversations .

The problem of semantic novelty detection is defined as follows .

Problem Definition : Given a set of natural language descriptions D = { d 1 , d 2 , ..... d n } of common scenes , build a model M using D to score the semantic novelty of a test NL description d with respect to D , i.e. , classifying d into one of the two classes { NORMAL , NOVEL } .

" NORMAL " means that d is a description of a common scene and " NOVEL " means d is a description of a se-mantically novel scene .

As the detection model M is built only with " NORMAL " class data , the task is an one-class text classification problem .

We are unaware of any existing work that can effectively solve this problem .

Although existing novelty / anomaly detection and one-class classification algorithms are applicable , since they are coarse-grained or topic-based , they perform poorly on our task ( see Sec . 5 ) .

Note that although we focus on solving the problem of semantic novelty detection of NL descriptions of scenes , the proposed task and solution framework are generally applicable to other applications .

This paper proposes a new technique , called GAT-MA ( Graph Attention network with Max-Margin loss and knowledge-based contrastive data Augmentation ) to identify NL description sentences of novel scenes .

Since our task is at the sentence level and fine-grained , we exploit Graph Attention Network ( GAT ) on the parsed dependency graph of each sentence , which fuses both semantic and syntactic information in the sentence for reasoning with the internal interactions of entities and actions .

To enable the model to capture long-range interactions , we stack multiple layers of GATs to build a deep GAT model with multi-hop graph attention .

We also create the pseudo novel training data based on the given normal training data through contrastive data augmentation .

Thus , GAT-MA is trained with the given original normal scene descriptions and the augmented pseudo novel scene descriptions ( Sec . 4 ) .

GAT-MA is evaluated using our newly created Novel Scene Description Detection ( NSD2 ) Dataset .

The results show that GAT-MA outperforms a wide range of latest novelty or anomaly detection baselines by very large margins .

We propose a new task of semantic novelty detection in text .

Whereas the existing work focuses on coarse-grained document-or topiclevel novelty , our task requires fine-grained sentence-level semantic & syntactic analysis .

We propose a highly effective technique called GAT-MA to solve the proposed semantic novelty detection problem , which is based on GAT with dependency parsing and knowledgebased contrastive data augmentation .

We create a new dataset called NSD2 for the proposed task .

The dataset can be used as a benchmark dataset by the NLP community .

Novelty detection is an important problem because anything novel is of interest .

This paper proposed a semantic novelty detection problem and designed a graph attention network based approach ( called GAT-MA ) exploiting parsing and data augmentation to solve the problem .

As there is no existing evaluation dataset for the proposed task , an evaluation dataset has been created .

Experimental comparisons with a wide range of baselines showed that GAT-MA outperforms them by very large margins .

We create a new dataset called NSD2 for the proposed task .


Long distance word reordering is a major challenge in statistical machine translation research .

Previous work has shown using source syntactic trees is an effective way to tackle this problem between two languages with substantial word order difference .

In this work , we further extend this line of exploration and propose a novel but simple approach , which utilizes a ranking model based on word order precedence in the target language to reposition nodes in the syntactic parse tree of a source sentence .

The ranking model is automatically derived from word aligned parallel data with a syntactic parser for source language based on both lexical and syntactical features .

We evaluated our approach on largescale Japanese-English and English-Japanese machine translation tasks , and show that it can significantly outperform the baseline phrasebased SMT system .

This work has been done while the first author was visiting

Modeling word reordering between source and target sentences has been a research focus since the emerging of statistical machine translation .

In phrase-based models ( Och , 2002 ; Koehn et al . , 2003 ) , phrase is introduced to serve as the fundamental translation element and deal with local reordering , while a distance based distortion model is used to coarsely depict the exponentially decayed word movement probabilities in language translation .

Further work in this direction employed lexi-calized distortion models , including both generative ( Koehn et al . , 2005 ) and discriminative ( Zens and Ney , 2006 ; Xiong et al . , 2006 ) variants , to achieve finer-grained estimations , while other work took into account the hierarchical language structures in translation ( Chiang , 2005 ; Galley and Manning , 2008 ) .

Long-distance word reordering between language pairs with substantial word order difference , such as Japanese with Subject-Object-Verb ( SOV ) structure and English with Subject-Verb-Object ( SVO ) structure , is generally viewed beyond the scope of the phrase-based systems discussed above , because of either distortion limits or lack of discriminative features for modeling .

The most notable solution to this problem is adopting syntax-based SMT models , especially methods making use of source side syntactic parse trees .

There are two major categories in this line of research .

One is tree-to-string model ( Quirk et al . , 2005 ; Liu et al . , 2006 ) which directly uses source parse trees to derive a large set of translation rules and associated model parameters .

The other is called syntax pre-reordering -an approach that re-positions source words to approximate target language word order as much as possible based on the features from source syntactic parse trees .

This is usually done in a preprocessing step , and then followed by a standard phrase-based SMT system that takes the re-ordered source sentence as input to finish the translation .

In this paper , we continue this line of work and address the problem of word reordering based on source syntactic parse trees for SMT .

Similar to most previous work , our approach tries to rearrange the source tree nodes sharing a common parent to mimic the word order in target language .

To this end , we propose a simple but effective ranking-based approach to word reordering .

The ranking model is automatically derived from the word aligned parallel data , viewing the source tree nodes to be reordered as list items to be ranked .

The ranks of tree nodes are determined by their relative positions in the target language -the node in the most front gets the highest rank , while the ending word in the target sentence gets the lowest rank .

The ranking model is trained to directly minimize the mis-ordering of tree nodes , which differs from the prior work based on maximum likelihood estimations of reordering patterns ( Li et al . , 2007 ; Genzel , 2010 ) , and does not require any special tweaking in model training .

The ranking model can not only be used in a pre-reordering based SMT system , but also be integrated into a phrasebased decoder serving as additional distortion features .

We evaluated our approach on large-scale Japanese-English and English-Japanese machine translation tasks , and experimental results show that our approach can bring significant improvements to the baseline phrase-based SMT system in both preordering and integrated decoding settings .

In the rest of the paper , we will first formally present our ranking-based word reordering model , then followed by detailed steps of modeling training and integration into a phrase-based SMT system .

Experimental results are shown in Section 5 .

Section 6 consists of more discussions on related work , and Section 7 concludes the paper .

In this paper we present a ranking based reordering method to reorder source language to match the word order of target language given the source side parse tree .

Reordering is formulated as a task to rank different nodes in the source side syntax tree according to their relative position in the target language .

The ranking model is automatically trained to minimize the mis-ordering of tree nodes in the training data .

Large scale experiment shows improvement on both reordering metric and SMT performance , with up to 1.73 point BLEU gain in our evaluation test .

In future work , we plan to extend the ranking model to handle reordering between multiple levels of source trees .

We also expect to explore better way to integrate ranking reorder model into SMT system instead of a simple penalty scheme .

Along the research direction of preprocessing the source language to facilitate translation , we consider to not only change the order of the source language , but also inject syntactic structure of the target language into source language by adding pseudo words into source sentences .


How can the development of ideas in a scientific field be studied over time ?

We apply unsupervised topic modeling to the ACL Anthology to analyze historical trends in the field of Computational Linguistics from 1978 to 2006 .

We induce topic clusters using Latent Dirichlet Allocation , and examine the strength of each topic over time .

Our methods find trends in the field including the rise of probabilistic methods starting in 1988 , a steady increase in applications , and a sharp decline of research in semantics and understanding between 1978 and 2001 , possibly rising again after 2001 .

We also introduce a model of the diversity of ideas , topic entropy , using it to show that COLING is a more diverse conference than ACL , but that both conferences as well as EMNLP are becoming broader over time .

Finally , we apply Jensen-Shannon divergence of topic distributions to show that all three conferences are converging in the topics they cover .

How can we identify and study the exploration of ideas in a scientific field over time , noting periods of gradual development , major ruptures , and the waxing and waning of both topic areas and connections with applied topics and nearby fields ?

One important method is to make use of citation graphs ( Garfield , 1955 ) .

This enables the use of graphbased algorithms like PageRank for determining researcher or paper centrality , and examining whether their influence grows or diminishes over time .

However , because we are particularly interested in the change of ideas in a field over time , we have chosen a different method , following Kuhn ( 1962 ) .

In Kuhn 's model of scientific change , science proceeds by shifting from one paradigm to another .

Because researchers ' ideas and vocabulary are constrained by their paradigm , successive incommensurate paradigms will naturally have different vocabulary and framing .

Kuhn 's model is intended to apply only to very large shifts in scientific thought rather than at the micro level of trends in research foci .

Nonetheless , we propose to apply Kuhn 's insight that vocabulary and vocabulary shift is a crucial indicator of ideas and shifts in ideas .

Our operationalization of this insight is based on the unsupervised topic model Latent Dirichlet Allocation ( LDA ; Blei et al . ( 2003 ) ) .

For many fields , doing this kind of historical study would be very difficult .

Computational linguistics has an advantage , however : the ACL Anthology , a public repository of all papers in the Computational Linguistics journal and the conferences and workshops associated with the ACL , COLING , EMNLP , and so on .

The ACL Anthology ( Bird , 2008 ) , and comprises over 14,000 documents from conferences and the journal , beginning as early as 1965 through 2008 , indexed by conference and year .

This resource has already been the basis of citation analysis work , for example , in the ACL Anthology Network of Joseph and Radev ( 2007 ) .

We apply LDA to the text of the papers in the ACL Anthology to induce topics , and use the trends in these topics over time and over conference venues to address questions about the development of the field .

Despite the relative youth of our field , computational linguistics has witnessed a number of research trends and shifts in focus .

While some trends are obvious ( such as the rise in machine learning methods ) , others may be more subtle .

Has the field gotten more theoretical over the years or has there been an increase in applications ?

What topics have declined over the years , and which ones have remained roughly constant ?

How have fields like Dialogue or Machine Translation changed over the years ?

Are there differences among the conferences , for example between COLING and ACL , in their interests and breadth of focus ?

As our field matures , it is important to go beyond anecdotal description to give grounded answers to these questions .

Such answers could also help give formal metrics to model the differences between the many conferences and venues in our field , which could influence how we think about reviewing , about choosing conference topics , and about long range planning in our field .

Our method discovers a number of trends in the field , such as the general increase in applications , the steady decline in semantics , and its possible reversal .

We also showed a convergence over time in topic coverage of ACL , COLING , and EMNLP as well an expansion of topic diversity .

This growth and convergence of the three conferences , perhaps influenced by the need to increase recall ( Church , 2005 ) seems to be leading toward a tripartite realization of a single new " latent " conference .


In this paper , we aim to understand whether current language and vision ( LaVi ) models truly grasp the interaction between the two modalities .

To this end , we propose an extension of the MS-COCO dataset , FOIL-COCO , which associates images with both correct and ' foil ' captions , that is , descriptions of the image that are highly similar to the original ones , but contain one single mistake ( ' foil word ' ) .

We show that current LaVi models fall into the traps of this data and perform badly on three tasks : a ) caption classification ( correct vs. foil ) ; b ) foil word detection ; c ) foil word correction .

Humans , in contrast , have near-perfect performance on those tasks .

We demonstrate that merely utilising language cues is not enough to model FOIL-COCO and that it challenges the state-of-the-art by requiring a fine-grained understanding of the relation between text and image .

Most human language understanding is grounded in perception .

There is thus growing interest in combining information from language and vision in the NLP and AI communities .

So far , the primary testbeds of Language and Vision ( LaVi ) models have been ' Visual Question Answering ' ( VQA ) ( e.g. Antol et al . ( 2015 ) ; Malinowski and Fritz ( 2014 ) ; Malinowski et al . ( 2015 ) ; Gao et al . ( 2015 ) ; Ren et al . ( 2015 ) ) and ' Image Captioning ' ( IC ) ( e.g. Hodosh et al . ( 2013 ) ; Fang et al . ( 2015 ) ; Chen and Lawrence Zitnick ( 2015 ) ; Donahue et al . ( 2015 ) ; Karpathy and Fei-Fei ( 2015 ) ; Vinyals et al . ( 2015 ) ) .

Whilst some models have seemed extremely successful on those tasks , it remains unclear how the reported results should be interpreted and what those Figure 1 : Is the caption correct or foil ( T1 ) ?

If it is foil , where is the mistake ( T2 ) and which is the word to correct the foil one ( T3 ) ?

There is an emerging feeling in the community that the VQA task should be revisited , especially as many current dataset can be handled by ' blind ' models which use language input only , or by simple concatenation of language and vision features ( Agrawal et al . , 2016 ; Jabri et al . , 2016 ; Zhang et al . , 2016 ; Goyal et al . , 2016a ) .

In IC too , Hodosh and Hockenmaier ( 2016 ) showed that , contrarily to what prior research had suggested , the task is far from been solved , since IC models are not able to distinguish between a correct and incorrect caption .

Such results indicate that in current datasets , language provides priors that make LaVi models successful without truly understanding and integrating language and vision .

Johnson et al . ( 2016 ) also point out that current data ' conflate multiple sources of error , making it hard to pinpoint model weaknesses ' , thus highlighting the need for diagnostic datasets .

Thirdly , existing IC evaluation metrics are sensitive to n-gram overlap and there is a need for measures that better simulate human judgments ( Hodosh et al . , 2013 ; Elliott and Keller , 2014 ; Anderson et al . , 2016 ) .

Our paper tackles the identified issues by proposing an automatic method for creating a large dataset of real images with minimal language bias and some diagnostic abilities .

Our dataset , FOIL ( Find One mismatch between Image and Language caption),1 consists of images associated with incorrect captions .

The captions are produced by introducing one single error ( or ' foil ' ) per caption in existing , human-annotated data ( Figure 1 ) .

This process results in a challenging error-detection / correction setting ( because the caption is ' nearly ' correct ) .

It also provides us with a ground truth ( we know where the error is ) that can be used to objectively measure the performance of current models .

We propose three tasks based on widely accepted evaluation measures : we test the ability of the system to a ) compute whether a caption is compatible with the image ( T1 ) ; b ) when it is incompatible , highlight the mismatch in the caption ( T2 ) ; c ) correct the mistake by replacing the foil word ( T3 ) .

The dataset presented in this paper ( Section 3 ) is built on top of MS-COCO ( Lin et al . , 2014 ) , and contains 297,268 datapoints and 97,847 images .

We evaluate two state-of-the-art VQA models : the popular one by Antol et al . ( 2015 ) , and the attention-based model by Lu et al . ( 2016 ) , and one popular IC model by ( Wang et al . , 2016 ) .

We show that those models perform close to chance level , while humans can perform the tasks accurately ( Section 4 ) .

Section 5 provides an analysis of our results , allowing us to diagnose three failures of LaVi models .

First , their coarse representations of language and visual input do not encode suitably structured information to spot mismatches between an utterance and the corresponding scene ( tested by T1 ) .

Second , their language representation is not finegrained enough to identify the part of an utterance that causes a mismatch with the image as it is ( T2 ) .

Third , their visual representation is also too poor to spot and name the visual area that corresponds to a captioning error ( T3 ) .

We have introduced FOIL-COCO , a large dataset of images associated with both correct and foil captions .

The error production is automatically generated , but carefully thought out , making the task of spotting foils particularly challenging .

By associating the dataset with a series of tasks , we allow for diagnosing various failures of current LaVi systems , from their coarse understanding of the correspondence between text and vision to their grasp of language and image structure .

Our hypothesis is that systems which , like humans , deeply integrate the language and vision modalities , should spot foil captions quite easily .

The SoA LaVi models we have tested fall through that test , implying that they fail to integrate the two modalities .

To complete the analysis of these results , we plan to carry out a further task , namely ask the system to detect in the image the area that produces the mismatch with the foil word ( the red box around the bird in Figure 1 . )

This extra step would allow us to fully diagnose the failure of the tested systems and confirm what is implicit in our results from task 3 : that the algorithms are unable to map particular elements of the text to their visual counterparts .

We note that the addition of this extra step will move this work closer to the textual / visual explanation research ( e.g. , ( Park et al . , 2016 ; Selvaraju et al . , 2016 ) ) .

We will then have a pipeline able to not only test whether a mistake can be detected , but also whether the system can explain its decision : ' the wrong word is dog because the cyclists are in fact approaching a bird , there , in the image ' .

LaVi models are a great success of recent research , and we are impressed by the amount of ideas , data and models produced in this stimulating area .

With our work , we would like to push the community to think of ways that models can better merge language and vision modalites , instead of merely using one to supplement the other .


Language is increasingly being used to define rich visual recognition problems with supporting image collections sourced from the web .

Structured prediction models are used in these tasks to take advantage of correlations between co-occurring labels and visual input but risk inadvertently encoding social biases found in web corpora .

In this work , we study data and models associated with multilabel object classification and visual semantic role labeling .

We find that ( a ) datasets for these tasks contain significant gender bias and ( b ) models trained on these datasets further amplify existing bias .

For example , the activity cooking is over 33 % more likely to involve females than males in a training set , and a trained model further amplifies the disparity to 68 % at test time .

We propose to inject corpus-level constraints for calibrating existing structured prediction models and design an algorithm based on Lagrangian relaxation for collective inference .

Our method results in almost no performance loss for the underlying recognition task but decreases the magnitude of bias amplification by 47.5 % and 40.5 % for multilabel classification and visual semantic role labeling , respectively .

Visual recognition tasks involving language , such as captioning ( Vinyals et al . , 2015 ) , visual question answering ( Antol et al . , 2015 ) , and visual semantic role labeling ( Yatskar et al . , 2016 ) , have emerged as avenues for expanding the diversity of information that can be recovered from images .

These tasks aim at extracting rich seman-tics from images and require large quantities of labeled data , predominantly retrieved from the web .

Methods often combine structured prediction and deep learning to model correlations between labels and images to make judgments that otherwise would have weak visual support .

For example , in the first image of Figure 1 , it is possible to predict a spatula by considering that it is a common tool used for the activity cooking .

Yet such methods run the risk of discovering and exploiting societal biases present in the underlying web corpora .

Without properly quantifying and reducing the reliance on such correlations , broad adoption of these models can have the inadvertent effect of magnifying stereotypes .

In this paper , we develop a general framework for quantifying bias and study two concrete tasks , visual semantic role labeling ( vSRL ) and multilabel object classification ( MLC ) .

In vSRL , we use the imSitu formalism ( Yatskar et al . , 2016 ( Yatskar et al . , , 2017 ) ) , where the goal is to predict activities , objects and the roles those objects play within an activity .

For MLC , we use MS-COCO ( Lin et al . , 2014 ; Chen et al . , 2015 ) , a recognition task covering 80 object classes .

We use gender bias as a running example and show that both supporting datasets for these tasks are biased with respect to a gender binary1 .

Our analysis reveals that over 45 % and 37 % of verbs and objects , respectively , exhibit bias toward a gender greater than 2:1 .

For example , as seen in Figure 1 , the cooking activity in imSitu is a heavily biased verb .

Furthermore , we show that after training state-of-the-art structured predictors , models amplify the existing bias , by 5.0 % for vSRL , and 3.6 % in MLC .

Each image is paired with a table describing a situation : the verb , cooking , its semantic roles , i.e agent , and noun values filling that role , i.e. woman .

In the imSitu training set , 33 % of cooking images have man in the agent role while the rest have woman .

After training a Conditional Random Field ( CRF ) , bias is amplified : man fills 16 % of agent roles in cooking images .

To reduce this bias amplification our calibration method adjusts weights of CRF potentials associated with biased predictions .

After applying our methods , man appears in the agent role of 20 % of cooking images , reducing the bias amplification by 25 % , while keeping the CRF vSRL performance unchanged .

To mitigate the role of bias amplification when training models on biased corpora , we propose a novel constrained inference framework , called RBA , for Reducing Bias Amplification in predictions .

Our method introduces corpus-level constraints so that gender indicators co-occur no more often together with elements of the prediction task than in the original training distribution .

For example , as seen in Figure 1 , we would like noun man to occur in the agent role of the cooking as often as it occurs in the imSitu training set when evaluating on a development set .

We combine our calibration constraint with the original structured predictor and use Lagrangian relaxation ( Korte and Vygen , 2008 ;

Rush and Collins , 2012 ) to reweigh bias creating factors in the original model .

We evaluate our calibration method on imSitu vSRL and COCO MLC and find that in both instances , our models substantially reduce bias amplification .

For vSRL , we reduce the average magnitude of bias amplification by 40.5 % .

For MLC , we are able to reduce the average magnitude of bias amplification by 47.5 % .

Overall , our calibration methods do not affect the performance of the underlying visual system , while substantially reducing the reliance of the system on socially biased correlations 2 .

2 Code and data are available at https://github . com / uclanlp / reducingbias

Structured prediction models can leverage correlations that allow them to make correct predictions even with very little underlying evidence .

Yet such models risk potentially leveraging social bias in their training data .

In this paper , we presented a general framework for visualizing and quantifying biases in such models and proposed RBA to calibrate their predictions under two different settings .

Taking gender bias as an example , our analysis demonstrates that conditional random fields can amplify social bias from data while our approach RBA can help to reduce the bias .

Our work is the first to demonstrate structured prediction models amplify bias and the first to propose methods for reducing this effect but significant avenues for future work remain .

While RBA can be applied to any structured predictor , it is unclear whether different predictors amplify bias more or less .

Furthermore , we presented only one method for measuring bias .

More extensive analysis could explore the interaction among predictor , bias measurement , and bias deamplification method .

Future work also includes applying bias reducing methods in other structured domains , such as pronoun reference resolution ( Mitkov , 2014 ) .


We present a pairwise context-sensitive Autoencoder for computing text pair similarity .

Our model encodes input text into context-sensitive representations and uses them to compute similarity between text pairs .

Our model outperforms the state-of-the-art models in two semantic retrieval tasks and a contextual word similarity task .

For retrieval , our unsupervised approach that merely ranks inputs with respect to the cosine similarity between their hidden representations shows comparable performance with the state-of-the-art supervised models and in some cases outperforms them .

Representation learning algorithms learn representations that reveal intrinsic low-dimensional structure in data ( Bengio et al . , 2013 ) .

Such representations can be used to induce similarity between textual contents by computing similarity between their respective vectors ( Huang et al . , 2012 ; Silberer and Lapata , 2014 ) .

Recent research has made substantial progress on semantic similarity using neural networks ( Rothe and Schütze , 2015 ; Dos Santos et al . , 2015 ; Severyn and Moschitti , 2015 ) .

In this work , we focus our attention on deep autoencoders and extend these models to integrate sentential or document context information about their inputs .

We represent context information as low dimensional vectors that will be injected to deep autoencoders .

To the best of our knowledge , this is the first work that enables integrating context into autoencoders .

In representation learning , context may appear in various forms .

For example , the context of a current sentence in a document could be either its neighboring sentences ( Lin et al . , 2015 ; Wang and Cho , 2015 ) , topics associated with the sentence ( Mikolov and Zweig , 2012 ; Le and Mikolov , 2014 ) , the document that contains the sentence ( Huang et al . , 2012 ) , as well as their combinations ( Ji et al . , 2016 ) .

It is important to integrate context into neural networks because these models are often trained with only local information about their individual inputs .

For example , recurrent and recursive neural networks only use local information about previously seen words in a sentence to predict the next word or composition .

1On the other hand , context information ( such as topical information ) often capture global information that can guide neural networks to generate more accurate representations .

We investigate the utility of context information in three semantic similarity tasks : contextual word sense similarity in which we aim to predict semantic similarity between given word pairs in their sentential context ( Huang et al . , 2012 ; Rothe and Schütze , 2015 ) , question ranking in which we aim to retrieve semantically equivalent questions with respect to a given test question ( Dos Santos et al . , 2015 ) , and answer ranking in which we aim to rank single-sentence answers with respect to a given question ( Severyn and Moschitti , 2015 ) .

The contributions of this paper are as follows :

( 1 ) integrating context information into deep autoencoders and ( 2 ) showing that such integration improves the representation performance of deep autoencoders across several different semantic similarity tasks .

Our model outperforms the state-of-the-art su-pervised baselines in three semantic similarity tasks .

Furthermore , the unsupervised version of our autoencoder show comparable performance with the supervised baseline models and in some cases outperforms them .

We introduce an effective approach to integrate sentential or document context into deep autoencoders and show that such integration is important in semantic similarity tasks .

In the future , we aim to investigate other types of linguistic context ( such as POS tag and word dependency information , word sense , and discourse relations ) and develop a unified representation learning framework that integrates such linguistic context with representation learning models .


Distance-based ( windowless ) word assocation measures have only very recently appeared in the NLP literature and their performance compared to existing windowed or frequency-based measures is largely unknown .

We conduct a largescale empirical comparison of a variety of distance-based and frequency-based measures for the reproduction of syntagmatic human assocation norms .

Overall , our results show an improvement in the predictive power of windowless over windowed measures .

This provides support to some of the previously published theoretical advantages and makes windowless approaches a promising avenue to explore further .

This study also serves as a first comparison of windowed methods across numerous human association datasets .

During this comparison we also introduce some novel variations of window-based measures which perform as well as or better in the human association norm task than established measures .

Automatic discovery of semantically associated words has attracted a large amount of attention in the last decades and a host of computational association measures have been proposed to deal with this task ( see Section 2 ) .

These measures traditionally rely on the co-ocurrence frequency of two words in a corpus to estimate a relatedness score .

There has been a recent emergence of distancebased language modelling techiques in NLP ( Savicki and Hlavacova , 2002 ; Terra and Clarke , 2004 ) in which the number of tokens separating words is the essential quantity .

While some of this work has considered distance-based alternatives to conventional association measures ( Hardcastle , 2005 ; Washtell , 2009 ) , there has been no principled empirical evaluation of these measures as predictors of human association .

We remedy this by conducting a thorough comparison of a wide variety of frequency-based and distance-based measures as predictors of human association scores as elicited in several different free word association tasks .

In this work we focus on first-order association measures as predictors of syntagmatic associations .

This is in contrast to second and higher-order measures which are better predictors of paradigmatic associations , or word similarity .

The distinction between syntagmatic and paradigmatic relationship types is neither exact nor mutually exclusive , and many paradigmatic relationships can be observed syntagmatically in the text .

Roughly in keeping with ( Rapp , 2002 ) , we hereby regard paradigmatic assocations as those based largely on word similarity ( i.e. including those typically classed as synonyms , antonyms , hypernyms , hyponyms etc ) , whereas syntagmatic associations are all those words which strongly invoke one another yet which can not readily be said to be similar .

Typically these will have an identifiable semantic or grammatical relationship ( meronym / holonym : stem -flower , verb / object : eat -food etc ) , or may have harder-to-classify topical or idiomatic relationships ( family -Christmas , rock -roll ) .

We will show in Section 3.2 that syntagmatic relations by themselves constitute a substantial 25 - 40 % of the strongest human responses to cue words .

Although the automatic detection of these assocations in text has received less attention than that of paradigmatic associations , they are nonetheless important in applications such as the resolution of bridging anaphora ( Vieira and Poesio , 2000 ) .

Furthermore , first-order associations are often the basis of higher-order vector wordspace models used for predicting paradigmatic relationships : i.e. through the observation of words which share similar sets of syntagmatic associations .

Therefore improvements made at the level we are concerned with may reasonably be expected to carry through to applications which hinge on the identification of paradigmatic relationships .

After a discussion of previous work in Section 2 , we formulate the exact association measures and parameter settings which we compare in Section 3 , where we also introduce the corpora and human association sets used .

Then , by using evaluations similar to those described in ( Baroni et al . , 2008 ) and by Rapp ( 2002 ) , we show that the best distance-based measures correlate better overall with human association scores than do the best window based configurations ( see Section 4 ) , and that they also serve as better predictors of the strongest human associations ( see Section 5 ) .

This paper presented the first empirical comparison of window-based and the relatively recently introduced windowless association measures , using their ability to reproduce human association scores as a testbed .

We show that the best windowless measures are always at least as good as the best window-based measures , both when it comes to overall correlation with human association scores and predicting the strongest human response .

In addition , for several human association sets , they perform significantly better .

Although not all parameter settings and corpus sizes could be explored , we conclude that it is worthwhile investigating windowless association measures further .

As a side-benefit , we have also introduced new variants of existing frequency-based association measures and shown them to perform as well as or better than their existing counterparts .

Although these measures were semi-principled in their construction , a deeper understanding of why they work so well is needed .

This may in turn lead to the construction of superior windowless measures .

In our own future work , we are especially interested in using higher-order windowless association measures for retrieving paradigmatic relations as well as exploring their use in various NLP applications .


This paper presents an effective method for generating natural language sentences from their underlying meaning representations .

The method is built on top of a hybrid tree representation that jointly encodes both the meaning representation as well as the natural language in a tree structure .

By using a tree conditional random field on top of the hybrid tree representation , we are able to explicitly model phrase-level dependencies amongst neighboring natural language phrases and meaning representation components in a simple and natural way .

We show that the additional dependencies captured by the tree conditional random field allows it to perform better than directly inverting a previously developed hybrid tree semantic parser .

Furthermore , we demonstrate that the model performs better than a previous state-of-the-art natural language generation model .

Experiments are performed on two benchmark corpora with standard automatic evaluation metrics .

One of the ultimate goals in the field of natural language processing ( NLP ) is to enable computers to converse with humans through human languages .

To achieve this goal , two important issues need to be studied .

First , it is important for computers to capture the meaning of a natural language sentence in a meaning representation .

Second , computers should be able to produce a humanunderstandable natural language sentence from its meaning representation .

These two tasks are referred to as semantic parsing and natural language generation ( NLG ) , respectively .

In this paper , we use corpus-based statistical methods for constructing a natural language generation system .

Given a set of pairs , where each pair consists of a natural language ( NL ) sentence and its formal meaning representation ( MR ) , a learning method induces an algorithm that can be used for performing language generation from other previously unseen meaning representations .

A crucial question in any natural language processing system is the representation used .

Meaning representations can be in the form of a tree structure .

In Lu et al . ( 2008 ) , we introduced a hybrid tree framework together with a probabilistic generative model to tackle semantic parsing , where tree structured meaning representations are used .

The hybrid tree gives a natural joint tree representation of a natural language sentence and its meaning representation .

A joint generative model for natural language and its meaning representation , such as that used in Lu et al . ( 2008 ) has several advantages over various previous approaches designed for semantic parsing .

First , unlike most previous approaches , the generative approach models a simultaneous generation process for both NL and MR .

One elegant property of such a joint generative model is that it allows the modeling of both semantic parsing and natural language generation within the same process .

Second , the generative process proceeds as a recursive top-down Markov process in a way that takes advantage of the tree structure of the MR .

The hybrid tree generative model proposed in Lu et al . ( 2008 ) was shown to give stateof-the-art accuracy in semantic parsing on benchmark corpora .

While semantic parsing with hybrid trees has been studied in Lu et al . ( 2008 ) , its inverse task -NLG with hybrid trees -has not yet been explored .

We believe that the properties that make the hybrid trees effective for semantic parsing also make them effective for NLG .

In this paper , we develop systems for the generation task by building on top of the generative model introduced in Lu et al . ( 2008 ) ( referred to as the LNLZ08 system ) .

We first present a baseline model by directly " inverting " the LNLZ08 system , where an NL sentence is generated word by word .

We call this model the direct inversion model .

This model is unable to model some long range global dependencies over the entire NL sentence to be generated .

To tackle several weaknesses exhibited by the baseline model , we next introduce an alternative , novel model that performs generation at the phrase level .

Motivated by conditional random fields ( CRF ) ( Lafferty et al . , 2001 ) , a different parameterization of the conditional probability of the hybrid tree that enables the model to encode some longer range dependencies amongst phrases and MRs is used .

This novel model is referred to as the tree CRF-based model .

Evaluation results for both models are presented , through which we demonstrate that the tree CRF-based model performs better than the direct inversion model .

We also compare the tree CRFbased model against the previous state-of-the-art model of Wong and Mooney ( 2007 ) .

Furthermore , we evaluate our model on a dataset annotated with several natural languages other than English ( Japanese , Spanish , and Turkish ) .

Evaluation results show that our proposed tree CRF-based model outperforms the previous model .

In this paper , we presented two novel models for the task of generating natural language sentences from given meaning representations , under a hybrid tree framework .

We first built a simple direct inversion model as a baseline .

Next , to address the limitations associated with the direct inversion model , a tree CRF-based model was introduced .

We evaluated both models on standard benchmark corpora .

Evaluation results show that the tree CRF-based model performs better than the direct inversion model , and that the tree CRF-based model also outperforms WASP -1 + + , which was a previous state-of-the-art system reported in the literature .


In this paper we examine how the differences in modelling between different data driven systems performing the same NLP task can be exploited to yield a higher accuracy than the best individual system .

We do this by means of an experiment involving the task of morpho-syntactic wordclass tagging .

Four well-known tagger generators ( Hidden Markov Model , Memory-Based , Transformation Rules and Maximum Entropy ) are trained on the same corpus data .

After comparison , their outputs are combined using several voting strategies and second stage classifiers .

All combination taggers outperform their best component , with the best combination showing a 19.1 % lower error rate than the best individual tagger .

In all Natural Language Processing ( NLP ) systems , we find one or more language models which are used to predict , classify and/or interpret language related observations .

Traditionally , these models were categorized as either rule-based / symbolic or corpusbased / probabilistic .

Recent work ( e.g. Brill 1992 ) has demonstrated clearly that this categorization is in fact a mix-up of two distinct Categorization systems : on the one hand there is the representation used for the language model ( rules , Markov model , neural net , case base , etc . ) and on the other hand the manner in which the model is constructed ( hand crafted vs. data driven ) .

Data driven methods appear to be the more popular .

This can be explained by the fact that , in general , hand crafting an explicit model is rather difficult , especially since what is being modelled , natural language , is not ( yet ) wellunderstood .

When a data driven method is used , a model is automatically learned from the implicit structure of an annotated training corpus .

This is much easier and can quickly lead to a model which produces results with a ' reasonably ' good quality .

Obviously , ' reasonably good quality ' is not the ultimate goal .

Unfortunately , the quality that can be reached for a given task is limited , and not merely by the potential of the learning method used .

Other limiting factors are the power of the hard-and software used to implement the learning method and the availability of training material .

Because of these limitations , we find that for most tasks we are ( at any point in time ) faced with a ceiling to the quality that can be reached with any ( then ) available machine learning system .

However , the fact that any given system can not go beyond this ceiling does not mean that machine learning as a whole is similarly limited .

A potential loophole is that each type of learning method brings its own ' inductive bias ' to the task and therefore different methods will tend to produce different errors .

In this paper , we are concerned with the question whether these differences between models can indeed be exploited to yield a data driven model with superior performance .

In the machine learning literature this approach is known as ensemble , stacked , or combined classifiers .

It has been shown that , when the errors are uncorrelated to a sufficient degree , the resulting combined classifier will often perform better than all the individual systems ( Ali and Pazzani 1996 ;

Chan and Stolfo 1995 ; Tumer and Gosh 1996 ) .

First , the combined votes will make the system more robust to the quirks of each learner 's particular bias .

Also , the use of information about each individual method 's behaviour in principle even admits the possibility to fix collective errors .

We will execute our investigation by means of an experiment .

The NLP task used in the experiment is morpho-syntactic wordclass tagging .

The reasons for this choice are several .

First of all , tagging is a widely researched and well-understood task ( cf .

Second , current performance levels on this task still leave room for improvement : ' state of the art ' performance for data driven automatic wordclass taggers ( tagging English text with single tags from a low detail tagset ) is 96 - 97 % correctly tagged words .

Finally , a number of rather different methods are available that generate a fully functional tagging system from annotated text .

Our experiment shows that , at least for the task at hand , combination of several different systems allows us to raise the performance ceiling for data driven systems .

Obviously there is still room for a closer examination of the differences between the combination methods , e.g. the question whether Memory-Based combination would have performed better if we had provided more training data than just Tune , and of the remaining errors , e.g. the effects of inconsistency in the data ( cf .

Ratnaparkhi 1996 on such effects in the Penn Treebank corpus ) .

Regardless of such closer investigation , we feel that our results are encouraging enough to extend our investigation of combination , starting with additional component taggers and selection strategies , and going on to shifts to other tagsets and/or languages .

But the investigation need not be limited to wordclass tagging , for we expect that there are many other NLP tasks where combination could lead to worthwhile improvements .


Adversarial attacks for discrete data ( such as texts ) have been proved significantly more challenging than continuous data ( such as images ) since it is difficult to generate adversarial samples with gradient-based methods .

Current successful attack methods for texts usually adopt heuristic replacement strategies on the character or word level , which remains challenging to find the optimal solution in the massive space of possible combinations of replacements while preserving semantic consistency and language fluency .

In this paper , we propose BERT-Attack , a high-quality and effective method to generate adversarial samples using pre-trained masked language models exemplified by BERT .

We turn BERT against its fine-tuned models and other deep neural models in downstream tasks so that we can successfully mislead the target models to predict incorrectly .

Our method outperforms state-of-theart attack strategies in both success rate and perturb percentage , while the generated adversarial samples are fluent and semantically preserved .

Also , the cost of calculation is low , thus possible for large-scale generations .

The code is available at https://github.com/ LinyangLee / BERT-Attack .

Despite the success of deep learning , recent works have found that these neural networks are vulnerable to adversarial samples , which are crafted with small perturbations to the original inputs ( Goodfellow et al . , 2014 ; Kurakin et al . , 2016 ; Chakraborty et al . , 2018 ) .

That is , these adversarial samples are imperceptible to human judges while they can mislead the neural networks to incorrect predictions .

Therefore , it is essential to explore these adversarial attack methods since the ultimate goal is to make sure the neural networks are highly reliable and robust .

While in computer vision fields , both attack strategies and their defense countermeasures are well-explored ( Chakraborty et al . , 2018 ) , the adversarial attack for text is still challenging due to the discrete nature of languages .

Generating of adversarial samples for texts needs to possess such qualities : ( 1 ) imperceptible to human judges yet misleading to neural models ; ( 2 ) fluent in grammar and semantically consistent with original inputs .

Previous methods craft adversarial samples mainly based on specific rules ( Li et al . , 2018 ; Gao et al . , 2018 ; Yang et al . , 2018 ; Alzantot et al . , 2018 ; Ren et al . , 2019 ; Jin et al . , 2019 ; Zang et al . , 2020 ) .

Therefore , these methods are difficult to guarantee the fluency and semantically preservation in the generated adversarial samples at the same time .

Plus , these manual craft methods are rather complicated .

They use multiple linguistic constraints like NER tagging or POS tagging .

Introducing contextualized language models to serve as an automatic perturbation generator could make these rules designing much easier .

The recent rise of pre-trained language models , such as BERT ( Devlin et al . , 2018 ) , push the performances of NLP tasks to a new level .

On the one hand , the powerful ability of a fine-tuned BERT on downstream tasks makes it more challenging to be adversarial attacked ( Jin et al . , 2019 ) .

On the other hand , BERT is a pre-trained masked language model on extremely large-scale unsupervised data and has learned general-purpose language knowledge .

Therefore , BERT has the potential to generate more fluent and semantic-consistent substitutions for an input text .

Naturally , both the properties of BERT motivate us to explore the possibility of attacking a fine-tuned BERT with another BERT as the attacker .

In this paper , we propose an effective and high-quality adversarial sample generation method : BERT-Attack , using BERT as a language model to generate adversarial samples .

The core algorithm of BERT-Attack is straightforward and consists of two stages : finding the vulnerable words in one given input sequence for the target model ; then applying BERT in a semantic-preserving way to generate substitutes for the vulnerable words .

With the ability of BERT , the perturbations are generated considering the context around .

Therefore , the perturbations are fluent and reasonable .

We use the masked language model as a perturbation generator and find perturbations that maximize the risk of making wrong predictions ( Goodfellow et al . , 2014 ) .

Differently from previous attacking strategies that require traditional single-direction language models as a constraint , we only need to inference the language model once as a perturbation generator rather than repeatedly using language models to score the generated adversarial samples in a trial and error process .

Experimental results show that the proposed BERT-Attack method successfully fooled its finetuned downstream model with the highest attack success rate compared with previous methods .

Meanwhile , the perturb percentage and the query number are considerably lower , while the semantic preservation is high .

• We propose a simple and effective method , named BERT-Attack , to effectively generate fluent and semantically-preserved adversarial samples that can successfully mislead stateof-the-art models in NLP , such as fine-tuned BERT for various downstream tasks .

• BERT-Attack has a higher attacking success rate and a lower perturb percentage with fewer access numbers to the target model compared with previous attacking algorithms , while does not require extra scoring models therefore extremely effective .

In this work , we propose a high-quality and effective method BERT-Attack to generate adversarial samples using BERT masked language model .

Experiment results show that the proposed method achieves a high success rate while maintaining a minimum perturbation .

Nevertheless , candidates generated from the masked language model can sometimes be antonyms or irrelevant to the original words , causing a semantic loss .

Thus , enhancing language models to generate more semantically related perturbations can be one possible solution to perfect BERT-Attack in the future .


We investigate the generation of onesentence Wikipedia biographies from facts derived from Wikidata slot-value pairs .

We train a recurrent neural network sequence-to-sequence model with attention to select facts and generate textual summaries .

Our model incorporates a novel secondary objective that helps ensure it generates sentences that contain the input facts .

The model achieves a BLEU score of 41 , improving significantly upon the vanilla sequence-to-sequence model and scoring roughly twice that of a simple template baseline .

Human preference evaluation suggests the model is nearly as good as the Wikipedia reference .

Manual analysis explores content selection , suggesting the model can trade the ability to infer knowledge against the risk of hallucinating incorrect information .

Despite massive effort , Wikipedia and other collaborative knowledge bases ( KBs ) have coverage and quality problems .

Popular topics are covered in great detail , but there is a long tail of specialist topics with little or no text .

Other text can be incorrect , whether by accident or vandalism .

We report on the task of generating textual summaries for people , mapping slot-value facts to onesentence encyclopaedic biographies .

In addition to initialising stub articles with only structured data , the resulting model could be used to improve consistency and accuracy of existing articles .

Figure 1 shows a Wikidata entry for Mathias Tuomi , with fact keys and values flattened into a sequence , and the first sentence from his Wikipedia article .

Some values are in the text , others are missing TITLE mathias tuomi SEX OR GENDER male DATE OF BIRTH 1985 - 09 - 03 OCCUPATION squash player CITIZENSHIP finland ( e.g. male ) or expressed differently ( e.g. dates ) .

We treat this knowlege-to-text task like translation , using a recurrent neural network ( RNN ) sequence-to-sequence model ( Sutskever et al . , 2014 ) that learns to select and realise the most salient facts as text .

This includes an attention mechanism to focus generation on specific facts , a shared vocabulary over input and output , and a multi-task autoencoding objective for the complementary extraction task .

We create a reference dataset comprising more than 400,000 knowledgetext pairs , handling the 15 most frequent slots .

We also describe a simple template baseline for comparison on BLEU and crowd-sourced human preference judgements over a heldout TEST set .

Our model obtains a BLEU score of 41.0 , compared to 33.1 without the autoencoder and 21.1 for the template baseline .

In a crowdsourced preference evaluation , the model outperforms the baseline and is preferred 40 % of the time to the Wikipedia reference .

Manual analysis of content selection suggests that the model can infer knowledge but also makes mistakes , and that the autoencoding objective encourages the model to select more facts without increasing sentence length .

The task formulation and models are a foundation for text completion and consistency in KBs .

We present a neural model for mapping between structured and unstructured data , focusing on creating Wikipedia biographic summary sentences from Wikidata slot-value pairs .

We introduce a sequence-to-sequence autoencoding RNN which improves upon base models by jointly learning to generate text and reconstruct facts .

Our analysis of the task suggests evaluation in this domain is challenging .

In place of a single score , we analyse statistical measures , human preference judgements and manual annotation to help characterise the task and understand system performance .

In the human preference evaluation , our best model outperforms template baselines and is preferred 40 % of the time to the gold standard Wikipedia reference .

Code and data is available at https:// github.com / andychisholm / mimo .