Domain adaptation is an important problem in named entity recognition ( NER ) . NER classifiers usually lose accuracy in the domain transfer due to the different data distribution between the source and the target domains . The major reason for performance degrading is that each entity type often has lots of domainspecific term representations in the different domains . The existing approaches usually need an amount of labeled target domain data for tuning the original model . However , it is a labor-intensive and time-consuming task to build annotated training data set for every target domain . We present a domain adaptation method with latent semantic association ( LaSA ) . This method effectively overcomes the data distribution difference without leveraging any labeled target domain data . LaSA model is constructed to capture latent semantic association among words from the unlabeled corpus . It groups words into a set of concepts according to the related context snippets . In the domain transfer , the original term spaces of both domains are projected to a concept space using LaSA model at first , then the original NER model is tuned based on the semantic association features . Experimental results on English and Chinese corpus show that LaSA-based domain adaptation significantly enhances the performance of NER . Named entities ( NE ) are phrases that contain names of persons , organizations , locations , etc . NER is an important task in information extraction and natural language processing ( NLP ) applications . Supervised learning methods can effectively solve NER problem by learning a model from manually labeled data ( Borthwick , 1999 ; Sang and Meulder , 2003 ; Gao et al . , 2005 ; Florian et al . , 2003 ) . However , empirical study shows that NE types have different distribution across domains ( Guo et al . , 2006 ) . Trained NER classifiers in the source domain usually lose accuracy in a new target domain when the data distribution is different between both domains . Domain adaptation is a challenge for NER and other NLP applications . In the domain transfer , the reason for accuracy loss is that each NE type often has various specific term representations and context clues in the different domains . For example , { " economist " , " singer " , " dancer " , " athlete " , " player " , " philosopher " , ... } are used as context clues for NER . However , the distribution of these representations are varied with domains . We expect to do better domain adaptation for NER by exploiting latent semantic association among words from different domains . Some approaches have been proposed to group words into " topics " to capture important relationships between words , such as Latent Semantic Indexing ( LSI ) ( Deerwester et al . , 1990 ) , probabilistic Latent Semantic Indexing ( pLSI ) ( Hofmann , 1999 ) , Latent Dirichlet Allocation ( LDA ) ( Blei et al . , 2003 ) . These models have been successfully employed in topic modeling , dimensionality reduction for text categorization ( Blei et al . , 2003 ) , ad hoc IR ( Wei and Croft . , 2006 ) , and so on . In this paper , we present a domain adaptation method with latent semantic association . We focus on capturing the hidden semantic association among words in the domain adaptation . We introduce the LaSA model to overcome the distribution difference between the source domain and the target domain . LaSA model is constructed from the unlabeled corpus at first . It learns latent semantic association among words from their related context snippets . In the domain transfer , words in the corpus are associated with a low-dimension concept space using LaSA model , then the original NER model is tuned using these generated semantic association features . The intuition behind our method is that words in one concept set will have similar semantic features or latent semantic association , and share syntactic and semantic context in the corpus . They can be considered as behaving in the same way for discriminative learning in the source and target domains . The proposed method associates words from different domains on a semantic level rather than by lexical occurrence . It can better bridge the domain distribution gap without any labeled target domain samples . Experimental results on English and Chinese corpus show that LaSA-based adaptation significantly enhances NER performance across domains . The rest of this paper is organized as follows . Section 2 briefly describes the related works . Section 3 presents a domain adaptation method based on latent semantic association . Section 4 illustrates how to learn LaSA model from the unlabeled corpus . Section 5 shows experimental results on large-scale English and Chinese corpus across domains , respectively . The conclusion is given in Section 6 . We present a domain adaptation method with LaSA model in this paper . LaSA model captures latent semantic association among words from the unlabeled corpus . It better groups words into a set of concepts according to the related context snippets . LaSAbased domain adaptation method projects words to a low-dimension concept feature space in the transfer . It effectively overcomes the data distribution gap across domains without using any labeled target domain data . Experimental results on English and Chinese corpus show that LaSA-based domain adaptation significantly enhances the performance of NER across domains . Especially , LaSA model effectively increases the accuracy of each NE type recognition in the domain transfer . Moreover , LaSA-based domain adaptation method works well across languages . To further reduce the accuracy loss , we will explore informative sampling to capture fine-grained data difference in the domain transfer .

Named entity recognition classifiers lose accuracy in domain transfers because each entity type has domain-specific term representations and existing approaches require expensive labeled data.

They propose to capture latent semantic associations among words in the unlabeled corpus and use them to tune original named entity models.

The proposed model improves the performance on the English and Chinese corpus across domains especially on each NE type recognition.


Manual evaluation of translation quality is generally thought to be excessively time consuming and expensive . We explore a fast and inexpensive way of doing it using Amazon 's Mechanical Turk to pay small sums to a large number of non-expert annotators . For $ 10 we redundantly recreate judgments from a WMT08 translation task . We find that when combined non-expert judgments have a high-level of agreement with the existing gold-standard judgments of machine translation quality , and correlate more strongly with expert judgments than Bleu does . We go on to show that Mechanical Turk can be used to calculate human-mediated translation edit rate ( HTER ) , to conduct reading comprehension experiments with machine translation , and to create high quality reference translations . Conventional wisdom holds that manual evaluation of machine translation is too time-consuming and expensive to conduct . Instead , researchers routinely use automatic metrics like Bleu ( Papineni et al . , 2002 ) as the sole evidence of improvement to translation quality . Automatic metrics have been criticized for a variety of reasons ( Babych and Hartley , 2004 ; Callison-Burch et al . , 2006 ; Chiang et al . , 2008 ) , and it is clear that they only loosely approximate human judgments . Therefore , having people evaluate translation output would be preferable , if it were more practical . In this paper we demonstrate that the manual evaluation of translation quality is not as expensive or as time consuming as generally thought . We use Amazon 's Mechanical Turk , an online labor market that is designed to pay people small sums of money to complete human intelligence teststasks that are difficult for computers but easy for people . We show that : • Non-expert annotators produce judgments that are very similar to experts and that have a stronger correlation than Bleu . • Mechanical Turk can be used for complex tasks like human-mediated translation edit rate ( HTER ) and creating multiple reference translations . • Evaluating translation quality through reading comprehension , which is rarely done , can be easily accomplished through creative use of Mechanical Turk . 2 Related work Snow et al . ( 2008 ) examined the accuracy of labels created using Mechanical Turk for a variety of natural language processing tasks . These tasks included word sense disambiguation , word similarity , textual entailment , and temporal ordering of events , but not machine translation . Snow et al . measured the quality of non-expert annotations by comparing them against labels that had been previously created by expert annotators . They report inter-annotator agreement between expert and non-expert annotators , and show that the average of many non-experts converges on performance of a single expert for many of their tasks . Although it is not common for manual evaluation results to be reported in conference papers , several large-scale manual evaluations of machine translation quality take place annually . These include public forums like the NIST MT Evaluation Workshop , IWSLT and WMT , as well as the project-specific Go / No Go evaluations for the DARPA GALE program . Various types of human judgments are used . NIST collects 5-point fluency and adequacy scores ( LDC , 2005 ) , IWSLT and WMT collect relative rankings ( Callison-Burch et al . , 2008 ; Paul , 2006 ) , and DARPA evaluates using HTER ( Snover et al . , 2006 ) . The details of these are provided later in the paper . Public evaluation campaigns provide a ready source of goldstandard data that non-expert annotations can be compared to . Mechanical Turk is an inexpensive way of gathering human judgments and annotations for a wide variety of tasks . In this paper we demonstrate that it is feasible to perform manual evaluations of machine translation quality using the web service . The low cost of the non-expert labor found on Mechanical Turk is cheap enough to collect redundant annotations , which can be utilized to ensure translation quality . By combining the judgments of many non-experts we are able to achieve the equivalent quality of experts . The suggests that manual evaluation of translation quality could be straightforwardly done to validate performance improvements reported in conference papers , or even for mundane tasks like tracking incremental system updates . This challenges the conventional wisdom which has long held that automatic metrics must be used since manual evaluation is too costly and timeconsuming . We have shown that Mechanical Turk can be used creatively to produce quite interesting things . We showed how a reading comprehension test could be created , administered , and graded , with only very minimal intervention . We believe that it is feasible to use Mechanical Turk for a wide variety of other machine translated tasks like creating word alignments for sentence pairs , verifying the accuracy of document-and sentence-alignments , performing non-simulated active learning experiments for statistical machine translation , even collecting training data for low resource languages like Urdu . The cost of using Mechanical Turk is low enough that we might consider attempting quixotic things like human-in-the-loop minimum error rate training ( Zaidan and Callison-Burch , 2009 ) , or doubling the amount of training data available for Urdu .

Because of the high cost required for manual evaluation, most works rely on automatic evaluation metrics although there are several drawbacks.

They investigate whether judgements by non-experts from Amazon's Mechanical Turk can be a fast and inexpensive means of evaluation for machine translation systems.

They found that non-expert judgements with high agreement correlate better with gold standard judgements than BLEU while keeping the cost low.


State-of-the-art networks that model relations between two pieces of text often use complex architectures and attention . In this paper , instead of focusing on architecture engineering , we take advantage of small amounts of labelled data that model semantic phenomena in text to encode matching features directly in the word representations . This greatly boosts the accuracy of our reference network , while keeping the model simple and fast to train . Our approach also beats a tree kernel model that uses similar input encodings , and neural models which use advanced attention and compare-aggregate mechanisms . Modeling a match between pieces of text is at the core of many NLP tasks . Recently , manual feature engineering methods have been shadowed by neural network approaches . These networks model the interaction of two pieces of text , or word-toword interactions across sentences , using sophisticated attention mechanisms ( Wang et al . , 2016a ; Santos et al . , 2016 ) and compare-aggregate frameworks ( He and Lin , 2016 ; Wang et al . , 2017 ) . Architectural complexity is tied to longer training times 1 . Meaningful features may take long time to emerge by only leveraging word representations and the training data of the task at hand . This is especially problematic with little data , as it often happens in question answering ( QA ) tasks , e.g. , answer sentence selection ( Wang et al . , 2007 ; Yang et al . , 2015 ) . Thus , effective word representations are crucial in neural network models to get state-of-the-art performance . In this work , we try to answer the following research questions : ( i ) in addition to lexical links , can we incorporate higher-level semantic links between the words in a question and a candidate answer passage , and ( ii ) can we show that such information has an impact on the quality of our model , and also allows us to keep the architecture simple ? We show that modeling semantic relations improves the performance of a neural network for answer sentence selection with ( i ) a little number of semantic annotations , and ( ii ) a little increase in training time w.r.t . more complex architecture . In this paper , we presented a neural network that models semantic links between questions and answers , in addition to lexical links . The annotations for establishing such links are produced by a set of fast neural components for question analysis , trained on publicly available datasets . The evaluation on two QA datasets shows that our approach can achieve state-of-the-art performance using a simple CNN , leading to a low complexity and training time . Our approach is an interesting first step towards a future architecture , in which we will jointly optimize the semantic annotators and the answer sentence selection model , in an end-to-end fashion .

State-of-the-art models that model relations between two texts use complex architectures and attention which requires a long time and large data at training.

They propose a method that directly models higher-level semantic links between two texts that are annotated by a fast model.

The proposed model outperforms a tree kernel model and complex neural models while keeping the model simple and the training fast.


Language representations are known to carry certain associations ( e.g. , gendered connotations ) which may lead to invalid and harmful predictions in downstream tasks . While existing methods are effective at mitigating such unwanted associations by linear projection , we argue that they are too aggressive : not only do they remove such associations , they also erase information that should be retained . To address this issue , we propose OS-CAR ( Orthogonal Subspace Correction and Rectification ) , a balanced approach of mitigation that focuses on disentangling associations between concepts that are deemed problematic , instead of removing concepts wholesale . We develop new measurements for evaluating information retention relevant to the debiasing goal . Our experiments on genderoccupation associations show that OSCAR is a well-balanced approach that ensures that semantic information is retained in the embeddings and unwanted associations are also effectively mitigated . Keeping in mind that removing bias and retaining information have to be done in synergy , we present how to obtain aggregated measurements for these two components . We will first describe the de- Word embeddings are used extensively across natural language processing ( NLP ) and succinctly capture not only the syntactic and semantic structure of language , but also word meaning in context . As such , word embeddings are essential building blocks for today 's state-of-the-art in NLP . But they are also known to capture a significant amount of stereotypical associations ( e.g. , Bolukbasi et al . , 2016 ; Zhao et al . , 2017 ; Dev and Phillips , 2019 ; Sun et al . , 2019 ) related to gender , race , nationality , or religion , which can manifest in unwanted and/or potentially harmful ways in downstream tasks ( Webster et al . , 2018 ; Zhao et al . , 2019 ; Dev et al . , 2020 ) . Such potentially problematic associations , when embedded in word representations , can lead to incorrect and unfair decisions about large groups of people . While the term " bias " has many meanings , in this paper we use it to refer to these unwanted stereotypical associations . Existing methods to mitigate these effects either require expensive retraining of vectors ( Lemoine et al . , 2018 ) which can be inefficient , or projecting out information contained along an entire subspace representing a protected concept ( such as gender or race ) in the embedding space ( e.g. , Bolukbasi et al . , 2016 ; Dev and Phillips , 2019 ; Ravfogel et al . , 2020 ) . Projective approaches are difficult to control as they are either insufficient : removing a subspace can still leave residual bias ( Gonen and Goldberg , 2019 ; Lauscher et al . , 2020 ) , or too aggressive : in the case of gender , also unnecessarily altering the association between the word pregnant and words like female and mother . In tasks such as coreference resolution , removing such associations could hinder reference resolution . To quantify how much valid information is retained , we look at the output space of models . Following Dev et al . ( 2020 ) , we use Natural Language Inference ( NLI ) as an effective quantitative probe . Here , we construct the hypothesis by making minimal edits to a premise , and observe model prediction conditioned on such changes . For example , Premise : A matriarch sold a watch . Hypothesis : A woman sold a watch . Here , the objective is to determine if the hypothesis is entailed by the premise , contradicted by it , or neither ( neutral to it ) . A GloVe-based NLI ( Parikh et al . , 2016 ) model , without any explicit form of bias mitigation , predicts label entail with a high probability of 97 % ; the notion of a matriarch being a woman is correctly identified by the model . However , after projective debiasing , the model classifies the pair as neutral with a probability 62 % while the probability of the label entail drops to much lower at 16 % . That is , aggressive mitigation of gender representations erases valid gender associations . Ideally , we should correct problematic associations without erasing valid ones . To this end , we propose OSCAR ( Othogonal Subspace Correction and Rectification ) which orthogonalizes and rectifies identified subspaces of concepts that are incorrectly associated in an embedding space . Embeddings outside the subspace are stretched in a graded manner or untouched . Our contributions are : 1 . We argue that mitigating unwanted stereotypical associations should go beyond information removal ( e.g. , projecting out features ) , and should also preserve pertinent associations . 2 . We present OSCAR1 , a completely different method from the existing projective approaches ; it uses orthogonalization of subspaces desired not to have interdependence , and so minimal change is made to embeddings to prevent loss of desired associations . 3 . We develop a combination of tests based on NLI to evaluate both bias mitigation and information retention . 4 . Our experiments show that OSCAR is a wellbalanced approach that mitigates biases as good as projective approaches while retaining more valid associations . Our contributions , focusing specifically on representations rather than classifiers , are important given the preponderance of distributed representations of text across NLP . Predictions from systems that use such representations , if unchecked , could lead to real-world decisions ( involving , e.g. hiring ) that " systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others " ( Friedman and Nissenbaum , 1996 ) . In an effort to prevent such transition of representational harms into allocational harms ( cf . Crawford , 2017 ; Abbasi et al . , 2019 ; Blodgett et al . , 2020 ) , we look at mitigating stereotyping biases at the source , i.e. the embedding space . 2 Bias , Gender and NLP Bias and Related Work . Social biases in tasks using machine learning have the potential to cause harms ( Barocas and Selbst , 2016 ) and are widely studied . Such biases in language technologies have been detected ( Bolukbasi et al . , 2016 ; Gonen and Goldberg , 2019 ) , measured ( Caliskan et al . , 2017 ; Webster et al . , 2018 ; Lauscher et al . , 2020 ) and mitigated ( Zhao et al . , 2018 ; Ravfogel et al . , 2020 ) . Most work focuses on gender bias , and in particular , the stereotypical associations of occupations to males and females ( Rudinger et al . , 2018 ; De-Arteaga et al . , 2019 ; Gaut et al . , 2019 ; Dev et al . , 2020 ) . In this work too , bias refers to stereotypical associations in language representations . Treatment of Gender . The concept of gender is a complex one ( cf . Larson , 2017 ; Dev et al . , 2021 ) . In our experiments , we identify a subspace in the representation associated with gender , typically derived by terms specifically associated with notions of male and female . This aligns with prominent recent NLP research ( Caliskan et al . , 2017 ; Bolukbasi et al . , 2016 ; Dev and Phillips , 2019 ; Ravfogel et al . , 2020 ) that try to measure and mitigate gender bias ( either implicitly or explicitly ) , directly focusing on male versus female associations . The underlying goal of this work is to disassociate two concepts . We mainly use concepts male-vsfemale and occupations to demonstrate this . There is explicit and measurable bias encoded in word vector representations ( Bolukbasi et al . , 2016 ) by the male-vs-female relation to occupations , and standard evaluation metrics ( Caliskan et al . , 2017 ; Dev et al . , 2020 ) are designed to measure this . As such many of our experiments are geared towards these male-vs-female issues . The notion of gender is not restricted to male and female . But because " conceptualizations of gender as binary are reinforced linguistically in English through pronoun conventions " ( Blodgett , 2021 ) , gender notions beyond male and female are either missing or poorly represented . As a result , their associations with other concepts are also not robust . Quantifying this effect , and evaluating its attenuation , is beyond the scope of this paper ; however the techniques we study for disassociating two concepts are amenable to such tasks in general . We remark that a potential harm towards nonmale / female people arises through omission ( Cao and Daumé III , 2020 ) , and as a result , we explicitly encourage further research on this topic . sign of our evaluation dataset in § 3.1 , then present evaluation metrics . Then , we will discuss how to measure biases in § 3.2 and information retention in § 3.3 . To be comprehensive , we will use metrics that are both intrinsic to embedding geometry and extrinsic ones that focus on downstream NLI model outputs . The propagation of undesirable and stereotypical associations learned from data into decisions made by language models maintains a vicious cycle of biases . Combating biases before deploying representations is thus extremely vital . But this poses its own challenges . Word embeddings capture a lot of information implicitly in relatively few dimensions . These implicit associations are what makes them state-of-the-art at tackling different language modeling tasks . Breaking down these associations for bias rectification , thus , has to be done carefully so as to not destroy the structure of the embeddings . OSCAR 's rectification of associations helps integrate both these aspects , allowing it to be more suitable at making word embeddings more usable . Moreover , being computationally lightweight and sub-differentiable , it is simple to apply adaptively without extensive retraining . OSCAR dissociates concepts which may be otherwise lost or overlapped in distributed representations by re-orthogonalizing them . We envision that this method can be used for many types of unwanted associations ( age-ability , religion-virtue perception , etc ) . Since it only decouples specific associations , informative components of these features will remain unchanged . All the experiments in this paper are based on English embeddings , models and tests ; extending our analysis to other languages is important future work . Further , we be-lieve OSCAR can extend in a straightforward way beyond NLP to other distributed , vectorized representations ( e.g. , of images , graphs , or spatial data ) which can also exhibit stereotypical associations .

Existing methods that remove harmful stereotypical associations from word embeddings either require inefficient retraining or remove information which should be retained.

They propose a method which orthogonalizes and rectifies incorrectly associated subspaces of concepts in an embedding space and a metric for evaluating information retention.

NLI-based evaluation on gender-occupation associations shows that the proposed approach is well-balanced ensuring semantic information is retained in the embeddings while mitigating biases.


Humor plays an important role in human languages and it is essential to model humor when building intelligence systems . Among different forms of humor , puns perform wordplay for humorous effects by employing words with double entendre and high phonetic similarity . However , identifying and modeling puns are challenging as puns usually involved implicit semantic or phonological tricks . In this paper , we propose Pronunciation-attentive Contextualized Pun Recognition ( PCPR ) to perceive human humor , detect if a sentence contains puns and locate them in the sentence . PCPR derives contextualized representation for each word in a sentence by capturing the association between the surrounding context and its corresponding phonetic symbols . Extensive experiments are conducted on two benchmark datasets . Results demonstrate that the proposed approach significantly outperforms the state-of-the-art methods in pun detection and location tasks . In-depth analyses verify the effectiveness and robustness of PCPR . During the last decades , social media has promoted the creation of a vast amount of humorous web contents ( Nijholt et al . , 2017 ) . Automatic recognition of humor has become an important task in the area of figurative language processing , which can benefit various downstream NLP applications such as dialogue systems , sentiment analysis , and machine translation ( Melby and Warner , 1995 ; Augello et al . , 2008 ; Ghosh et al . , 2015 ; Bertero and Fung , 2016 ; Blinov et al . , 2019 ) . However , humor is one of the most complicated behaviors in natural language semantics and sometimes it is even difficult for humans to interpret . In most cases , understanding humor requires adequate background knowledge and a rich context . Puns are a form of humorous approaches using the different meanings of identical words or words with similar pronunciations to explain texts or utterances . There are two main types of puns . Homographic puns rely on multiple interpretations of the same word . As shown in Table 1 , the phrase all right means good condition or opposite to left ; the word reaction means chemical change or action . The two meanings of the same expression are consistent with its context , which creates a humorous pun in both sentences when there is a clear contrast between two meanings . On the other hand , heterographic puns take advantage of phonologically same or similar words . For example , the word pairs sale and sail , weak and week in Table 1 have the same or similar pronunciations . The sentences are funny because both words fit the same context . Understanding puns is a big fish to fry for deep comprehension of complex semantics . These two forms of puns have been studied in literature from different angles . To recognize puns in a sentence , word sense disambiguation techniques ( WSD ) ( Navigli , 2009 ) have been employed to identify the equitable intention of words in utterances ( Pedersen , 2017 ) . External knowledge bases such as WordNet ( Miller , 1998b ) have been applied in determining word senses of pun words ( Oele and Evang , 2017 ) . However , these methods can not tackle heterographic puns with distinct word spellings and knowledge bases that only contain a limited vocabulary . To resolve the issues of sparseness and heterographics , the word embedding techniques ( Mikolov et al . , 2013 ; Pennington et al . , 2014 ) provide flexible representations to model puns ( Hurtado et al . , 2017 ; Indurthi and Oota , 2017 ; Cai et al . , 2018 ) . However , a word may have different meanings regarding its contexts . Especially , an infrequent meaning of the word might be utilized for creating a pun . Therefore , static word embeddings are insufficient to represent words . In addition , some puns are created by replacing a word with another word with the same or similar pronunciation as examples shown in Table 1 . Therefore , to recognize puns , it is essential to model the association between words in the sentence and the pronunciation of words . Despite existing approaches attempt to leverage phonological structures to understand puns ( Doogan et al . , 2017 ; Jaech et al . , 2016 ) , there is a lack of a general framework to model these two types of signals in a whole . In this paper , we propose Pronunciation-attentive Contextualized Pun Recognition ( PCPR ) to jointly model the contextualized word embeddings and phonological word representations for pun recognition . To capture the phonological structures of words , we break each word into a sequence of phonemes as its pronunciation so that homophones can have similar phoneme sets . For instance , the phonemes of the word pun are { P , AH , N } . In PCPR , we construct a pronunciation attentive module to identify important phonemes of each word , which can be applied in other tasks related to phonology . We jointly encode the contextual and phonological features into a self-attentive embedding to tackle both pun detection and location tasks . We summarize our contributions as following . • To the best of our knowledge , PCPR is the first work to jointly model contextualized word embeddings and pronunciation embeddings to recognize puns . Both contexts and phonological properties are beneficial to pun recognition . In this paper , we propose a novel approach , PCPR , for pun detection and location by leveraging a contextualized word encoder and modeling phonemes as word pronunciations . Moreover , we would love to apply the proposed model to other problems , such as general humor recognition , irony discovery , and sarcasm detection , as the future work .

Puns involve implicit semantic or phonological tricks however there is no general framework to model these two types of signals as a whole.

They propose to jointly model contextualized word embeddings and phonological word representations by breaking each word into a sequence of phonemes for pun detection.

The proposed approach outperforms the state-of-the-art methods in pun detection and location tasks.


Part-of-speech ( POS ) tag distributions are known to exhibit sparsity -a word is likely to take a single predominant tag in a corpus . Recent research has demonstrated that incorporating this sparsity constraint improves tagging accuracy . However , in existing systems , this expansion come with a steep increase in model complexity . This paper proposes a simple and effective tagging method that directly models tag sparsity and other distributional properties of valid POS tag assignments . In addition , this formulation results in a dramatic reduction in the number of model parameters thereby , enabling unusually rapid training . Our experiments consistently demonstrate that this model architecture yields substantial performance gains over more complex tagging counterparts . On several languages , we report performance exceeding that of more complex state-of-the art systems . 1 Since the early days of statistical NLP , researchers have observed that a part-of-speech tag distribution exhibits " one tag per discourse " sparsity -words are likely to select a single predominant tag in a corpus , even when several tags are possible . Simply assigning to each word its most frequent associated tag in a corpus achieves 94.6 % accuracy on the WSJ portion of the Penn Treebank . This distributional sparsity of syntactic tags is not unique to English 1 The source code for the work presented in this paper is available at http://groups.csail.mit.edu / rbg / code / typetagging/. -similar results have been observed across multiple languages . Clearly , explicitly modeling such a powerful constraint on tagging assignment has a potential to significantly improve the accuracy of an unsupervised part-of-speech tagger learned without a tagging dictionary . In practice , this sparsity constraint is difficult to incorporate in a traditional POS induction system ( Mérialdo , 1994 ; Johnson , 2007 ; Gao and Johnson , 2008 ; Grac ¸a et al . , 2009 ; Berg-Kirkpatrick et al . , 2010 ) . These sequence models-based approaches commonly treat token-level tag assignment as the primary latent variable . By design , they readily capture regularities at the token-level . However , these approaches are ill-equipped to directly represent type-based constraints such as sparsity . Previous work has attempted to incorporate such constraints into token-level models via heavy-handed modifications to inference procedure and objective function ( e.g. , posterior regularization and ILP decoding ) ( Grac ¸a et al . , 2009 ; Ravi and Knight , 2009 ) . In most cases , however , these expansions come with a steep increase in model complexity , with respect to training procedure and inference time . In this work , we take a more direct approach and treat a word type and its allowed POS tags as a primary element of the model . The model starts by generating a tag assignment for each word type in a vocabulary , assuming one tag per word . Then , tokenlevel HMM emission parameters are drawn conditioned on these assignments such that each word is only allowed probability mass on a single assigned tag . In this way we restrict the parameterization of a token-level HMM to reflect lexicon sparsity . This model admits a simple Gibbs sampling algorithm where the number of latent variables is proportional to the number of word types , rather than the size of a corpus as for a standard HMM sampler ( Johnson , 2007 ) . There are two key benefits of this model architecture . First , it directly encodes linguistic intuitions about POS tag assignments : the model structure reflects the one-tag-per-word property , and a typelevel tag prior captures the skew on tag assignments ( e.g. , there are fewer unique determiners than unique nouns ) . Second , the reduced number of hidden variables and parameters dramatically speeds up learning and inference . We evaluate our model on seven languages exhibiting substantial syntactic variation . On several languages , we report performance exceeding that of state-of-the art systems . Our analysis identifies three key factors driving our performance gain : 1 ) selecting a model structure which directly encodes tag sparsity , 2 ) a type-level prior on tag assignments , and 3 ) a straightforward naïve-Bayes approach to incorporate features . The observed performance gains , coupled with the simplicity of model implementation , makes it a compelling alternative to existing more complex counterparts . We have presented a method for unsupervised partof-speech tagging that considers a word type and its allowed POS tags as a primary element of the model . This departure from the traditional token-based tagging approach allows us to explicitly capture typelevel distributional properties of valid POS tag as-signments as part of the model . The resulting model is compact , efficiently learnable and linguistically expressive . Our empirical results demonstrate that the type-based tagger rivals state-of-the-art tag-level taggers which employ more sophisticated learning mechanisms to exploit similar constraints . In this paper , we make a simplifying assumption of one-tag-per-word . This assumption , however , is not inherent to type-based tagging models . A promising direction for future work is to explicitly model a distribution over tags for each word type . We hypothesize that modeling morphological information will greatly constrain the set of possible tags , thereby further refining the representation of the tag lexicon .

Assuming there is only one tag for a word is a powerful heuristic for Part-of-speech tagging but incorporating this into a model leads to complexity.

They propose an unsupervised method that directly incorporates a one-tag-per-word assumption into a HMM-based model.

Their proposed method reduces the number of model parameters which results in faster training speed and also outperforms more complex systems.


We present a game-theoretic model of bargaining over a metaphor in the context of political communication , find its equilibrium , and use it to rationalize observed linguistic behavior . We argue that game theory is well suited for modeling discourse as a dynamic resulting from a number of conflicting pressures , and suggest applications of interest to computational linguists . A 13 Dec 1992 article in The Times starts thus : The European train chugged out of the station last night ; for most of the day it looked as if it might be stalled there for some time . It managed to pull away at around 10:30 pm only after the Spanish prime minister , Felipe Gonzalez , forced the passengers in the first class carriages into a last minute whip round to sweeten the trip for the European Community 's poor four : Spain , Portugal , Greece and Ireland . The fat controller , Helmut Kohl , beamed with satisfaction as the deal was done . The elegantlysuited Francois Mitterrand was equally satisfied . But nobody was as pleased as John Major , stationmaster for the UK presidency , for whom the agreement marked a scarce high point in a battered premiership . The departure had actually been delayed by seven months by Danes on the line . Just when that problem was solved , there was the voluble outbreak , orchestrated by Spain , from the poor four passengers demanding that they should travel free and be given spending money , too . The coupling of the carriages may not be reliably secure but the pan-European express is in motion . That few seem to agree the destination suggests that future arguments are inevitable at every set of points . Next stop : Copenhagen . Apart from an entertaining read , the extended metaphor provides an elaborate conceptual correspondence between a familiar domain of train journeys and the unfolding process of European integration . Carriages are likened to nation states ; passengers to their peoples ; treaties to stations ; politicians to responsible rail company employees . In a compact form , the metaphor gives expression to both the small and the large scale of the process . It provides for the recent history : Denmark 's failure to ratify the 1992 Maastricht treaty until opt-outs were negotiated later that year is compared to dissenters sabotaging the journey by laying on the tracks ( Danes on the line ) ; negotiations over the Cohesion Fund that would provide less developed regions with financial aid to help them comply with convergence criteria are likened to second class carriages with poor passengers for whom the journey had to be subsidized . At a more general level , the European integration is a purposeful movement towards some destination according to a worked out plan , getting safely through negotiation and implementation from one treaty to another , as a train moving on its rails through subsequent stations , with each nation being separate yet tied with everyone else . Numerous inferences regarding speed , timetables , stations , passengers , different classes of tickets , temporary obstacles on the tracks , and so on can be made by the reader based on the knowledge of train journeys , giving him or her a feeling of an enhanced understanding1 of the highly complex process of European integration . So apt was the metaphor that political fights were waged over its details ( Musolff , 2000 ) . Worries about destination were given an eloquent expression by Margaret Thatcher ( Sunday Times , 20 Sept 1992 ): She warned EC leaders to stop their endless round of summits and take notice of their own people . " There is a fear that the European train will thunder forward , laden with its customary cargo of gravy , towards a destination neither wished for nor understood by electorates . But the train can be stopped , " she said . The metaphor proved flexible enough for further elaboration . John Major , a Conservative PM of Britain , spoke on June 1st , 1994 about his vision of the decision making at the EU level , saying that he had never believed that Europe must act as one on every issue , and advocating " a sensible new approach , varying when it needs to , multitrack , multi-speed , multi-layered . " He attempted to turn a largely negative Conservative take on the European train ( see Thatcher above ) into a tenable positive vision -each nation-carriage is now presumably a rather autonomous entity , waiting on a side track for the right locomotive , in a huge yet smoothly operating railroad system . Major 's political opponents offered their counter-frames . In both cases , the imagery of a large transportation system was taken up , yet turned around to suggest that " multi , for everyone " amounts to Britain being in " the slow lane , " and a different image was suggested that makes the negative evaluation of Britain 's opt-outs more poignant -a football metaphor , where relegation to the second division is a sign of a weak performance , and a school metaphor , where Britain is portrayed as an under-achiever : John Cunningham , Labour He has admitted that his Government would let Britain fall behind in Europe . He is apparently willing to offer voluntary relegation to the second division in Europe , and he is n't even prepared to put up a fight . I believe that in any two-speed Europe , Britain must be up with those in the fast lane . Clearly Mr Major does not . Paddy Ashdown , Liberal Democrat Are you really saying that the best that Britain can hope for under your leadership is ... the slow lane of a two-speed Europe ? Most people in this country will want to aim higher , and will reject your view of a ' drop-out ' Britain . The pro-European camp rallied around the " Britain in the slow lane " version as a critical stance towards the government 's European policy . Of the alternative metaphors , the school metaphor has some traction in the Euro discourse , where the European ( mainly German ) financial officers are compared to school authorities , and governments struggling to meet the strict convergence criteria to enter the Euro are compared to pupils that barely make the grade with Britain as a ' drop-out ' who gave up even trying ( Musolff , 2000 ) . The fact that European policy is being communicated and negotiated via a metaphor is not surprising ; after all , " there is always someone willing to help us think by providing us with a metaphor that accords with HIS views . " 2 From the point of view of the dynamics of political discourse , the puzzle is rather the apparent tendency of politicians to be compelled by the rival 's metaphorical framework . Thatcher tries to turn the train metaphor used by the pro-EU camp around . Yet , assuming metaphors are matters of choice , why should Thatcher feel constrained by her rival 's choice , why does n't she ignore it and merely suggest a new metaphor of her own design ? As the evidence above suggests , this is not Thatcher 's idiosyncrasy , as Major and his rivals acted similarly . Can this dynamic be explained ? In this article , we use the explanatory framework of game theory , seeking to rationalize the observed behavior by designing a game that would produce , at equilibrium , the observed dynamics . Specifically , we formalize the notion that the price of " locking " the public into a metaphorical frame of reference is that a politician is coerced into staying within the metaphor as well , even if he or she is at the receiving end of a rival 's rhetorical move . Since the use of game theory is not common in computational linguistics , we first explain its main attributes , justify our decision to make use of it , and draw connections to research questions that can benefit from its application ( section 2 ) . Next , we design the game of bargaining over a metaphor , and find its equilibrium ( section 3 ) , followed by a discussion ( section 4 ) . This article addressed a specific communicative setting ( rival politicians trying to " sell " to the public their versions of the unfolding realities and necessary policies ) and a specific linguistic tool ( an extended metaphor ) , showing that the particular use made of metaphor in such setting can be rationalized based on the characteristics of the setting . Various questions now arise . Given the central role played by the public gratification constraint in our model , would conversational situations without the need to persuade the public , such as meetings of small groups of peers or phone conversations between friends , tend less to the use of extended metaphor ? Conversely , does the use of extended metaphor in other settings testify to the existence of presumed onlookers who need to be " captured " in a particular version of reality -as in pedagogic or poetic context ? Considerations of the participants ' agendas and their impact on the ensuing dynamics of the exchange would we believe lead to further interest in game theoretic models when addressing complex social dynamics in situations like collaborative authoring , debates , or dating , and will augment the existing mostly statistical approaches with a broader picture of the relevant communication .

Metaphors used in political arguments provide elaborate conceptual correspondences with a tendency of politicians to be compelled by the rival's metaphorical framework to be explained.

They propose a game-theoric model of bargaining over a metaphor which is suitable to model its dynamics and use to rationalize observed linguistic behavior.

They show that the proposed framework can rationalize political communications with the use of extended metaphors based on the characteristics of the setting.


We examine a methodology using neural language models ( LMs ) for analyzing the word order of language . This LM-based method has the potential to overcome the difficulties existing methods face , such as the propagation of preprocessor errors in count-based methods . In this study , we explore whether the LMbased method is valid for analyzing the word order . As a case study , this study focuses on Japanese due to its complex and flexible word order . To validate the LM-based method , we test ( i ) parallels between LMs and human word order preference , and ( ii ) consistency of the results obtained using the LM-based method with previous linguistic studies . Through our experiments , we tentatively conclude that LMs display sufficient word order knowledge for usage as an analysis tool . Finally , using the LMbased method , we demonstrate the relationship between the canonical word order and topicalization , which had yet to be analyzed by largescale experiments . Model Verbs whose type-A examples prefer the ACC-DAT order Verbs whose type-B examples prefer the ACC-DAT order CLM " 預ける " ( deposit ) , " 置く " ( place ) , " 持つ " ( have ) , " 入 れる " ( put in ) , " 納める " ( pay ) , " 郵送 " ( mail ) , " 供 給 " ( supply ) , " 出す " ( put out ) , " 運ぶ " ( transport ) , " 流 す " ( shed ) , " 掛ける " ( hang ) , " 飾る " ( decorate ) , " 広 げる " ( spread ) , " 移す " ( transfer ) , " 残す " ( leave ) , " 配 送 " ( deliver ) , " 送る " ( send ) , " 投げる " ( throw ) , " 送 付 " ( send ) , " 返却 " ( return ) , " 届ける " ( send ) , " 戻す " ( return ) , " 着ける " ( wear ) , " 上げる " ( increase ) , " 落と す " ( drop ) , " 載せる " ( publish ) , " 変更 " ( change ) , " 納 入 " ( deliver ) , " 卸す " ( unload ) , " 掲載 " ( publish ) , " 通 す " ( get X through ) " 配 布 " ( distribute ) , " 渡 す " ( pass ) , " プ レ ゼ ン ト " ( present ) , " 合わせる " ( match ) , " 見せる " ( show ) , " 提 供 " ( offer ) , " 与える " ( give ) , " 当てる " ( hit ) , " 回す " ( turn ) , " 追加 " ( add ) , " 貸す " ( lend ) , " 展示 " ( exhibit ) , " 据える " ( lay ) , " 依頼 " ( request ) , " 挿入 " ( insert ) , " 纏 める " ( collect ) , " 請求 " ( claim ) SLM " 預ける " ( deposit ) , " 置く " ( place ) , " 頼む " ( ask ) , " 入 れる " ( put in ) , " 納める " ( pay ) , " 郵送 " ( mail ) , " 出 す " ( put out ) , " 運ぶ " ( transport ) , " 流す " ( shed ) , " 掛け る " ( hang ) , " 広げる " ( spread ) , " 移す " ( transfer ) , " 残 す " ( leave ) , " リクエスト " ( request ) , " 配送 " ( deliver ) , " 送る " ( send ) , " 投げる " ( throw ) , " 送付 " ( send ) , " 求 める " ( ask ) , " 提出 " ( submit ) , " 届ける " ( deliver ) , " 要 求 " ( request ) , " 戻す " ( return ) , " 寄付 " ( donate ) , " 寄贈 " ( donation ) , " 着ける " ( wear ) , " 乗せる " ( place ) , " 上げ る " ( increase ) , " 落とす " ( drop ) , " 貼る " ( stick ) , " 分け る " ( divide ) , " ばらまく " ( scatter ) , " はめる " ( fit ) , " 支 払う " ( pay ) , " 配達 " ( deliver ) , " 卸す " ( unload ) , " 纏め る " ( collect ) , " 通す " ( get X through ) " プレゼント " ( present ) , " 持つ " ( have ) , " 合わせる " ( match ) , " 見せる " ( show ) , " 向ける " ( point ) , " 提供 " ( offer ) , " 装備 " ( equip ) , " 追加 " ( add ) , " 展示 " ( exhibit ) , " 据える " ( lay ) , " 採用 " ( adopt ) Speakers sometimes have a range of options for word order in conveying a similar meaning . A typical case in English is dative alternation : ( 1 ) a. A teacher gave a student a book . b. A teacher gave a book to a student . Even for such a particular alternation , several studies ( Bresnan et al . , 2007 ; Hovav and Levin , 2008 ; Colleman , 2009 ) investigated the factors determining this word order and found that the choice is not random . For analyzing such linguistic phenomena , linguists repeat the cycle of constructing hypotheses and testing their validity , usually through psychological experiments or count-based methods . However , these approaches sometimes face difficulties , such as scalability issues in psychological experiments and the propagation of preprocessor errors in count-based methods . Compared to the typical approaches for evaluating linguistic hypotheses , approaches using LMs have potential advantages ( Section 3.2 ) . In this study , we examine the methodology of using LMs for analyzing word order ( Figure 1 ) . To validate the LM-based method , we first examine if there is a parallel between canonical word order and generation probability of LMs for each word order . Futrell and Levy ( 2019 ) reported that English LMs have human-like word order preferences , which can be one piece of evidence for validating the LM-based method . However , it is not clear whether the above assumption is valid even in languages with more flexible word order . In this study , we specifically focus on the Japanese language due to its complex and flexible word order . There are many claims on the canonical word order of Japanese , and it has attracted considerable attention from linguists and natural language processing ( NLP ) researchers for decades ( Hoji , 1985 ; Saeki , 1998 ; Miyamoto , 2002 ; Matsuoka , 2003 ; Koizumi and Tamaoka , 2004 ; Nakamoto et al . , 2006 ; Shigenaga , 2014 ; Sasano and Okumura , 2016 ; Orita , 2017 ; Asahara et al . , 2018 ) . We investigated the validity of using Japanese LMs for canonical word order analysis by conducting two sets of experiments : ( i ) comparing word order preference in LMs to that in Japanese speakers ( Section 4 ) , and ( ii ) checking the consistency Topic Time Location Subject ( Adverb ) Indirect object Direct object Verb Notation TOP TIM LOC NOM -DAT ACC -Typical particle " は " ( wa ) " に " ( ni ) " で " ( de ) " が " ( ga ) -"に " ( ni ) " を " ( o ) -Related section 6 5.2 5.2 5.2 5.3 5.1 5.1 5.1 Table 1 : Overview of the typical cases in Japanese , their typical particles , and the sections where the corresponding case is analyzed . The well-known canonical word order of Japanese is listed from left to right . between the preference of LMs with previous linguistic studies ( Section 5 ) . From our experiments , we tentatively conclude that LMs display sufficient word order knowledge for usage as an analysis tool , and further explore potential applications . Finally , we analyzed the relationship between topicalization and word order of Japanese by taking advantage of the LM-based method ( Section 6 ) . In summary , we : • Discuss and validate the use of LMs as a tool for word order analysis as well as investigate the sensitivity of LMs against different word orders in non-European language ( Section 3 ) ; • Find encouraging parallels between the results obtained with the LM-based method and those with the previously established method on various hypotheses of canonical word order of Japanese ( Sections 4 and 5 ) ; and • Showcase the advantages of an LM-based method through analyzing linguistic phenomena that is difficult to explore with the previous data-driven methods ( Section 6 ) . We have proposed to use LMs as a tool for analyzing word order in Japanese . Our experimental results support the validity of using Japanese LMs for canonical word order analysis , which has the potential to broaden the possibilities of linguistic research . From an engineering view , this study supports the use of LMs for scoring Japanese word order automatically . From the viewpoint of the linguistic field , we provide additional empirical evidence to various word order hypotheses as well as demonstrate the validity of the LM-based method . We plan to further explore the capability of LMs on other linguistic phenomena related to word order , such as " given new ordering " ( Nakagawa , 2016 ; Asahara et al . , 2018 ) . Since LMs are language-agnostic , analyzing word order in another language with the LM-based method would also be an interesting direction to investigate . Furthermore , we would like to extend a comparison between machine and human language processing beyond the perspective of word order .

Linguistical approaches to analyze word order phenomena have scalability and preprocessor error propagation problems, and the use of language models is limited in English.

They validate language models as a tool to study word order in Japanese by examining the relationship between canonical word order and generation probability.

They show that language models have sufficient word order knowledge in Japanese to be used as a tool for linguists.


A conventional approach to improving the performance of end-to-end speech translation ( E2E-ST ) models is to leverage the source transcription via pre-training and joint training with automatic speech recognition ( ASR ) and neural machine translation ( NMT ) tasks . However , since the input modalities are different , it is difficult to leverage source language text successfully . In this work , we focus on sequencelevel knowledge distillation ( SeqKD ) from external text-based NMT models . To leverage the full potential of the source language information , we propose backward SeqKD , SeqKD from a target-to-source backward NMT model . To this end , we train a bilingual E2E-ST model to predict paraphrased transcriptions as an auxiliary task with a single decoder . The paraphrases are generated from the translations in bitext via back-translation . We further propose bidirectional SeqKD in which SeqKD from both forward and backward NMT models is combined . Experimental evaluations on both autoregressive and non-autoregressive models show that SeqKD in each direction consistently improves the translation performance , and the effectiveness is complementary regardless of the model capacity . End-to-end speech translation ( E2E-ST ) ( Bérard et al . , 2016 ) , which aims to convert source speech to text in another language directly , is an active research area . Because direct ST is a more difficult task than automatic speech recognition ( ASR ) and machine translation ( MT ) , various techniques have been proposed to ease the training process by using source transcription . Examples include pretraining ( Bérard et al . , 2018 ; Wang et al . , 2020c ; Bansal et al . , 2019 ; Wang et al . , 2020d ) , multi-task learning ( Weiss et al . , 2017 ; Bérard et al . , 2018 ; Bahar et al . , 2019 ) , knowledge distillation ( Liu et al . , 2019 ) , meta-learning ( Indurthi et al . , 2020 ) , twopass decoding ( Anastasopoulos and Chiang , 2018 ; Sperber et al . , 2019 ) , and interactive decoding ( Liu et al . , 2020 ; Le et al . , 2020 ) . However , as input modalities between ST and MT tasks are different , an auxiliary MT task is not always helpful , especially when additional bitext is not available ( Bahar et al . , 2019 ) . Moreover , because monotonic speech-to-transcription alignments encourage the ASR task to see surface-level local information , an auxiliary ASR task helps the E2E-ST model to extract acoustic representations , not semantic ones , from speech . Sequence-level knowledge distillation ( Se-qKD ) ( Kim and Rush , 2016 ) is another approach to transferring knowledge from one model to another . Recent studies have shown that SeqKD has the effect of reducing the complexity of training data and thus eases the training of student models , e.g. , non-autoregressive ( NAR ) models ( Gu et al . , 2018 ; Zhou et al . , 2019a ; Ren et al . , 2020 ) . Paraphrasing , which represents text in a different form but with the same meaning , can also be regarded as SeqKD when using neural paraphrasing via back-translation ( Mallinson et al . , 2017 ; Wieting et al . , 2017 ; Federmann et al . , 2019 ) . It has been studied to improve the reference diversity for MT system evaluations ( Thompson and Post , 2020 ; Bawden et al . , 2020a , b ) and the performance of low-resource neural MT ( NMT ) models ( Zhou et al . , 2019b ; Khayrallah et al . , 2020 ) . In this work , due to its simplicity and effectiveness , we focus on SeqKD from text-based NMT models to improve the performance of a bilingual E2E-ST model . In order to fully leverage source language information , we propose backward Se-qKD , which targets paraphrased source transcriptions generated from a target-to-source backward NMT model as an auxiliary task . Then , a single ST decoder is trained to predict both source and target language text as in a multilingual setting ( Inaguma et al . , 2019 ) . This way , the decoder is biased to capture semantic representations from speech , un-like joint training with an auxiliary ASR task . We also propose bidirectional SeqKD , which combines SeqKD from two NMT models in both language directions . Therefore , the E2E-ST models can fully exploit the knowledge embedded in both forward and backward NMT models . Experimental evaluations demonstrate that Se-qKD from each direction consistently improves the translation performance of both autoregressive and non-autoregressive E2E-ST models . We also confirm that bidirectional SeqKD outperforms unidirectional SeqKD and that the effectiveness is maintained in large models . To fully leverage knowledge in both source and target language directions for bilingual E2E-ST models , we have proposed bidirectional SeqKD , in which both forward SeqKD from a source-to-target NMT model and backward SeqKD from a target-tosource NMT model are combined . Backward Se-qKD is performed by targeting source paraphrases generated via back-translation from the original translations in bitext . Then , the E2E-ST model is enhanced by training to generate both source and target language text with a single decoder . We experimentally confirmed that SeqKD from each direction boosted the translation performance of both autoregressive and non-autoregressive E2E-ST models , and the effectiveness was additive . Multi-referenced training with the original and distilled text gave further gains . We also showed that bidirectional SeqKD was effective regardless of model sizes .

Existing works on end-to-end speech recognition models use the source transcriptions for performance improvements but it is challenging due to the modality gap.

They propose a bidirectional sequence knowledge distillation which learns from text-based NMT systems with a single decoder to enhance the model to capture semantic representations.

Evaluations on autoregressive and non-autoregressive models show that the proposed method improves in both directions and the results are consistent in different model sizes.


This paper explores the problem of computing text similarity between verb phrases describing skilled human behavior for the purpose of finding approximate matches . Four parsers are evaluated on a large corpus of skill statements extracted from an enterprise-wide expertise taxonomy . A similarity measure utilizing common semantic role features extracted from parse trees was found superior to an information-theoretic measure of similarity and comparable to the level of human agreement . Knowledge-intensive industries need to become more efficient at deploying the right expertise as quickly and smoothly as possible , thus it is desired to have systems that can quickly match and deploy skilled individuals to meet customer needs . The searches in most of the current matching systems are based on exact matches between skill statements . However , exact matching is very likely to miss individuals who are very good matches to the job but did n't select the exact skills that appeared in the open job description . It is always hard for individuals to find the perfect skills to describe their skill sets . For example , an individual might not know whether to choose a skill stating that refers to " maintaining " a given product or " supporting " it or whether to choose a skill about maintaining a " database " or about maintaining " DB2 " . Thus , it is desirable for the job search system to be able to find approximate matches , instead of only exact matches , between available individuals and open job positions . More specifically , a skill similarity computation is needed to allow searches to be expanded to related skills , and return more potential matches . In this paper , we present our work on developing a skill similarity computation based upon semantic commonalities between skill statements . Although there has been much work on text similarity metrics ( Lin , 1998a ; Corley and Mihalcea , 2005 ) , most approaches treat texts as a bag of words and try to find shared words with certain statistical properties based on corpus frequencies . As a result , the structural information in the text is ignored in these approaches . We will describe a new semantic approach that takes the structural information of the text into consideration and matches skill statements on corresponding semantic roles . We will demonstrate that it can outperform standard statistical text similarity techniques , and reach the level of human agreement . In Section 2 , we first describe the skill statements we extracted from an enterprise-wide expertise taxonomy . In Section 3 , we describe the performance of a standard statistical approach on this task . This motivates our semantic approach of matching skill statements on corresponding semantic roles . We also compare and evaluate the performance of four natural language parsers ( the Charniak parser , the Stanford parser , the ESG parser , and MINIPAR ) for the purpose of our task . An inter-rater agreement study and evaluation of our approach will be presented in Section 4 . We end with a discussion and conclusion . In this paper , we have presented our work on a semantic similarity computation for skill statements in natural language . We compared and evaluated four different natural language parsers for our task , and matched skills on their corresponding semantic roles extracted from the parse trees generated by one of these parsers . The evaluation results showed that the skill similarity computation based on semantic role matching can outperform a standard statistical approach and reach the level of human agreement . The extracted semantic role information can also be incorporated into the standard statistical approaches as additional features . One way is to give higher weights to those semantic role features deemed most important . This approach has achieved a high performance for a text categorization task when combining extracted keywords with the full text ( Hulth and Megyesi , 2006 ) . We have shown that good results can be achieved for a domain-specific text matching task by performing a simple word-based feature comparison on corresponding structural elements of texts . We have shown that the structural elements of importance can be identified by domain-specific pattern analysis of corresponding parse trees . We believe this approach can generalize to other domains where phrases , sentences , or other short texts need to be compared .

Existing systems for skilled expertise matching use exact matching between skill statements resulting in missing good matches and calling for a system with approximate matching.

They evaluate four different parsers to take structural information into consideration by matching skill statements on corresponding semantic roles from generated parse trees.

The proposed similarity measure outperforms a standard statistical information-theoretic measure and is comparable to a human agreement.


Continual relation learning aims to continually train a model on new data to learn incessantly emerging novel relations while avoiding catastrophically forgetting old relations . Some pioneering work has proved that storing a handful of historical relation examples in episodic memory and replaying them in subsequent training is an effective solution for such a challenging problem . However , these memorybased methods usually suffer from overfitting the few memorized examples of old relations , which may gradually cause inevitable confusion among existing relations . Inspired by the mechanism in human long-term memory formation , we introduce episodic memory activation and reconsolidation ( EMAR ) to continual relation learning . Every time neural models are activated to learn both new and memorized data , EMAR utilizes relation prototypes for memory reconsolidation exercise to keep a stable understanding of old relations . The experimental results show that EMAR could get rid of catastrophically forgetting old relations and outperform the state-of-the-art continual learning models . The code and datasets are released on https://github.com / thunlp/ ContinualRE . Relation extraction aims at detecting relations between entities from text , e.g. , extracting the relation " the president of " from the given sentence " Newton served as the president of the Royal Society " , which could serve as external resource for various downstream applications ( Dong et al . , 2015 ; Xiong et al . , 2017 ; Schlichtkrull et al . , 2018 ) . The conventional RE methods ( Riedel et al . , 2013 ; Zeng et al . , 2014 ; Lin et al . , 2016 ) mostly focus on recognizing relations for a fixed pre-defined relation set , and can not handle rapidly emerging novel relations in the real world . Some researchers therefore explore to detect and learn incessantly emerging relations in an open scenario . As shown in Figure 1 , their efforts can be formulated into a two-step pipeline : ( 1 ) Open Relation Learning extracts phrases and arguments to construct patterns of specific relations , and then discovers unseen relation types by clustering patterns , and finally expands sufficient examples of new relation types from large-scale textual corpora ; ( 2 ) Continual Relation Learning continually uses those expanded examples of new relations to train an effective classifier . The classifier is trained on a sequence of tasks for handling both existing and novel relations , where each task has its own relation set . Although continual relation learning is vital for learning emerging relations , there are rare explorations for this field . A straightforward solution is to store all historical data and re-train models every time new relations and examples come in . Nevertheless , it is computationally expensive since relations are in sustainable growth . Moreover , the huge example number of each relation makes frequently mixing new and old examples become infeasible in the real world . Therefore , storing all data is not practical in continual relation learning . In view of this , the recent preliminary work ( Wang et al . , 2019 ) indicates that the main challenge of continual relation learning is the catastrophic forgetting problem , i.e. , it is hard to learn new relations and meanwhile avoid forgetting old relations , considering memorizing all the data is almost impossible . Recent work ( Shin et al . , 2017 ; Kemker and Kanan , 2018 ; Chaudhry et al . , 2019 ) has shown that the memory-based approaches , maintaining episodic memory to save a few training examples in old tasks and re-training memorized examples during training new tasks , are one of the most effective solutions to the catastrophic forgetting problem , especially for continual learning in NLP scenarios ( Wang et al . , 2019 ; d'Autume et al . , 2019 ) . However , existing memory-based models still suffer from an overfitting problem : when adapting them for continual relation learning , they may frequently change feature distribution of old relations , gradually overfit a few examples in memory , and finally become confused among old relations after long-term training . In fact , these memory-based methods are similar to long-term memory model of mammalian memory in neuroscience ( McClelland et al . , 1995 ; Bontempi et al . , 1999 ) . Although researchers in neuroscience are not clear about secrets inside the human brain , they reach a consensus that the formation of long-term memory relies on continually replaying and consolidating information ( Tononi and Cirelli , 2006 ; Boyce et al . , 2016 ; Yang et al . , 2014 ) , corresponding to the episodic memory and memory replay in continual learning models . Yet later work ( Nader et al . , 2000 ; Lee et al . , 2004 ; Alberini , 2005 ) in neuroscience indicates that reactivation of consolidated memory triggers a reconsolidation stage to continually maintain memory , and memory is easy to be changed or erased in this stage . To apply some reconsolidation exercises can help memory go through this stage and keep long-term memory stable . Intuitively , the ex-isting memory-based models seem like continual memory activation without reconsolidation exercises , and thus become sensitive and volatile . Inspired by the reconsolidation mechanism in human long-term memory formation , we introduce episodic memory activation and reconsolidation ( EMAR ) to continual relation learning in this paper . More specifically , when training models on new relations and their examples , we first adopt memory replay to activate neural models on examples of both new relations and memory , and then utilize a special reconsolidation module to let models avoid excessively changing and erasing feature distribution of old relations . As the core of relation learning is to grasp relation prototypes rather than rote memorization of relation examples , our reconsolidation module requires models to be able to distinguish old relation prototypes after each time memory is replayed and activated . As compared with pioneering explorations to improve episodic memory replay ( Chaudhry et al . , 2019 ; Wang et al . , 2019 ) , with toughly keeping feature distribution of old relations invariant , EMAR is more flexible in feature spaces and powerful in remembering relation prototypes . We conduct sufficient experiments on several RE datasets , and the results show that EMAR effectively alleviates the catastrophic forgetting problem and significantly outperforms the stateof-the-art continual learning models . Further experiments and analyses indicate the reasons for the effectiveness of EMAR , proving that it can utilize a few examples in old tasks to reconsolidate old relation prototypes and keep better distinction among old relations after long-term training . To alleviate catastrophically forgetting old relations in continual relation learning , we introduce episodic memory activation and reconsolidation ( EMAR ) , inspired by the mechanism in human long-term memory formation . Compared with existing memory-based methods , EMAR requires models to understand the prototypes of old relations rather than to overfit a few specific memorized examples , which can keep better distinction among relations after long-term training . We conduct experiments on three benchmarks in relation extraction and carry out extensive experimental results as well as empirical analyses , showing the effectiveness of EMAR on utilizing memorized examples . For future work , how to combine open relation learning and continual relation learning together to complete the pipeline for emerging relations still remains a problem , and we will continue to work on it .

Storing histories of examples is shown to be effective for continual relation learning however existing methods suffer from overfitting to memorize a few old memories.

They propose a human memory mechanism inspired by memory activation and reconsolidation method which aims to keep a stable understanding of old relations.

The proposed method mitigates catastrophic forgetting of old relations and achieves state-of-the-art on several relation extraction datasets showing it can use memorized examples.


This paper proposes to study a fine-grained semantic novelty detection task , which can be illustrated with the following example . It is normal that a person walks a dog in the park , but if someone says " A man is walking a chicken in the park , " it is novel . Given a set of natural language descriptions of normal scenes , we want to identify descriptions of novel scenes . We are not aware of any existing work that solves the problem . Although existing novelty or anomaly detection algorithms are applicable , since they are usually topic-based , they perform poorly on our fine-grained semantic novelty detection task . This paper proposes an effective model ( called GAT-MA ) to solve the problem and also contributes a new dataset . Experimental evaluation shows that GAT-MA outperforms 11 baselines by large margins . Novelty or anomaly detection has been an important research topic since 1970s ( Barnett and Lewis , 1994 ) due to numerous applications ( Chalapathy et al . , 2018 ; Pang et al . , 2021 ) . Recently , it has also become important for natural language processing ( NLP ) . Many researchers have studied the problem in the text classification setting ( Fei and Liu , 2016 ; Shu et al . , 2017 ; Xu et al . , 2019 ; Lin and Xu , 2019 ; Zheng et al . , 2020 ) . However , these text novelty classifiers are mainly coarse-grained , working at the document or topic level . Given a text document , their goal is to detect whether the text belongs to a known class or unknown class . This paper introduces a new text novelty detection problem -fine-grained semantic novelty detection . Specifically , given a text description d , we detect whether d represents a semantically novel fact or not . This work considers text data that describe scenes of real-world phenomena in natural language ( NL ) . In our daily lives , we observe different real-world phenomena ( events , activities , situations , etc . ) and often describe these observations ( referred as " scenes " onwards ) in NL to others or write about them . It is quite natural to observe scenes that we have not seen before ( i.e. , novel scenes ) . For example , it is a common scene that " A person walks a dog in the park " , but if someone says " A man is walking a chicken in the park " , it is quite unexpected and novel . Detecting such semantic novelty requires complex conceptual and semantic reasoning over text and thus , is a challenging NLP problem . Note that conceptually , the judgement of the novelty of a scene is subjective and might differ from person to person . However , there are some scenes for which a majority of people have agreement about their novelty . A good example of such majority-view of novelty is the widely-spread meme pictures on social media , which contain novel interactions between objects . In this work , we restrict our research to this majority-based view of novelty and leave the personalized novelty view angle for the future work . In this work , we leverage the captions of images from popular datasets like COCO , Flickr , etc . , to build a semantic novelty detection dataset ( Sec . 3 ) , 1where we consider an image as a scene and the corresponding image captions as different NL descriptions of the scene . Detecting text describing semantically novel observations have many applications , e.g. , recommending novel news , novel images & videos ( based on their text descriptions ) , social media posts and conversations . The problem of semantic novelty detection is defined as follows . Problem Definition : Given a set of natural language descriptions D = { d 1 , d 2 , ..... d n } of common scenes , build a model M using D to score the semantic novelty of a test NL description d with respect to D , i.e. , classifying d into one of the two classes { NORMAL , NOVEL } . " NORMAL " means that d is a description of a common scene and " NOVEL " means d is a description of a se-mantically novel scene . As the detection model M is built only with " NORMAL " class data , the task is an one-class text classification problem . We are unaware of any existing work that can effectively solve this problem . Although existing novelty / anomaly detection and one-class classification algorithms are applicable , since they are coarse-grained or topic-based , they perform poorly on our task ( see Sec . 5 ) . Note that although we focus on solving the problem of semantic novelty detection of NL descriptions of scenes , the proposed task and solution framework are generally applicable to other applications . This paper proposes a new technique , called GAT-MA ( Graph Attention network with Max-Margin loss and knowledge-based contrastive data Augmentation ) to identify NL description sentences of novel scenes . Since our task is at the sentence level and fine-grained , we exploit Graph Attention Network ( GAT ) on the parsed dependency graph of each sentence , which fuses both semantic and syntactic information in the sentence for reasoning with the internal interactions of entities and actions . To enable the model to capture long-range interactions , we stack multiple layers of GATs to build a deep GAT model with multi-hop graph attention . We also create the pseudo novel training data based on the given normal training data through contrastive data augmentation . Thus , GAT-MA is trained with the given original normal scene descriptions and the augmented pseudo novel scene descriptions ( Sec . 4 ) . GAT-MA is evaluated using our newly created Novel Scene Description Detection ( NSD2 ) Dataset . The results show that GAT-MA outperforms a wide range of latest novelty or anomaly detection baselines by very large margins . Our main contributions are as follows : 1 . We propose a new task of semantic novelty detection in text . Whereas the existing work focuses on coarse-grained document-or topiclevel novelty , our task requires fine-grained sentence-level semantic & syntactic analysis . 2 . We propose a highly effective technique called GAT-MA to solve the proposed semantic novelty detection problem , which is based on GAT with dependency parsing and knowledgebased contrastive data augmentation . 3 . We create a new dataset called NSD2 for the proposed task . The dataset can be used as a benchmark dataset by the NLP community . Novelty detection is an important problem because anything novel is of interest . This paper proposed a semantic novelty detection problem and designed a graph attention network based approach ( called GAT-MA ) exploiting parsing and data augmentation to solve the problem . As there is no existing evaluation dataset for the proposed task , an evaluation dataset has been created . Experimental comparisons with a wide range of baselines showed that GAT-MA outperforms them by very large margins .

Existing works on novelty or abnormally detection are coarse-grained only focusing on the document or sentence level as a text classification task.

They propose a fine-grained semantic novelty detection problem where systems detect whether a textual description is a novel fact, coupled with a graph attention-based model.

The proposed model outperforms 11 baseline models on the created dataset from an image caption dataset for the proposed task by large margins.


Long distance word reordering is a major challenge in statistical machine translation research . Previous work has shown using source syntactic trees is an effective way to tackle this problem between two languages with substantial word order difference . In this work , we further extend this line of exploration and propose a novel but simple approach , which utilizes a ranking model based on word order precedence in the target language to reposition nodes in the syntactic parse tree of a source sentence . The ranking model is automatically derived from word aligned parallel data with a syntactic parser for source language based on both lexical and syntactical features . We evaluated our approach on largescale Japanese-English and English-Japanese machine translation tasks , and show that it can significantly outperform the baseline phrasebased SMT system . * This work has been done while the first author was visiting Modeling word reordering between source and target sentences has been a research focus since the emerging of statistical machine translation . In phrase-based models ( Och , 2002 ; Koehn et al . , 2003 ) , phrase is introduced to serve as the fundamental translation element and deal with local reordering , while a distance based distortion model is used to coarsely depict the exponentially decayed word movement probabilities in language translation . Further work in this direction employed lexi-calized distortion models , including both generative ( Koehn et al . , 2005 ) and discriminative ( Zens and Ney , 2006 ; Xiong et al . , 2006 ) variants , to achieve finer-grained estimations , while other work took into account the hierarchical language structures in translation ( Chiang , 2005 ; Galley and Manning , 2008 ) . Long-distance word reordering between language pairs with substantial word order difference , such as Japanese with Subject-Object-Verb ( SOV ) structure and English with Subject-Verb-Object ( SVO ) structure , is generally viewed beyond the scope of the phrase-based systems discussed above , because of either distortion limits or lack of discriminative features for modeling . The most notable solution to this problem is adopting syntax-based SMT models , especially methods making use of source side syntactic parse trees . There are two major categories in this line of research . One is tree-to-string model ( Quirk et al . , 2005 ; Liu et al . , 2006 ) which directly uses source parse trees to derive a large set of translation rules and associated model parameters . The other is called syntax pre-reordering -an approach that re-positions source words to approximate target language word order as much as possible based on the features from source syntactic parse trees . This is usually done in a preprocessing step , and then followed by a standard phrase-based SMT system that takes the re-ordered source sentence as input to finish the translation . In this paper , we continue this line of work and address the problem of word reordering based on source syntactic parse trees for SMT . Similar to most previous work , our approach tries to rearrange the source tree nodes sharing a common parent to mimic the word order in target language . To this end , we propose a simple but effective ranking-based approach to word reordering . The ranking model is automatically derived from the word aligned parallel data , viewing the source tree nodes to be reordered as list items to be ranked . The ranks of tree nodes are determined by their relative positions in the target language -the node in the most front gets the highest rank , while the ending word in the target sentence gets the lowest rank . The ranking model is trained to directly minimize the mis-ordering of tree nodes , which differs from the prior work based on maximum likelihood estimations of reordering patterns ( Li et al . , 2007 ; Genzel , 2010 ) , and does not require any special tweaking in model training . The ranking model can not only be used in a pre-reordering based SMT system , but also be integrated into a phrasebased decoder serving as additional distortion features . We evaluated our approach on large-scale Japanese-English and English-Japanese machine translation tasks , and experimental results show that our approach can bring significant improvements to the baseline phrase-based SMT system in both preordering and integrated decoding settings . In the rest of the paper , we will first formally present our ranking-based word reordering model , then followed by detailed steps of modeling training and integration into a phrase-based SMT system . Experimental results are shown in Section 5 . Section 6 consists of more discussions on related work , and Section 7 concludes the paper . In this paper we present a ranking based reordering method to reorder source language to match the word order of target language given the source side parse tree . Reordering is formulated as a task to rank different nodes in the source side syntax tree according to their relative position in the target language . The ranking model is automatically trained to minimize the mis-ordering of tree nodes in the training data . Large scale experiment shows improvement on both reordering metric and SMT performance , with up to 1.73 point BLEU gain in our evaluation test . In future work , we plan to extend the ranking model to handle reordering between multiple levels of source trees . We also expect to explore better way to integrate ranking reorder model into SMT system instead of a simple penalty scheme . Along the research direction of preprocessing the source language to facilitate translation , we consider to not only change the order of the source language , but also inject syntactic structure of the target language into source language by adding pseudo words into source sentences .

Long distance word reordering remains a challenge for statistical machine translation and existing approaches do it during the preprocessing.

They propose a ranking-based reordering approach where the ranking model is automatically derived from the word aligned parallel data using a syntax parser.

Large scale evaluation of Japanese-English and English-Japanese shows that the proposed approach significantly outperforms the baseline phrase-based statistical machine translation system.


How can the development of ideas in a scientific field be studied over time ? We apply unsupervised topic modeling to the ACL Anthology to analyze historical trends in the field of Computational Linguistics from 1978 to 2006 . We induce topic clusters using Latent Dirichlet Allocation , and examine the strength of each topic over time . Our methods find trends in the field including the rise of probabilistic methods starting in 1988 , a steady increase in applications , and a sharp decline of research in semantics and understanding between 1978 and 2001 , possibly rising again after 2001 . We also introduce a model of the diversity of ideas , topic entropy , using it to show that COLING is a more diverse conference than ACL , but that both conferences as well as EMNLP are becoming broader over time . Finally , we apply Jensen-Shannon divergence of topic distributions to show that all three conferences are converging in the topics they cover . How can we identify and study the exploration of ideas in a scientific field over time , noting periods of gradual development , major ruptures , and the waxing and waning of both topic areas and connections with applied topics and nearby fields ? One important method is to make use of citation graphs ( Garfield , 1955 ) . This enables the use of graphbased algorithms like PageRank for determining researcher or paper centrality , and examining whether their influence grows or diminishes over time . However , because we are particularly interested in the change of ideas in a field over time , we have chosen a different method , following Kuhn ( 1962 ) . In Kuhn 's model of scientific change , science proceeds by shifting from one paradigm to another . Because researchers ' ideas and vocabulary are constrained by their paradigm , successive incommensurate paradigms will naturally have different vocabulary and framing . Kuhn 's model is intended to apply only to very large shifts in scientific thought rather than at the micro level of trends in research foci . Nonetheless , we propose to apply Kuhn 's insight that vocabulary and vocabulary shift is a crucial indicator of ideas and shifts in ideas . Our operationalization of this insight is based on the unsupervised topic model Latent Dirichlet Allocation ( LDA ; Blei et al . ( 2003 ) ) . For many fields , doing this kind of historical study would be very difficult . Computational linguistics has an advantage , however : the ACL Anthology , a public repository of all papers in the Computational Linguistics journal and the conferences and workshops associated with the ACL , COLING , EMNLP , and so on . The ACL Anthology ( Bird , 2008 ) , and comprises over 14,000 documents from conferences and the journal , beginning as early as 1965 through 2008 , indexed by conference and year . This resource has already been the basis of citation analysis work , for example , in the ACL Anthology Network of Joseph and Radev ( 2007 ) . We apply LDA to the text of the papers in the ACL Anthology to induce topics , and use the trends in these topics over time and over conference venues to address questions about the development of the field . Despite the relative youth of our field , computational linguistics has witnessed a number of research trends and shifts in focus . While some trends are obvious ( such as the rise in machine learning methods ) , others may be more subtle . Has the field gotten more theoretical over the years or has there been an increase in applications ? What topics have declined over the years , and which ones have remained roughly constant ? How have fields like Dialogue or Machine Translation changed over the years ? Are there differences among the conferences , for example between COLING and ACL , in their interests and breadth of focus ? As our field matures , it is important to go beyond anecdotal description to give grounded answers to these questions . Such answers could also help give formal metrics to model the differences between the many conferences and venues in our field , which could influence how we think about reviewing , about choosing conference topics , and about long range planning in our field . Our method discovers a number of trends in the field , such as the general increase in applications , the steady decline in semantics , and its possible reversal . We also showed a convergence over time in topic coverage of ACL , COLING , and EMNLP as well an expansion of topic diversity . This growth and convergence of the three conferences , perhaps influenced by the need to increase recall ( Church , 2005 ) seems to be leading toward a tripartite realization of a single new " latent " conference .

How topics or ideas have developed over time in NLP community remains unknown while there are analysis over the ACL anthology citation graph.

They propose to use Latent Dirichlet Allocation for studying topic shift over time and a model to compute the diversity of ideas and topic entropy.

They found that COLING has more diversity than ACL, but all the conferences are becoming to cover more topics over time, and applications increase generally.