Named entity phrases are some of the most difficult phrases to translate because new phrases can appear from nowhere , and because many are domain specific , not to be found in bilingual dictionaries . We present a novel algorithm for translating named entity phrases using easily obtainable monolingual and bilingual resources . We report on the application and evaluation of this algorithm in translating Arabic named entities to English . We also compare our results with the results obtained from human translations and a commercial system for the same task . Named entity phrases are being introduced in news stories on a daily basis in the form of personal names , organizations , locations , temporal phrases , and monetary expressions . While the identification of named entities in text has received significant attention ( e.g. , Mikheev et al . ( 1999 ) and Bikel et al . ( 1999 ) ) , translation of named entities has not . This translation problem is especially challenging because new phrases can appear from nowhere , and because many named-entities are domain specific , not to be found in bilingual dictionaries . A system that specializes in translating named entities such as the one we describe here would be an important tool for many NLP applications . Statisti-cal machine translation systems can use such a system as a component to handle phrase translation in order to improve overall translation quality . Cross-Lingual Information Retrieval ( CLIR ) systems could identify relevant documents based on translations of named entity phrases provided by such a system . Question Answering ( QA ) systems could benefit substantially from such a tool since the answer to many factoid questions involve named entities ( e.g. , answers to who questions usually involve Persons / Organizations , where questions involve Locations , and when questions involve Temporal Expressions ) . In this paper , we describe a system for Arabic-English named entity translation , though the technique is applicable to any language pair and does not require especially difficult-to-obtain resources . The rest of this paper is organized as follows . In Section 2 , we give an overview of our approach . In Section 3 , we describe how translation candidates are generated . In Section 4 , we show how monolingual clues are used to help re-rank the translation candidates list . In Section 5 , we describe how the candidates list can be extended using contextual information . We conclude this paper with the evaluation results of our translation algorithm on a test set . We also compare our system with human translators and a commercial system . We have presented a named entity translation algorithm that performs at near human translation accuracy when translating Arabic named entities to English . The algorithm uses very limited amount of hard-to-obtain bilingual resources and should be easily adaptable to other languages . We would like to apply to other languages such as Chinese and Japanese and to investigate whether the current algorithm would perform as well or whether new algorithms might be needed . Currently , our translation algorithm does not use any dictionary of named entities and they are translated on the fly . Translating a common name incorrectly has a significant effect on the translation accuracy . We would like to experiment with adding a small named entity translation dictionary for common names and see if this might improve the overall translation accuracy .

Translating named entities is challenging since they can appear from nowhere, and cannot be found in bilingual dictionaries because they are domain specific.

They propose an algorithm for Arabic-English named entity translation which uses easily obtainable monolingual and bilingual resources and a limited amount of hard-to-obtain bilingual resources.

The proposed algorithm is compared with human translators and a commercial system and it performs at near human translation.


Probabilistic Latent Semantic Analysis ( PLSA ) models have been shown to provide a better model for capturing polysemy and synonymy than Latent Semantic Analysis ( LSA ) . However , the parameters of a PLSA model are trained using the Expectation Maximization ( EM ) algorithm , and as a result , the trained model is dependent on the initialization values so that performance can be highly variable . In this paper we present a method for using LSA analysis to initialize a PLSA model . We also investigated the performance of our method for the tasks of text segmentation and retrieval on personal-size corpora , and present results demonstrating the efficacy of our proposed approach . In modeling a collection of documents for information access applications , the documents are often represented as a " bag of words " , i.e. , as term vectors composed of the terms and corresponding counts for each document . The term vectors for a document collection can be organized into a term by document co-occurrence matrix . When directly using these representations , synonyms and polysemous terms , that is , terms with multiple senses or meanings , are not handled well . Methods for smoothing the term distributions through the use of latent classes have been shown to improve the performance of a number of information access tasks , including retrieval over smaller collections ( Deerwester et al . , 1990 ) , text segmentation ( Brants et al . , 2002 ) , and text classification ( Wu and Gunopulos , 2002 ) . The Probabilistic Latent Semantic Analysis model ( PLSA ) ( Hofmann , 1999 ) provides a probabilistic framework that attempts to capture polysemy and synonymy in text for applications such as retrieval and segmentation . It uses a mixture decomposition to model the co-occurrence data , and the probabilities of words and documents are obtained by a convex combination of the aspects . The mixture approximation has a well defined probability distribution and the factors have a clear probabilistic meaning in terms of the mixture component distributions . The PLSA model computes the relevant probability distributions by selecting the model parameter values that maximize the probability of the observed data , i.e. , the likelihood function . The standard method for maximum likelihood estimation is the Expectation Maximization ( EM ) algorithm . For a given initialization , the likelihood function increases with EM iterations until a local maximum is reached , rather than a global maximum , so that the quality of the solution depends on the initialization of the model . Additionally , the likelihood values across different initializations are not comparable , as we will show . Thus , the likelihood function computed over the training data can not be used as a predictor of model performance across different models . Rather than trying to predict the best performing model from a set of models , in this paper we focus on finding a good way to initialize the PLSA model . We will present a framework for using Latent Semantic Analysis ( LSA ) ( Deerwester et al . , 1990 ) to better initialize the parameters of a corresponding PLSA model . The EM algorithm is then used to further refine the initial estimate . This combination of LSA and PLSA leverages the advantages of both . This paper is organized as follows : in section 2 , we review related work in the area . In section 3 , we summarize related work on LSA and its probabilistic interpretation . In section 4 we review the PLSA model and in section 5 we present our method for initializing a PLSA model using LSA model parameters . In section 6 , we evaluate the performance of our framework on a text segmentation task and several smaller information retrieval tasks . And in section 7 , we summarize our results and give directions for future work . We have presented LSA-PLSA , an approach for improving the performance of PLSA by leveraging the best features of PLSA and LSA . Our approach uses LSA to initialize a PLSA model , allowing for arbitrary weighting schemes to be incorporated into a PLSA model while leveraging the optimization used to improve the estimate of the PLSA parameters . We have evaluated the proposed framework on two tasks : personalsize information retrieval and text segmentation . The LSA-PLSA model outperformed PLSA on all tasks . And in all cases , combining PLSA-based models outperformed a single model . The best performance was obtained with combined models when one of the models was the LSA-PLSA model . When combining multiple PLSA models , the use of LSA-PLSA in combination with either two PLSA models or one PLSA and one LSA model improved performance while reducing the running time over the combination of four or more PLSA models as used by others . Future areas of investigation include quantifying the expected performance of the LSAinitialized PLSA model by comparing performance to that of the empirically best performing model and examining whether tempered EM could further improve performance .

EM algorithm-baed Probabilistic latent semantic analysis models provide high variance in performance and models with different initializations are not comparable.

They propose to use Latent Semantic Analysis to initialize probabilistic latent semantic analysis models, EM algorithm is further used to refine the initial estimate.

They show that the model initialized in the proposed method always outperforms existing methods.


We rely on arguments in our daily lives to deliver our opinions and base them on evidence , making them more convincing in turn . However , finding and formulating arguments can be challenging . In this work , we present the Arg-CTRL-a language model for argument generation that can be controlled to generate sentence-level arguments for a given topic , stance , and aspect . We define argument aspect detection as a necessary method to allow this fine-granular control and crowdsource a dataset with 5,032 arguments annotated with aspects . Our evaluation shows that the Arg-CTRL is able to generate high-quality , aspectspecific arguments , applicable to automatic counter-argument generation . We publish the model weights and all datasets and code to train the Arg-CTRL . 1 Nuclear reactors produce radioactive waste ... Language models ( Bengio et al . , 2003 ) allow to generate text through learned distributions of a language and have been applied to a variety of areas like machine translation ( Bahdanau et al . , 2015 ) , summarization ( Paulus et al . , 2018 ) , or dialogue systems ( Wen et al . , 2017 ) . A rather new field for these models is the task of producing text with argumentative content ( Wang and Ling , 2016 ) . We believe this technology can support humans in the challenging task of finding and formulating arguments . A politician might use this to prepare for a debate with a political opponent or for a press conference . It may be used to support students in writing argumentative essays or to enrich one-sided discussions with counter-arguments . In contrast to retrieval methods , generation allows to combine and stylistically adapt text ( e.g. arguments ) based on a given input ( usually the beginning of a sentence ) . Current argument generation models , however , produce lengthy texts and allow the user little control over the aspect the argument should address ( Hua et al . , 2019 ; Hua and Wang , 2018 ) . We show that argument generation can be enhanced by allowing for a fine-grained control and limiting the argument to a single but concise sentence . Controllable language models like the CTRL ( Keskar et al . , 2019 ) allow to condition the model at training time to certain control codes . At inference , these can be used to direct the model 's output with regard to content or style . We build upon this architecture to control argument generation based solely on a given topic , stance , and argument aspect . For instance , to enforce focus on the aspect of cancer for the topic of nuclear energy , we input a control code " Nuclear Energy CON cancer " that creates a contra argument discussing this aspect , for instance : " Studies show that people living next to nuclear power plants have a higher risk of developing cancer . " . To obtain control codes from training data , we pre-define a set of topics to retrieve documents for and rely on an existing stance detection model to classify whether a sentence argues in favor ( pro ) or against ( con ) the given topic ( Stab et al . , 2018a ) . Regarding argument aspect detection , however , past work has two drawbacks : it either uses simple rule-based extraction of verb-and noun-phrases ( Fujii and Ishikawa , 2006 ) or the definition of aspects is based on target-concepts located within the same sentence ( Gemechu and Reed , 2019 ) . Aspects as we require and define them are not bound to any part-of-speech tag and ( 1 ) hold the core reason upon which the conclusion / evidence is built and ( 2 ) encode the stance towards a general but not necessarily explicitly mentioned topic the argument discusses . For instance : Topic : Nuclear Energy Argument : Running nuclear reactors is costly as it involves long-time disposal of radioactive waste . The evidence of this argument is based upon the two underlined aspects . While these aspects encode a negative stance towards the topic of " Nuclear Energy " , the topic itself is not mentioned explicitly in the argument . Our final controlled argument generation pipeline ( see Figure 1 ) works as follows : ( 1 ) We gather several million documents for eight different topics from two large data sources . All sentences are classified into pro- , con- , and non-arguments . We detect aspects of all arguments with a model trained on a novel dataset and concatenate arguments with the same topic , stance , and aspect into training documents . ( 2 ) We use the collected classified data to condition the Arg-CTRL on the topics , stances , and aspects of all gathered arguments . ( 3 ) At inference , passing the control code [ Topic ] [ Stance ] [ Aspect ] to the model will generate an argument that follows these commands . Our evaluation shows that the Arg-CTRL is able to produce aspect-specific , high-quality arguments , applicable to automatic counter-argument generation . The contributions are as follows : ( i ) We adapt and fine-tune the CTRL for aspect-controlled neural argument generation . ( ii ) We show that detecting argument aspects and conditioning the generation model on them are necessary steps to control the model 's training process and its perspective while generating . ( iii ) We propose several methods to analyze and evaluate the quality of ( controllable ) argument generation models . ( iv ) We develop a new scheme to annotate argument aspects and release a dataset with 5,032 samples . We apply the concept of controlled neural text generation to the domain of argument generation . Our Arg-CTRL is conditioned on topics , stances , and aspects and can reliably create arguments using these control codes . We show that arguments generated with our approach are genuine and of high argumentative and grammatical quality in general . Moreover , we show that our approach can be used to generate counter-arguments in a transparent and interpretable way . We fine-tune the Arg-CTRL on two different data sources and find that using mixed data from Common-Crawl results in a higher quality of generated arguments than using user discussions from Reddit-Comments . Further , we define argument aspect detection for controlled argument generation and introduce a novel annotation scheme to crowdsource argument aspect annotations , resulting in a high-quality dataset . We publish the model weights , data , and all code necessary to train the Arg-CTRL .

Argumentative content generation can support humans but current models produce lengthy texts and offer a little controllability on aspects of the argument for users.

They train a controllable language model on a corpus annotated with control codes provided by a stance detection model and introduce a dataset for evaluation.

The proposed model can generate arguments that are genuine and argumentative and grammatically correct and also counter-arguments in a transparent and interpretable way.


We describe a question answering model that applies to both images and structured knowledge bases . The model uses natural language strings to automatically assemble neural networks from a collection of composable modules . Parameters for these modules are learned jointly with network-assembly parameters via reinforcement learning , with only ( world , question , answer ) triples as supervision . Our approach , which we term a dynamic neural module network , achieves state-of-theart results on benchmark datasets in both visual and structured domains . This paper presents a compositional , attentional model for answering questions about a variety of world representations , including images and structured knowledge bases . The model translates from questions to dynamically assembled neural networks , then applies these networks to world representations ( images or knowledge bases ) to produce answers . We take advantage of two largely independent lines of work : on one hand , an extensive literature on answering questions by mapping from strings to logical representations of meaning ; on the other , a series of recent successes in deep neural models for image recognition and captioning . By constructing neural networks instead of logical forms , our model leverages the best aspects of both linguistic compositionality and continuous representations . Our model has two components , trained jointly : first , a collection of neural " modules " that can be freely composed ( Figure 1b ) ; second , a network layout predictor that assembles modules into complete deep networks tailored to each question ( Figure 1a ) . Previous work has used manually-specified modular structures for visual learning ( Andreas et al . , 2016 ) . Here we : • learn a network structure predictor jointly with module parameters themselves • extend visual primitives from previous work to reason over structured world representations Training data consists of ( world , question , answer ) triples : our approach requires no supervision of network layouts . We achieve state-of-the-art performance on two markedly different question answering tasks : one with questions about natural images , and another with more compositional questions about United States geography.1 We have introduced a new model , the dynamic neural module network , for answering queries about both structured and unstructured sources of information . Given only ( question , world , answer ) triples as training data , the model learns to assemble neural networks on the fly from an inventory of neural models , and simultaneously learns weights for these modules so that they can be composed into novel structures . Our approach achieves state-of-the-art results on two tasks . We believe that the success of this work derives from two factors : Continuous representations improve the expressiveness and learnability of semantic parsers : by replacing discrete predicates with differentiable neural network fragments , we bypass the challenging combinatorial optimization problem associated with induction of a semantic lexicon . In structured world representations , neural predicate representations allow the model to invent reusable attributes and relations not expressed in the schema . Perhaps more importantly , we can extend compositional questionanswering machinery to complex , continuous world representations like images . Semantic structure prediction improves generalization in deep networks : by replacing a fixed network topology with a dynamic one , we can tailor the computation performed to each problem instance , using deeper networks for more complex questions and representing combinatorially many queries with comparatively few parameters . In practice , this results in considerable gains in speed and sample efficiency , even with very little training data . These observations are not limited to the question answering domain , and we expect that they can be applied similarly to tasks like instruction following , game playing , and language generation .

Existing works on visual learning use manually-specified modular structures.

They propose a question-answering model trained jointly to translate from questions to dynamically assembled neural networks then produce answers with using images or knowledge bases.

The proposed model achieves state-of-the-arts on visual and structured domain datasets showing that coutinous representations improve the expressiveness and learnability of semantic parsers.


Large pre-trained language models reach stateof-the-art results on many different NLP tasks when fine-tuned individually ; They also come with a significant memory and computational requirements , calling for methods to reduce model sizes ( green AI ) . We propose a twostage model-compression method to reduce a model 's inference time cost . We first decompose the matrices in the model into smaller matrices and then perform feature distillation on the internal representation to recover from the decomposition . This approach has the benefit of reducing the number of parameters while preserving much of the information within the model . We experimented on BERTbase model with the GLUE benchmark dataset and show that we can reduce the number of parameters by a factor of 0.4x , and increase inference speed by a factor of 1.45x , while maintaining a minimal loss in metric performance . Deep learning models have been demonstrated to achieve state-of-the-art results , but require large parameter storage and computation . It 's estimated that training a Transformer model with a neural architecture search has a CO 2 emissions equivalent to nearly five times the lifetime emissions of the average U.S. car , including its manufacturing ( Strubell et al . , 2019 ) . Alongside the increase in deep learning models complexity , in the NLP domain , there has been a shift in the NLP modeling paradigm from training a randomly initialized model to fine-tuning a large and computational heavy pre-trained language model ( Howard and Ruder , 2018 ; Peters et al . , 2018 ; Devlin et al . , 2018 ; Radford , 2018 ; Radford et al . , 2019 ; Dai et al . , 2019 ; Yang et al . , 2019 ; Lample and Conneau , 2019 ; Liu et al . , 2019b ; Raffel et al . , 2019 ; Lan et al . , 2019 ; Lewis et al . , 2019 ) . While re-using pre-trained models offsets the training costs , inference time costs of the finetuned models remain significant , and are showstoppers in many applications . The main challenge with pre-trained models is how can we reduce their size while saving the information contained within them . Recent work , approached this by keeping some of the layers while removing others ( Sanh et al . , 2019 ; Sun et al . , 2019 ; Xu et al . , 2020 ) . A main drawback of such approach is in its coarse-grained nature : removing entire layers might discard important information contained within the model , and working at the granularity of layers makes the trade-off between compression and accuracy of a model hard to control . Motivated by this , in this work we suggest a more finegrained approach which decomposes each matrix to two smaller matrices and then perform feature distillation on the internal representation to recover from the decomposition . This approach has the benefit of preserving much of the information while reducing the number of parameters . Alongside the advantage of preserving the information within each layer , there is also a memory flexibility advantage compared to removing entire layers ; As a result of decomposing each matrix to two smaller matrices , we can store each of the two matrices in two different memory blocks . This has the benefit of distributing the model matrices in many small memory blocks , which is useful when working in shared CPU-based environments . We evaluated our approach on the General Language Understanding Evaluation ( GLUE ) benchmark dataset ( Wang et al . , 2018 ) and show that our approach is superior or competitive in the different GLUE tasks to previous approaches which remove entire layers . Furthermore , we study the effects of different base models to decompose and show the superiority of decomposing a fine-tuned model compared to a pre-trained model or a ran-domly initialized model . Finally , we demonstrate the trade-off between compression and accuracy of a model . We presented a way to compress pre-trained large language models fine-tuned for specific tasks , while preserving much of the information contained within them , by using matrix decomposition to two small matrices . For future work it might be interesting to combine this approach with another approach such as pruning or quantization to achieve smaller models .

Existing coarse-grained approaches for reducing the inference time of pretraining models remove layers, posing a trade-off between compression and the accuracy of a model.

They propose a model-compression method which decompresses the matrix and performs feature distillation on the internal representations to recover from the decomposition.

The proposed method reduces the model size by 0.4x and increases inference speed by 1.45x while keeping the performance degradation minimum on the GLUE benchmark.


Several studies on sentence processing suggest that the mental lexicon keeps track of the mutual expectations between words . Current DSMs , however , represent context words as separate features , thereby loosing important information for word expectations , such as word interrelations . In this paper , we present a DSM that addresses this issue by defining verb contexts as joint syntactic dependencies . We test our representation in a verb similarity task on two datasets , showing that joint contexts achieve performances comparable to single dependencies or even better . Moreover , they are able to overcome the data sparsity problem of joint feature spaces , in spite of the limited size of our training corpus . Distributional Semantic Models ( DSMs ) rely on the Distributional Hypothesis ( Harris , 1954 ; Sahlgren , 2008 ) , stating that words occurring in similar contexts have similar meanings . On such theoretical grounds , word co-occurrences extracted from corpora are used to build semantic representations in the form of vectors , which have become very popular in the NLP community . Proximity between word vectors is taken as an index of meaning similarity , and vector cosine is generally adopted to measure such proximity , even though other measures have been proposed ( Weeds et al . , 2004 ; Santus et al . , 2016 ) . Most of DSMs adopt a bag-of-words approach , that is they turn a text span ( i.e. , a word window or a parsed sentence ) into a set of words and they register separately the co-occurrence of each word with a given target . The problem with this approach is that valuable information concerning word interrelations in a context gets lost , because words co-occurring with a target are treated as independent features . This is why works like Ruiz-Casado et al . ( 2005 ) , Agirre et al . ( 2009 ) and Melamud et al . ( 2014 ) proposed to introduce richer contexts in distributional spaces , by using entire word windows as features . These richer contexts proved to be helpful to semantically represent verbs , which are characterized by highly context-sensitive meanings , and complex argument structures . In fact , two verbs may share independent words as features despite being very dissimilar from the semantic point of view . For instance kill and heal share the same object nouns in The doctor healed the patient and the The poison killed the patient , but are highly different if we consider their joint dependencies as a single context . Nonetheless , richer contexts like these suffer from data sparsity , therefore requiring either larger corpora or complex smoothing processes . In this paper , we propose a syntactically savvy notion of joint contexts . To test our representation , we implement several DSMs and we evaluate them in a verb similarity task on two datasets . The results show that , even using a relatively small corpus , our syntactic joint contexts are robust with respect to data sparseness and perform similarly or better than single dependencies in a wider range of parameter settings . The paper is organized as follows . In Section 2 , we provide psycholinguistic and computational background for this research , describing recent models based on word windows . In Section 3 , we describe our reinterpretation of joint contexts with syntactic dependencies . Evaluation settings and results are presented in Section 4 . In this paper , we have presented our proposal for a new type of vector representation based on joint features , which should emulate more closely the general knowledge about event participants that seems to be the organizing principle of our mental lexicon . A core issue of previous studies was the data sparseness challenge , and we coped with it by means of a more abstract , syntactic notion of joint context . The models using joint dependencies were able at least to perform comparably to traditional , dependency-based DSMs . In our experiments , they even achieved the best correlation scores across several parameter settings , especially after the application of SVD . We want to emphasize that previous works such as Agirre et al . ( 2009 ) already showed that large word windows can have a higher discriminative power than indipendent features , but they did it by using a huge training corpus . In our study , joint context-based representations derived from a small corpus such as RCV1 are already showing competitive performances . This result strengthens our belief that dependencies are a possible solution for the data sparsity problem of joint feature spaces . We also believe that verb similarity might not be the best task to show the usefulness of joint contexts for semantic representation . The main goal of the present paper was to show that joint contexts are a viable option to exploit the full potential of distributional information . Our successful tests on verb similarity prove that syntactic joint contexts do not suffer of data sparsity and are also able to beat other types of representations based on independent word features . Moreover , syntactic joint contexts are much simpler and more competitive with respect to window-based ones . The good performance in the verb similarity task motivates us to further test syntactic joint contexts on a larger range of tasks , such as word sense disambiguation , textual entailment and classification of semantic relations , so that they can unleash their full potential . Moreover , our proposal opens interesting perspectives for computational psycholinguistics , especially for modeling those semantic phenomena that are inherently related to the activation of event knowledge ( e.g. thematic fit ) .

Providing richer contexts to Distributional Semantic Models improves by taking word interrelations into account but it would suffer from data sparsity.

They propose a Distributional Semantic Model that incorporates verb contexts as joint syntactic dependencies so that it emulates knowledge about event participants.

They show that representations obtained by the proposed model outperform more complex models on two verb similarity datasets with a limited training corpus.


In this paper , we propose Inverse Adversarial Training ( IAT ) algorithm for training neural dialogue systems to avoid generic responses and model dialogue history better . In contrast to standard adversarial training algorithms , IAT encourages the model to be sensitive to the perturbation in the dialogue history and therefore learning from perturbations . By giving higher rewards for responses whose output probability reduces more significantly when dialogue history is perturbed , the model is encouraged to generate more diverse and consistent responses . By penalizing the model when generating the same response given perturbed dialogue history , the model is forced to better capture dialogue history and generate more informative responses . Experimental results on two benchmark datasets show that our approach can better model dialogue history and generate more diverse and consistent responses . In addition , we point out a problem of the widely used maximum mutual information ( MMI ) based methods for improving the diversity of dialogue response generation models and demonstrate it empirically . In recent years , neural end-to-end dialogue response generation models ( Sordoni et al . , 2015 ; Serban et al . , 2016 ; Bordes et al . , 2016 ) has gained increasing popularity with the recent advancements of neural sequence-to-sequence ( seq2seq ) learning models ( Sutskever et al . , 2014 ; Vaswani et al . , 2017 ) . While neural dialogue models can generate seemingly fluent responses , due to the over-simplified maximum likelihood estimation ( MLE ) training objective and the high frequency of generic responses in training corpora , they tend to produce dull and generic responses such as " I do n't know " much more often than that humans generally do ( Li et al . , 2015 ) , which makes dialogue agents less engaging and ineffective . In addition , recent research on whether neural dialogue systems use dialogue history effectively ( Sankar et al . , 2019 ) shows that most neural dialogue agents fail to take the dialogue history into account when generating responses . This problem makes neural dialogue systems tend to generate responses irrelevant to the current topic of the conversation and are not consistent with the dialogue history . This problem may also intensify the generic response problem , as dull responses are generally off-topic and irrelevant to the dialogue history . To address the above issues , in this paper , we propose Inverse Adversarial Training ( IAT ) algorithm for training neural dialogue systems to avoid generic responses and model dialogue history better , thus generating diverse and informative responses . Conventional adversarial training methods generally generate label-preserving adversarial inputs with carefully designed methods and train the model to generate the same output to enhance the model 's robustness . In contrast , our approach perturbs in input dialogue history such that a good dialogue model should not generate the same output if the output is non-generic and relevant to the dialogue history . We name our proposed method as inverse adversarial training because it is related to conventional adversarial training methods which aim to improve the model 's adversarial robustness but our proposed objective is motivated in the opposite direction . Note that our work is not directly related to TextGANs as well as their applications on dialogue response generation . Specifically , the proposed inverse adversarial training assigns higher rewards to generated responses or ground-truth responses if their likeli-hood decreases more when the dialogue history is perturbed , and penalize the model when it generates responses whose likelihood is almost unchanged given either original or perturbed dialogue history as input . This encourages the model to generate more relevant and informative responses and capture dialogue history better . The proposed IAT algorithm can be used in both supervised and self-supervised fashion ( with / without reference response ) , which can be viewed as a form of reward-augmented maximum likelihood ( RAML ) method ( Norouzi et al . , 2016 ) that improves the original MLE objective or a rewarding scheme for RL-based text generation algorithms . The inverse adversarial learning framework is also conceptually related to self-adversarial learning ( Zhou et al . , 2020 ) where the the comparison is made between different checkpoints of the same model to provide reward for RL training of the NLG model . In addition , we identify a limitation of the widely-used maximum mutual information ( MMI ) based methods for improving the diversity of dialogue response generation models . This will be discussed in detail in section 2.1 and empirically demonstrated in section 4.2 . We conduct experiments on two dialogue datasets , OpenSubtitiles and DailyDialog , to demonstrate the effectiveness of the proposed approach . Experimental results show IAT helps neural dialogue systems model dialogue history better and generate more diverse and informative responses . In this work , we introduce inverse adversarial training ( IAT ) algorithm that is able to simultaneously reduce the dull response problem and help neural dialogue systems model dialogue history better . IAT measures the relevance and consistency of responses by the difference of their likelihood conditioning on either original and perturbed dialogue history . In this way , it is able to prevent the dialogue system from preferring generic responses , even they are often of high frequency in the training corpora . Our method also encourages the dialogue agent to model dialogue history better by penalizing the model when generating the same responses given perturbed dialogue history . Experimental results on two benchmark datasets show that the proposed inverse adversarial training algorithm helps dialogue models capture dialogue history better and generate more diverse and consistent responses . We also identify a limitation of the widely-used MMI based methods for improving the diversity of dialogue response generation models and empirically demonstrate the existence of this problem through our experimetns .

Neural end-to-end dialogue models generate fluent yet dull and generic responses without taking dialogue histories into account due to the over-simplified maximum likelihood estimation objective.

They propose an algorithm which encourages to be sensitive to perturbations in dialogue histories and generates more diverse and consistent responses by applying penalization.

The proposed approach can model dialogue history better and generate more diverse and consistent responses on OpenSubtitles and DailyDialog.


demonstrated that corpus-extracted models of semantic knowledge can predict neural activation patterns recorded using fMRI . This could be a very powerful technique for evaluating conceptual models extracted from corpora ; however , fMRI is expensive and imposes strong constraints on data collection . Following on experiments that demonstrated that EEG activation patterns encode enough information to discriminate broad conceptual categories , we show that corpus-based semantic representations can predict EEG activation patterns with significant accuracy , and we evaluate the relative performance of different corpus-models on this task . Models of semantic relatedness induced from corpus data have proven effective in a number of empirical tasks ( Sahlgren , 2006 ) and there is increasing interest in whether distributional information extracted from corpora correlates with aspects of speakers ' semantic knowledge : see Lund and Burgess ( 1996 ) , Landauer and Dumais ( 1997 ) , Almuhareb ( 2006 ) , Padó and Lapata ( 2007 ) , Schulte i m Walde ( 2008 ) , among many others . For this purpose , corpus models have been tested on datasets that are based on semantic judgements ( metalinguistic or meta-cognitive intuitions about synonymy , semantic distance , category-membership ) or behavioural experiments ( semantic priming , property generation , free association ) . While all these data are valuable , they are indirect reflections of semantic knowledge , and when the predictions they make diverge from those of corpora , interpretation is problematic : is the corpus model missing essential aspects of semantics , or are non-semantic factors biasing the data elicited from informants ? Reading semantic processes and representations directly from the brain would be an ideal way to get around these limitations . Until recently , analysis of linguistic quantities using neural data collected with EEG ( measurement at the scalp of voltages induced by neuronal firing ) or fMRI ( measurement of changes of oxygen concentrations in the brain tied to cognitive processes ) had neither the advantages of corpora ( scale ) nor of informants ( finer grained judgements ) . However , some clear patterns of differential activity have been found for broad semantic classes . Viewing images of natural ( typically animals and plants ) and non-natural ( typically artefacts like tools or vehicles ) objects elicits different loci of activity in fMRI ( Martin and Chao , 2001 ) and EEG ( Kiefer , 2001 ) , that persist across participants . Differences have also been found in response to auditorily or visually presented words of different lexical classes , such as abstract / concrete , and verb / noun ( Pulvermüller , 2002 ) . But interpretation of such group results remains somewhat difficult , as they may be consistent with more than one distinction : the natural / artefactual division just mentioned , may rather be between living / nonliving entities , dynamic / static entities , or be based on embodied experience ( e.g. manipulable or not ) . More recently , however , machine learning and other numerical techniques have been successfully applied to extract semantic information from neural data in a more discriminative fashion , down to the level of individual concepts . The work presented here builds on two strands of previous work : Murphy et al . ( 2008 ) use EEG data to perform semantic categorisation on single stimuli ; and Mitchell et al . ( 2008 ) introduce an fMRIbased method that detects word level distinctions by learning associations between features of neural activity and semantic features derived from a corpus . We combine these innovations by introducing a method that extracts featural representations from the EEG signal , and uses corpusbased models to predict word level distinctions in patterns of EEG activity . The proposed method achieves a performance level significantly above chance ( also when distinguishing between concepts from the same semantic category , e.g. , dog and cat ) , and approaching that achieved with fMRI . The paper proceeds as follows . The next section describes a simple behavioural experiment where Italian-speaking participants had to name photographic images of mammals and tools while their EEG activity was being recorded , and continues to detail how the rich and multidimensional signals collected were reduced to a small set of optimally informative features using a new method . Section 3 describes a series of corpus-based semantic models derived from both a raw-text web corpus , and from various parsings of a conventional corpus . In Section 4 we describe the training of a series of linear models , that each learn the associations between a set of corpus semantic features and an individual EEG activity feature . By combining these models it is possible to predict the EEG activity pattern for a single unseen word , and compare this to the observed pattern for the corresponding concept . Results ( Section 5 ) show that these predictions succeed at a level significantly above chance , both for coarser distinctions between words in different superordinate categories ( e.g. , differentiating between drill and gorilla ) , and , at least for the model based on the larger web corpus , for those within the same category ( e.g. , drill vs spanner , koala vs gorilla ) . Our results show that corpus-extracted conceptual models can be used to distinguish between the EEG activation levels associated with conceptual categories to a degree that is significantly above chance . Though category specific patterns are detectable in the EEG signal alone ( as illustrated by the PCA analysis in figure 3 ) , on that basis we can not be sure that semantics is being detected . Some other property of the stimuli that co-varies with the semantic classes of interest could be responsible , such as visual complexity , conceptual familiarity , lexical frequency , or phonological form . Only by cross-training with individual corpus features and showing that these hold a predictive relationship to neural activity have we been able to establish that EEG patterns encode semantics . Present evidence indicates that fMRI may provide richer data for training such models than EEG ( Mitchell and colleagues obtain an average accuracy of 77 % , and 65 % for the within category setting ) . However , fMRI has several clear disadvantages as a tool for language researchers . First of all , the fine spatial resolution it provides ( down to 2 - 3 mm ) , while of great interest to neuroscientists , is not in itself linguistically informative . Its coarse temporal resolution ( of the order of several seconds ) , makes it ill-suited to analysing on-line linguistic processes . EEG on the other hand , despite its low spatial resolution ( several centimetres ) , gives millisecond-level temporal resolution , Finally , there is no prospect of fMRI being miniaturised , while wearable EEG systems are already becoming commercially available , making experimentation in more ecological settings a possibility ( e.g. , playing with a child , meeting at a desk , walking around ) . In short , while EEG can be used to carry out systematic investigations of categorical distinctions , doing so with fMRI would be prohibitively expensive . Present results indicate that distinctions between categories are easier than distinctions between category elements ; and that selecting the conceptual features by hand gives better results than discovering them automatically . Both of these results however may be due to limitations of the current method . One limitation is that we have been using the same set of features for all concepts , which is likely to blur the distinctions between members of a category more than those between categories . A second limitation of our present methodology is that it is constrained to use very small numbers of semantic features , which limits its applicability . For example it is hard to conceive of a small set of verbs , or other parts-ofspeech , whose co-occurrence patterns could successfully characterise the full range of meaning found in the human lexicon . Even the more economical corpus-extracted conceptual models tend to run in the hundreds of features ( Almuhareb , 2006 ) . We are currently working on variations in the method that will address these shortcomings . The web-based model with manually picked features outperformed all la Repubblica-based models . However , the results attained with repubblica-window-svd are encouraging , especially considering that we are reporting results for an EEG feature configuration optimised for the web data ( see footnote 5 ) , and that la Repubblica is several orders of magnitude smaller than the web . That data sparseness might be the main issue with la Repubblica models is suggested by the fact that repubblica-window-svd is the least sparse of them , since it does not filter data by position or dependency path , and compresses information from many verbs via SVD . In future research , we plan to extract richer models from larger corpora . And as the discriminative accuracy of crosstraining techniques improves , further insights into the relative validity of corpus representations will be attainable . One research aim is to see if individual corpus semantic properties are encoded neurally , so providing strong evidence for a particular model . These techniques may also prove more objective and reliable in evaluating representations of abstract concepts , for which it is more difficult to collect reliable judgements from informants .

The expensive cost of using fMRI hinders studies on the relationship between corpus-extracted models of semantic knowledge and neural activation patterns.

They propose to use EEG activation patterns instead of fMRI to reduce the cost.

They show that using EEG signals with corpus-based models, they can predict word level distinctions significantly above chance.


This paper introduces a new parser evaluation corpus containing around 700 sentences annotated with unbounded dependencies , from seven different grammatical constructions . We run a series of off-theshelf parsers on the corpus to evaluate how well state-of-the-art parsing technology is able to recover such dependencies . The overall results range from 25 % accuracy to 59 % . These low scores call into question the validity of using Parseval scores as a general measure of parsing capability . We discuss the importance of parsers being able to recover unbounded dependencies , given their relatively low frequency in corpora . We also analyse the various errors made on these constructions by one of the more successful parsers . Statistical parsers are now obtaining Parseval scores of over 90 % on the WSJ section of the Penn Treebank ( Bod , 2003 ; Petrov and Klein , 2007 ; Huang , 2008 ; Carreras et al . , 2008 ) . McClosky et al . ( 2006 ) report an F-score of 92.1 % using selftraining applied to the reranker of Charniak and Johnson ( 2005 ) . Such scores , in isolation , may suggest that statistical parsing is close to becoming a solved problem , and that further incremental improvements will lead to parsers becoming as accurate as POS taggers . A single score in isolation can be misleading , however , for a number of reasons . First , the single score is an aggregate over a highly skewed distribution of all constituent types ; evaluations which look at individual constituent or dependency types show that the accuracies on some , semantically important , constructions , such as coordination and PP-attachment , are much lower ( Collins , 1999 ) . Second , it is well known that the accuracy of parsers trained on the Penn Treebank degrades when they are applied to different genres and domains ( Gildea , 2001 ) . Finally , some researchers have argued that the Parseval metrics ( Black et al . , 1991 ) are too forgiving with respect to certain errors and that an evaluation based on syntactic dependencies , for which scores are typically lower , is a better test of parser performance ( Lin , 1995 ; Carroll et al . , 1998 ) . In this paper we focus on the first issue , that the performance of parsers on some constructions is much lower than the overall score . The constructions that we focus on are various unbounded dependency constructions . These are interesting for parser evaluation for the following reasons : one , they provide a strong test of the parser 's knowledge of the grammar of the language , since many instances of unbounded dependencies are difficult to recover using shallow techniques in which the grammar is only superficially represented ; and two , recovering these dependencies is necessary to completely represent the underlying predicateargument structure of a sentence , useful for applications such as Question Answering and Information Extraction . To give an example of the sorts of constructions we are considering , and the ( in)ability of parsers to recover the corresponding unbounded dependencies , none of the parsers that we have tested were able to recover the dependencies shown in bold from the following sentences : We have also developed techniques for recognizing and locating underground nuclear tests through the waves in the ground which they generate . By Monday , they hope to have a sheaf of documents both sides can trust . In this paper we have demonstrated that current parsing technology is poor at recovering some of the unbounded dependencies which are crucial for fully representing the underlying predicateargument structure of a sentence . We have also argued that correct recovery of such dependencies will become more important as parsing technology improves , despite the relatively low frequency of occurrence of the corresponding grammatical constructions . We also see this more focused parser evaluation methodology -in this case construction-focused -as a way of improving parsing technology , as an alternative to the exclusive focus on incremental improvements in overall accuracy measures such as Parseval .

While recent statistical parsers perform well on Penn Treebank, the results can be misleading due to several reasons originating from evaluation and datasets.

They propose a new corpus with unbounded dependencies from difference grammatical constructions.

Their evaluation of existing parsers with the proposed corpus shows lower scores than reported in previous works indicating a poor ability to recover unbounded dependencies.


Learning entailment rules is fundamental in many semantic-inference applications and has been an active field of research in recent years . In this paper we address the problem of learning transitive graphs that describe entailment rules between predicates ( termed entailment graphs ) . We first identify that entailment graphs exhibit a " tree-like " property and are very similar to a novel type of graph termed forest-reducible graph . We utilize this property to develop an iterative efficient approximation algorithm for learning the graph edges , where each iteration takes linear time . We compare our approximation algorithm to a recently-proposed state-of-the-art exact algorithm and show that it is more efficient and scalable both theoretically and empirically , while its output quality is close to that given by the optimal solution of the exact algorithm . Performing textual inference is in the heart of many semantic inference applications such as Question Answering ( QA ) and Information Extraction ( IE ) . A prominent generic paradigm for textual inference is Textual Entailment ( TUE ) ( Dagan et al . , 2009 ) . In TUE , the goal is to recognize , given two text fragments termed text and hypothesis , whether the hypothesis can be inferred from the text . For example , the text " Cyprus was invaded by the Ottoman Empire in 1571 " implies the hypothesis " The Ottomans attacked Cyprus " . Semantic inference applications such as QA and IE crucially rely on entailment rules ( Ravichandran and Hovy , 2002 ; Shinyama and Sekine , 2006 ) or equivalently inference rules , that is , rules that describe a directional inference relation between two fragments of text . An important type of entailment rule specifies the entailment relation between natural language predicates , e.g. , the entailment rule ' X invade Y → X attack Y ' can be helpful in inferring the aforementioned hypothesis . Consequently , substantial effort has been made to learn such rules ( Lin and Pantel , 2001 ; Sekine , 2005 ; Szpektor and Dagan , 2008 ; Schoenmackers et al . , 2010 ) . Textual entailment is inherently a transitive relation , that is , the rules ' x → y ' and ' y → z ' imply the rule ' x → z ' . Accordingly , Berant et al . ( 2010 ) formulated the problem of learning entailment rules as a graph optimization problem , where nodes are predicates and edges represent entailment rules that respect transitivity . Since finding the optimal set of edges respecting transitivity is NP-hard , they employed Integer Linear Programming ( ILP ) to find the exact solution . Indeed , they showed that applying global transitivity constraints improves rule learning comparing to methods that ignore graph structure . More recently , Berant et al . ( Berant et al . , 2011 ) introduced a more efficient exact algorithm , which decomposes the graph into connected components and then applies an ILP solver over each component . Despite this progress , finding the exact solution remains NP-hard -the authors themselves report they were unable to solve some graphs of rather moderate size and that the coverage of their method is limited . Thus , scaling their algorithm to data sets with tens of thousands of predicates ( e.g. , the extractions of Fader et al . ( 2011 ) ) is unlikely . In this paper we present a novel method for learning the edges of entailment graphs . Our method computes much more efficiently an approximate solution that is empirically almost as good as the exact solution . To that end , we first ( Section 3 ) conjecture and empirically show that entailment graphs exhibit a " tree-like " property , i.e. , that they can be reduced into a structure similar to a directed forest . Then , we present in Section 4 our iterative approximation algorithm , where in each iteration a node is removed and re-attached back to the graph in a locally-optimal way . Combining this scheme with our conjecture about the graph structure enables a linear algorithm for node re-attachment . Section 5 shows empirically that this algorithm is by orders of magnitude faster than the state-of-the-art exact algorithm , and that though an optimal solution is not guaranteed , the area under the precision-recall curve drops by merely a point . To conclude , the contribution of this paper is twofold : First , we define a novel modeling assumption about the tree-like structure of entailment graphs and demonstrate its validity . Second , we exploit this assumption to develop a polynomial approximation algorithm for learning entailment graphs that can scale to much larger graphs than in the past . Finally , we note that learning entailment graphs bears strong similarities to related tasks such as Taxonomy Induction ( Snow et al . , 2006 ) and Ontology induction ( Poon and Domingos , 2010 ) , and thus our approach may improve scalability in these fields as well . Learning large and accurate resources of entailment rules is essential in many semantic inference applications . Employing transitivity has been shown to improve rule learning , but raises issues of efficiency and scalability . The first contribution of this paper is a novel modeling assumption that entailment graphs are very similar to FRGs , which is analyzed and validated empirically . The main contribution of the paper is an efficient polynomial approximation algorithm for learning entailment rules , which is based on this assumption . We demonstrate empirically that our method is by orders of magnitude faster than the state-of-the-art exact algorithm , but still produces an output that is almost as good as the optimal solution . We suggest our method as an important step towards scalable acquisition of precise entailment resources . In future work , we aim to evaluate TNF on large graphs that are automatically generated from huge corpora . This of course requires substantial efforts of pre-processing and test-set annotation . We also plan to examine the benefit of TNF in learning similar structures , e.g. , taxonomies or ontologies .

Current inefficient algorithms aim to obtain entailment rules for semantic inference hindering the use of large resources.

They propose an efficient polynomial approximation algorithm that exploits their observation, entailment graphs have a "tree-like" property.

Their iterative algorithm runs by orders of magnitude faster than current exact state-of-the-art solutions while keeping close quality.


Sponsored search is at the center of a multibillion dollar market established by search technology . Accurate ad click prediction is a key component for this market to function since the pricing mechanism heavily relies on the estimation of click probabilities . Lexical features derived from the text of both the query and ads play a significant role , complementing features based on historical click information . The purpose of this paper is to explore the use of word embedding techniques to generate effective text features that can capture not only lexical similarity between query and ads but also the latent user intents . We identify several potential weaknesses of the plain application of conventional word embedding methodologies for ad click prediction . These observations motivated us to propose a set of novel joint word embedding methods by leveraging implicit click feedback . We verify the effectiveness of these new word embedding models by adding features derived from the new models to the click prediction system of a commercial search engine . Our evaluation results clearly demonstrate the effectiveness of the proposed methods . To the best of our knowledge this work is the first successful application of word embedding techniques for the task of click prediction in sponsored search . Sponsored search is a multibillion dollar market ( Easley and Kleinberg , 2010 ) that makes most search engine revenue and is one of the most successful ways for advertisers to reach their intended audiences . When search engines deliver results to a user , sponsored advertisement impressions ( ads ) are shown alongside the organic search results ( Figure 1 ) . Typically the advertiser pays the search engine based on the pay-per-click model . In this model the advertiser pays only if the impression that accompanies the search results is clicked . The price is usually set by a generalized second-price ( GSP ) auction ( Edelman et al . , 2005 ) that encourages advertisers to bid truthfully . An advertiser wins if the expected revenue for this advertiser , which is the bid Figure 1 : Sponsored ads when " pizza " was searched at Yahoo ! ( www.yahoo.com ) . price times the expected click probability ( also know as click through rate , or CTR ) , is ranked the highest . The price the advertiser pays , known as cost-per-click ( CPC ) , is the bid price for the second ranked advertiser times the ratio of the expected CTR between the second and first ranked advertisers . From this discussion it should be clear that CTR plays a key role in deciding both the ranking and the pricing of the ads . Therefore it is very important to predict CTR accurately . The state of the art search engine typically uses a machine learning model to predict CTR by exploiting various features that have been found useful in practice . These include historical click performance features such as historical click probability for the query , the ad , the user , and a combination of these ; contextual features such as temporal and geographical information ; and text-based features such as query keywords or ad title and description . Among these , historical click performance features often have the most predictive power for queries , ads and users that have registered many impressions . For queries , ads and users that have not registered many impressions , however , historical CTR may have too high a variance to be useful . Hillard et al . ( 2011 ) observed that the number of impressions and clicks recorded on query-ad pairs have a very long tail : only 61 % of queries has greater than three clicks . They also reported a drastic drop in the accuracy of the click prediction model when fewer historical observations are available . Furthermore , fine-grained historical CTR information takes a huge amount of space , which makes it costly to maintain . On the other hand , text features are always readily available , and thus are particularly useful for those cases for which there is insufficient historical information . Multiple researchers , for example ( Richardson , 2007 ; Cheng and Cantú-Paz , 2010 ) , reported the usage of text features including simple lexical similarity scores between the query and ads , word or phrase overlaps and the number of overlapping words and characters . Such features rely on the assumption that query-ad overlap is correlated with perceived relevance . While this is true to a certain extent , the use of simple lexical similarity can not capture semantic information such as synonyms , entities of the same type and strong relationships between entities ( e.g. CEO-company , brandmodel , part-of ) . Recently a host of studies on word embedding have been conducted ; all map words into a vector space such that semantically relevant words are placed near each other in the space ( Mikolov et al . , 2013a ; Pennington et al . , 2014 ; Baroni et al . , 2014 ) . The use of continuous word vectors has been shown to be helpful for a wide range of NLP tasks by better capturing both syntactic and semantic information than simple lexical features ( Socher et al . , 2012a ) . No previous research on sponsored search has successfully used word embeddings to generate text features . In this paper , we explore the use of word embeddings for click prediction . However , it is clear that conventional word embeddings ( which solely rely on word co-occurrence in a context window ) can only offer limited discriminative power because queries and ad text are typically very short . In addition , conventional word embeddings can not capture user intents , preferences and desires . Wang et al . ( 2013 ) showed that specific frequently occurring lexical patterns , e.g. , x% off , guaranteed return in x days and official site , are effective in triggering users desires , and thus lead to significant differences in CTR . Conventional word embeddings can not capture these phenomena since they do not incorporate the implicit feedback users provide through clicks and non-clicks . These observations naturally lead us to leverage click feedback to infuse users ' intentions and desires into the vector space . The simplest way to harness click feedback is to train conventional word embedding models on a corpus that only includes clicked impressions , where each " sentence " is constructed by mixing the query and ad text . Having trained a word embedding model , we simply take the average of word vectors of the query and ads respectively to obtain sentence ( or paragraph ) vectors , which in turn are used to compute the similarity scores between the query and ads . Our experiments show that this method does improve click prediction performance . However , this method has several potential weaknesses . First , the use of only clicked impressions ignores the large amount of negative signals contained in the non-clicked ad impressions . Second , the use of indirect signals ( word co-occurrences ) can be noisy or even harmful to our ultimate goal ( accurate click prediction ) when it is combined with direct signals ( impressions with click feedback ) . Third , without explicit consideration about the averaging step in the training process of word embedding models , a simple averaging scheme across word vectors may be a suboptimal . We therefore propose several joint word embedding models ; all of these aim to put query vectors close to relevant ad vectors by explicitly utilizing both positive and negative click feedback . We evaluate all these models against a large sponsored search data set from a commercial search engine , and demonstrate that our proposed models significantly improve click prediction performance . The rest of this paper is organized as follows . In Section 2 we present a brief summary of related work . In Section 3 we give some background information on ad click prediction in sponsored search . In Section 4 we describe our methods . In Section 5 we discuss our experiments . We finish with some conclusions and future directions in Section 6 . In this paper we explored the use of word embedding techniques to overcome the shortcomings of traditional lexical features for ad click prediction in sponsored search . We identified several potential weaknesses of the plain application of conventional word embedding methodologies : the lack of the right machinery to harness both positive and negative click feedback , the limited utility of pure word co-occurrence signals , and no consideration of vector composition in the word embedding training process . We proposed a set of new implicit feedback-based joint word embedding methods to address those issues . We evaluated the new word embedding methods in the context of a very good baseline click prediction system , on a large scale data set collected from Yahoo ! search engine logs . Our experimental results clearly demonstrate the effectiveness of the proposed methods . We also presented several examples for qualitative analysis to advance our understanding on how each algorithm really contributes to the improved performance . To the best of our knowledge this work is the first successful application of word embedding techniques for the sponsored search task . There are multiple interesting research directions for future work . One of these directions is to extend the vocabulary by identifying significant phrases ( as well as words ) before training word vectors . Hillard et al . ( 2011 ) employed Conditional Random Fields to divide queries with multiple words into segments and collected historical CTR on the segment level . We also like to investigate more structured embedding methods such as RNNs ( probably for ad descriptions ) . In case the computational cost of such methods are too high to be practical for sponsored search , we can employ them only for a small fraction of ads filtered by faster methods . It may be possible to deal with the implicit negative feedback of unclicked ad impressions in a more principled way by adopting ranking-based loss functions . However , this is only possible with the extra cost of identifying and aggregating related ads into a single transaction . Though not directly related to NLP , yet another promising direction is to jointly embed not only text data but also a variety of user activities ( e.g. , organic search results , mobile app usages , other daily activities ) all together in the same vector space . Since many of the different sources contain their own unique information , we might be able to obtain a much better understanding about the user state and intent through this rich joint embedding space . Joint embedding with rich information can also help us to perform automatic clustering of users , eventually leading to powerful smoothing methods for personalized historical CTR statistics .

Conventional word embeddings with a simple integration of click feedback information and averaging to obtain sentence representations do not work well for ad click prediction.

They propose several joint word embedding methods to leverage positive and negative click feedback which put query vectors close to relevant ad vectors.

The use of features obtained from the new models improves on a large sponsored search data of commercial Yahoo! search engine.


We propose a new model for unsupervised POS tagging based on linguistic distinctions between open and closed-class items . Exploiting notions from current linguistic theory , the system uses far less information than previous systems , far simpler computational methods , and far sparser descriptions in learning contexts . By applying simple language acquisition techniques based on counting , the system is given the closed-class lexicon , acquires a large open-class lexicon and then acquires disambiguation rules for both . This system achieves a 20 % error reduction for POS tagging over state-of-the-art unsupervised systems tested under the same conditions , and achieves comparable accuracy when trained with much less prior information . All recent research on unsupervised tagging , as well as the majority of work on supervised taggers , views POS tagging as a sequential labeling problem and treats all POS tags , both closed-and open-class , as roughly equivalent . In this work we explore a different understanding of the tagging problem , viewing it as a process of first identifying functional syntactic contexts , which are flagged by closed-class items , and then using these functional contexts to determine the POS labels . This disambiguation model differs from most previous work in three ways : 1 ) it uses different encodings over two distinct domains ( roughly open-and closed-class words ) with complementary distribution ( and so decodes separately ) ; 2 ) it is deterministic and 3 ) it is non-lexicalized . By learning disambiguation models for open-and closed-classes separately , we found that the deterministic , rulebased model can be learned from unannotated data by a simple strategy of selecting a rule in each appropriate context with the highest count . In contrast to this , most previous work on unsupervised tagging ( especially for English ) concentrates on improving the parameter estimation techniques for training statistical disambiguation models from unannotated data . For example , ( Smith&Eisner , 2005 ) proposes contrastive estimation ( CE ) for log-linear models ( CRF ) , achieving the current state-of-the-art performance of 90.4 % ; ( Goldwater&Griffiths , 2007 ) applies a Bayesian approach to improve maximumlikelihood estimation ( MLE ) for training generative models ( HMM ) . In the main experiments of both of these papers , the disambiguation model is learned , but the algorithms assume a complete knowledge of the lexicon with all possible tags for each word . In this work , we propose making such a large lexicon unnecessary by learning the bulk of the lexicon along with learning a disambiguation model . Little previous work has been done on this natural and simple idea because the clusters found by previous induction schemes are not in line with the lexical categories that we care about . ( Chan , 2008 ) is perhaps the first with the intention of generating " a discrete set of clusters . " By applying similar techniques to ( Chan , 2008 ) , which we discuss later , we can generate clusters that closely approximate the central open-class lexical categories , a major advance , but we still require a closed-class lexicon specifying possible tags for these words . This asymmetry in our lexicon acquisition model conforms with our understanding of natural language as structured data over two distinct domains with complementary distribution : open-class ( lexical ) and closed-class ( functional ) . Provided with only a closed-class lexicon of 288 words , about 0.6 % of the full lexicon , the system acquires a large open-class lexicon and then acquires disambiguation rules for both closed-and open-class words , achieving a tagging accuracy of 90.6 % for a 24k dataset , as high as the current state-of-the-art ( 90.4 % ) achieved with a complete dictionary . In the test condition where both algorithms are provided with a full lexicon , and are trained and evaluated over the same 96k dataset , we reduce the tagging error by up to 20 % . In Section 2 we explain our understanding of the POS tagging problem in detail and define the notions of functional context and open-and closedclass elements . Then we will introduce our methods for acquiring the lexicon ( Section 3 ) and learning disambiguation models ( Section 4 , 5 and 6 ) step by step . Results are reported in Section 7 followed by Section 8 which discusses the linguistic motivation behind this work and the simplicity and efficiency of our model . In this work on unsupervised tagging , we combine lexicon acquisition with the learning of a POS disambiguation model . Moreover , the disambiguation model we used is deterministic , nonlexicalized and defined over two distinct domains with complementary distribution ( open-and closed-class ) . Building a lexicon based on induced clusters requires our morphological knowledge of three special endings in English : -ing , -ed and -s ; on the other hand , to reduce the feature space used for category induction , we utilize vectors of functional features only , exploiting our knowledge of the role of determiners and modal verbs . However , the above information is restricted to the lexicon acquisition model . Taking a lexicon as input , which either consists of a known closed-class lexicon together with an acquired open-class lexicon or is composed by automatic extraction from the Penn Treebank , we need NO language-specific knowledge for learning the disambiguation model . We would like to point the reader to ( Chan , 2008 ) for more discussion on Category induction14 ; and discussions below will concentrate on the proposed disambiguation model . Current Chomskian theory , developed in the Minimalist Program ( MP ) ( Chomsky , 2006 ) , argues ( very roughly speaking ) that the syntactic structure of a sentence is built around a scaffolding provided by a set of functional elements15 . Each of these provides a large tree fragment ( roughly corresponding to what Chomsky calls a phase ) that provide the piece parts for full utterances . Chomsky observes that when these fragments combine , only the very edge of the fragments can change and that the internal structure of these fragments is rigid ( he labels this observation the Phase Impenetrability Condition , PIC ) . With the belief in PIC , we propose the concept of functional context , in which category property can be determined ; also we notice the distinct distribution of the elements ( functional ) on the edge of phase and those ( lexical ) assembled within the phase . Instead of chasing the highest possible performance by using the strongest method possible , we wanted to explore how well a deterministic , nonlexicalized model , following certain linguistic intuitions , can approach the NLP problem . For the unsupervised tagging task , this simple model , with less than two hundred rules learned , even outperforms non-deterministic generative models with ten of thousands of parameters . Another motivation for our pursuit of this deterministic , non-lexicalized model is computational efficiency 16 . It takes less than 3 minutes total for our model to acquire the lexicon , learn the disambiguation model , tag raw data and evaluate the output for a 96k dataset on a small laptop17 . And a model using only counting and selecting is common in the research field of language acquisition and perhaps more compatible to the way humans process language . We are certainly aware that our work does not yet address two problems : 1 ) . How the system can be adapted to work for other languages and 2 ) How to automatically obtain the knowledge of functional elements . We believe that , given the proper understanding of functional elements , our system will be easily adapted to other languages , but we clearly need to test this hypothesis . Also , we are highly interested in completing our system by incorporating the acquisition of functional elements . ( Chan , 2008 ) presents an extensive discussion of his work on morphological induction and ( Mintz et al . , 2002 ) presents interesting psychological experiments we can build on to acquire closed-class words .

Current approaches tackle unsupervised POS tagging as a sequential labelling problem and require a complete knowledge of the lexicon.

They propose to first identify functional syntactic contexts and then use them to make predictions for POS tagging.

The proposed method achieves equivalent performance by using 0.6% of the lexical knowledge used in baseline models.


Non-autoregressive Transformer is a promising text generation model . However , current non-autoregressive models still fall behind their autoregressive counterparts in translation quality . We attribute this accuracy gap to the lack of dependency modeling among decoder inputs . In this paper , we propose CNAT , which learns implicitly categorical codes as latent variables into the non-autoregressive decoding . The interaction among these categorical codes remedies the missing dependencies and improves the model capacity . Experiment results show that our model achieves comparable or better performance in machine translation tasks than several strong baselines . Non-autoregressive Transformer ( NAT , Gu et al . , 2018 ; Wang et al . , 2019 ; Lee et al . , 2018 ; Ghazvininejad et al . , 2019 ) is a promising text generation model for machine translation . It introduces the conditional independent assumption among the target language outputs and simultaneously generates the whole sentence , bringing in a remarkable efficiency improvement ( more than 10× speed-up ) versus the autoregressive model . However , the NAT models still lay behind the autoregressive models in terms of BLEU ( Papineni et al . , 2002 ) for machine translation . We attribute the low-quality of NAT models to the lack of dependencies modeling for the target outputs , making it harder to model the generation of the target side translation . A promising way is to model the dependencies of the target language by the latent variables . A line of research works ( Kaiser et al . , 2018 ; Roy et al . , 2018 ; Shu et al . , 2019 ; Ma et al . , 2019 ) introduce latent variable modeling to the non-autoregressive Transformer and improves translation quality . The latent variables could be regarded as the springboard to bridge the modeling gap , introducing more informative decoder inputs than the previ-ously copied inputs . More specifically , the latentvariable based model first predicts a latent variable sequence conditioned on the source representation , where each variable represents a chunk of words . The model then simultaneously could generate all the target tokens conditioning on the latent sequence and the source representation since the target dependencies have been modeled into the latent sequence . However , due to the modeling complexity of the chunks , the above approaches always rely on a large number ( more than 2 15 , Kaiser et al . , 2018 ; Roy et al . , 2018 ) of latent codes for discrete latent spaces , which may hurt the translation efficiencythe essential goal of non-autoregressive decoding . Akoury et al . ( 2019 ) introduce syntactic labels as a proxy to the learned discrete latent space and improve the NATs ' performance . The syntactic label greatly reduces the search space of latent codes , leading to a better performance in both quality and speed . However , it needs an external syntactic parser to produce the reference syntactic tree , which may only be effective in limited scenarios . Thus , it is still challenging to model the dependency between latent variables for non-autoregressive decoding efficiently . In this paper , we propose to learn a set of latent codes that can act like the syntactic label , which is learned without using the explicit syntactic trees . To learn these codes in an unsupervised way , we use each latent code to represent a fuzzy target category instead of a chunk as the previous research ( Akoury et al . , 2019 ) . More specifically , we first employ vector quantization ( Roy et al . , 2018 ) to discretize the target language to the latent space with a smaller number ( less than 128 ) of latent variables , which can serve as the fuzzy word-class information each target language word . We then model the latent variables with conditional random fields ( CRF , Lafferty et al . , 2001 ; Sun et al . , 2019 ) . To avoid the mismatch of the training and inference for latent variable modeling , we propose using a gated neural network to form the decoder inputs . Equipping it with scheduled sampling ( Bengio et al . , 2015 ) , the model works more robustly . Experiment results on WMT14 and IWSLT14 show that CNAT achieves the new state-of-theart performance without knowledge distillation . With the sequence-level knowledge distillation and reranking techniques , the CNAT is comparable to the current state-of-the-art iterative-based model while keeping a competitive decoding speedup . We propose CNAT , which implicitly models the categorical codes of the target language , narrowing the performance gap between the nonautoregressive decoding and autoregressive decoding . Specifically , CNAT builds upon the latent Transformer and models the target-side categorical information with vector quantization and conditional random fields ( CRF ) model . We further employ a gated neural network to form the decoder inputs . Equipped with the scheduled sampling , CNAT works more robust . As a result , the CNAT achieves a significant improvement and moves closer to the performance of the Transformer on machine translation . For each latent variable , we list the top 3 frequent pos tags and their corresponding percentages .

Non-autoregressive translation models fall behind their autoregressive counterparts in translation quality due to the lack of dependency modelling for the target outputs.

They propose a non-autoregressive transformer-based model which implicitly learns categorical codes as latent variables into the decoding to complement missing dependencies.

The proposed model achieves state-of-the-art without knowledge distillation and a competitive decoding speedup with an interactive-based model when coupled with knowledge distillation and reranking techniques.


The clustering-based unsupervised relation discovery method has gradually become one of the important methods of open relation extraction ( OpenRE ) . However , high-dimensional vectors can encode complex linguistic information which leads to the problem that the derived clusters can not explicitly align with the relational semantic classes . In this work , we propose a relationoriented clustering model and use it to identify the novel relations in the unlabeled data . Specifically , to enable the model to learn to cluster relational data , our method leverages the readily available labeled data of pre-defined relations to learn a relationoriented representation . We minimize distance between the instance with same relation by gathering the instances towards their corresponding relation centroids to form a cluster structure , so that the learned representation is cluster-friendly . To reduce the clustering bias on predefined classes , we optimize the model by minimizing a joint objective on both labeled and unlabeled data . Experimental results show that our method reduces the error rate by 29.2 % and 15.7 % , on two datasets respectively , compared with current SOTA methods . Relation extraction ( RE ) , a crucial basic task in the field of information extraction , is of the utmost practical interest to various fields including web search ( Xiong et al . , 2017 ) , knowledge base completion ( Bordes et al . , 2013 ) , and question answering ( Yu et al . , 2017 ) . However , conventional RE paradigms such as supervision and distant supervision are generally designed for pre-defined relations , which can not deal with new emerging relations in the real world . Under this background , open relation extraction ( OpenRE ) has been widely studied for its use Figure 1 : Although both instances S 2 and S 3 express founded relation while S 1 expresses CEO relation , the distance between S 1 and S 2 is still smaller than that between S 2 and S 3 . This is because there may be more similar surface information ( e.g. word overlapping ) or syntactic structure between S 1 and S 2 , thus the derived clusters can not explicitly align with relations . in extracting new emerging relational types from open-domain corpora . The approaches used to handle open relations roughly fall into one of two groups . The first group is open information extraction ( OpenIE ) ( Etzioni et al . , 2008 ; Yates et al . , 2007 ; Fader et al . , 2011 ) , which directly extracts related phrases as representations of different relational types . However , if not properly canonicalized , the extracted relational facts can be redundant and ambiguous . The second group is unsupervised relation discovery ( Yao et al . , 2011 ; Shinyama and Sekine , 2006 ; Simon et al . , 2019 ) . In this type of research , much attention has been focused on unsupervised clustering-based RE methods , which cluster and recognize relations from high-dimensional representations ( Elsahar et al . , 2017 ) . Recently , the self-supervised signals in pretrained language model are further exploited for clustering optimization ( Hu et al . , 2020 ) . However , many studies show that highdimensional embeddings can encode complex linguistic information such as morphological ( Peters et al . , 2018 ) , local syntactic ( Hewitt and Manning , 2019 ) , and longer range semantic information ( Jawahar et al . , 2019 ) . Consequently , the distance of representation is not completely consistent with relational semantic similarity . Although Hu et al . ( 2020 ) use self-supervised signals to optimize clustering , there is still no guarantee that the learned clusters will explicitly align with the desired relational semantic classes ( Xing et al . , 2002 ) . As shown in Figure 1 , we use the method proposed by Hu et al . ( 2020 ) to get the instance representations . Although both instances S 2 and S 3 express the founded relation , the euclidean distance between them is larger than that between S 1 and S 2 , which express different relation . Obviously , the clustering algorithm tends to group instances S 1 and S 2 together , rather than S 2 and S 3 which express the same relation . In this work , we propose a relation-oriented clustering method . To enable the model to learn to cluster relational data , pre-defined relations and their existing labeled instances are leveraged to optimize a non-linear mapping , which transforms high-dimensional entity pair representations into relation-oriented representations . Specifically , we minimize distance between the instances with same relation by gathering the instances representation towards their corresponding relation centroids to form the cluster structure , so that the learned representation is cluster-friendly . In order to reduce the clustering bias on the predefined classes , we iteratively train the entity pair representations by optimizing a joint objective function on the labeled and unlabeled subsets of the data , improving both the supervised classification of the labeled data , and the clustering of the unlabeled data . In addition , the proposed method can be easily extended to incremental learning by classifying the pre-defined and novel relations with a unified classifier , which is often desirable in real-world applications . Our experimental results show that our method outperforms current state-of-the-art methods for OpenRE . Our codes are publicly available at Github * . To summarize , the main contributions of our work are as follows : ( 1 ) we propose a novel relation-oriented clustering method RoCORE to enable model to learn to cluster relational data ; ( 2 ) the proposed method achieves the incremental learning of unlabeled novel relations , which is often desirable in real-world applications ; ( 3 ) experimental results show that our method reduces * https://github.com / Ac-Zyx / RoCORE . the error rate by 29.2 % and 15.7 % , on two realworld datasets respectively , compared with current state-of-the-art OpenRE methods . In this work , we introduce a relation-oriented clustering method that extends the current unsupervised clustering-based OpenRE method . The proposed method leverages the labeled data of pre-defined relations to learn a relation-oriented representation from which the derived clusters explicitly align with relational classes . Iterative joint training method effectively reduces the unwanted bias on labeled data . In addition , the proposed method can be easily extended to incremental learning of novel relations . Experimental results show that our method outperforms SOTA methods for OpenRE .

Even though high-dimensional vectors that can encode complex information used for relation extraction are not guaranteed to be consistent with relational semantic similarity.

They propose to use available relation labeled data to obtain relation-oriented representation by minimizing the distance between the same relation instances.

The proposed approach can reduce error rates significantly from the best models for open relation extraction.


Prior use of machine learning in genre classification used a list of labels as classification categories . However , genre classes are often organised into hierarchies , e.g. , covering the subgenres of fiction . In this paper we present a method of using the hierarchy of labels to improve the classification accuracy . As a testbed for this approach we use the Brown Corpus as well as a range of other corpora , including the BNC , HGC and Syracuse . The results are not encouraging : apart from the Brown corpus , the improvements of our structural classifier over the flat one are not statistically significant . We discuss the relation between structural learning performance and the visual and distributional balance of the label hierarchy , suggesting that only balanced hierarchies might profit from structural learning . Automatic genre identification ( AGI ) can be traced to the mid-1990s ( Karlgren and Cutting , 1994 ; Kessler et al . , 1997 ) , but this research became much more active in recent years , partly because of the explosive growth of the Web , and partly because of the importance of making genre distinctions in NLP applications . In Information Retrieval , given the large number of web pages on any given topic , it is often difficult for the users to find relevant pages that are in the right genre ( Vidulin et al . , 2007 ) . As for other applications , the accuracy of many tasks , such as machine translation , POS tagging ( Giesbrecht and Evert , 2009 ) or identification of discourse relations ( Webber , 2009 ) relies of defining the language model suitable for the genre of a given text . For example , the accuracy of POS tagging reaching 96.9 % on newspaper texts drops down to 85.7 % on forums ( Giesbrecht and Evert , 2009 ) , i.e. , every seventh word in forums is tagged incorrectly . This interest in genres resulted in a proliferation of studies on corpus development of web genres and comparison of methods for AGI . The two corpora commonly used for this task are KI-04 ( Meyer zu Eissen and Stein , 2004 ) and Santinis ( Santini , 2007 ) . The best results reported for these corpora ( with 10-fold cross-validation ) reach 84.1 % on KI-04 and 96.5 % accuracy on Santinis ( Kanaris and Stamatatos , 2009 ) . In our research ( Sharoff et al . , 2010 ) we produced even better results on these two benchmarks ( 85.8 % and 97.1 % , respectively ) . However , this impressive accuracy is not realistic in vivo , i.e. , in classifying web pages retrieved as a result of actual queries . One reason comes from the limited number of genres present in these two collections ( eight genres in KI-04 and seven in Santinis ) . As an example , only front pages of online newspapers are listed in Santinis , but not actual newspaper articles , so once an article is retrieved , it can not be assigned to any class at all . Another reason why the high accuracy is not useful concerns the limited number of sources in each collection , e.g. , all FAQs in Santinis come from either a website with FAQs on hurricanes or another one with tax advice . In the end , a classifier built for FAQs on this training data relies on a high topic-genre correlation in this particular collection and fails to spot any other FAQs . There are other corpora , which are more diverse in the range of their genres , such as the fifteen genres of the Brown Corpus ( Kučera and Francis , 1967 ) or the seventy genres of the BNC ( Lee , 2001 ) , but because of the number of genres in them and the diversity of documents within each genre , the accuracy of prior work on these collections is much less impressive . For example , Karlgren and Cutting ( 1994 ) using linear discriminant analysis achieve an accuracy of 52 % without us-ing cross-validation ( the entire Brown Corpus was used as both the test set and training set ) , with the accuracy improving to 65 % when the 15 genres are collapsed into 10 , and to 73 % with only 4 genres ( Figure 1 ) . This result suggests the importance of the hierarchy of genres . Firstly , making a decision on higher levels might be easier than on lower levels ( fiction or non-fiction rather than science fiction or mystery ) . Secondly , we might be able to improve the accuracy on lower levels , by taking into account the relevant position of each node in the hierarchy ( distinguishing between reportage or editorial becomes easier when we know they are safely under the category of press ) . This paper explores a way of using information on the hierarchy of labels for improving fine-grained genre classification . To the best of our knowledge , this is the first work presenting structural genre classification and distance measures for genres . In Section 2 we present a structural reformulation of Support Vector Machines ( SVMs ) that can take similarities between different genres into account . This formulation necessitates the development of distance measures between different genres in a hierarchy , of which we present three different types in Section 3 , along with possible estimation procedures for these distances . We present experiments with these novel structural SVMs and distance measures on three different corpora in Section 4 . Our experiments show that structural SVMs can outperform the non-structural standard . However , the improvement is only statistically significant on the Brown corpus . In Section 5 we investigate potential reasons for this , including the ( im)balance of different genre hierarchies and problems with our distance measures . In this paper , we have evaluated structural learning approaches to genre classification using sev-eral different genre distance measures . Although we were able to improve on non-structural approaches for the Brown corpus , we found it hard to improve over flat SVMs on other corpora . As potential reasons for this negative result , we suggest that current genre hierarchies are either not of sufficient depth or are visually or distributionally imbalanced . We think further investigation into the relationship between hierarchy balance and structural learning is warranted . Further investigation is also needed into the appropriateness of n-gram features for genre identification as well as good measures of genre distance . In the future , an important task would be the refinement or unsupervised generation of new hierarchies , using information theoretic or data-driven approaches . For a full assessment of hierarchical learning for genre classification , the field of genre studies needs a testbed similar to the Reuters or 20 Newsgroups datasets used in topic-based IR with a balanced genre hierarchy and a representative corpus of reliably annotated webpages . With regard to algorithms , we are also interested in other formulations for structural SVMs and their large-scale implementation as well as the combination of different distance measures , for example in ensemble learning .

Existing genre classification methods achieve high accuracy without regarding hierarchical structures exhibiting unrealistic experimental setups such as a limited number of genres and sources.

They propose a structural reformulation of the Support Vector Machine to take hierarchical information of genres into account by using similarities between different genres.

The proposed model outperforms non-hierarchical models on only one corpus and they discuss that it may be due to insufficient depth or inbalance of hierarchies.