In this paper , we explore correlation of dependency relation paths to rank candidate answers in answer extraction . Using the correlation measure , we compare dependency relations of a candidate answer and mapped question phrases in sentence with the corresponding relations in question . Different from previous studies , we propose an approximate phrase mapping algorithm and incorporate the mapping score into the correlation measure . The correlations are further incorporated into a Maximum Entropy-based ranking model which estimates path weights from training . Experimental results show that our method significantly outperforms state-ofthe-art syntactic relation-based methods by up to 20 % in MRR . Answer Extraction is one of basic modules in open domain Question Answering ( QA ) . It is to further process relevant sentences extracted with Passage / Sentence Retrieval and pinpoint exact answers using more linguistic-motivated analysis . Since QA turns to find exact answers rather than text snippets in recent years , answer extraction becomes more and more crucial . Typically , answer extraction works in the following steps : • Recognize expected answer type of a question . • Annotate relevant sentences with various types of named entities . • Regard the phrases annotated with the expected answer type as candidate answers . • Rank candidate answers . In the above work flow , answer extraction heavily relies on named entity recognition ( NER ) . On one hand , NER reduces the number of candidate answers and eases answer ranking . On the other hand , the errors from NER directly degrade answer extraction performance . To our knowledge , most top ranked QA systems in TREC are supported by effective NER modules which may identify and classify more than 20 types of named entities ( NE ) , such as abbreviation , music , movie , etc . However , developing such named entity recognizer is not trivial . Up to now , we have n't found any paper relevant to QA-specific NER development . So , it is hard to follow their work . In this paper , we just use a general MUC-based NER , which makes our results reproducible . A general MUC-based NER ca n't annotate a large number of NE classes . In this case , all noun phrases in sentences are regarded as candidate answers , which makes candidate answer sets much larger than those filtered by a well developed NER . The larger candidate answer sets result in the more difficult answer extraction . Previous methods working on surface word level , such as density-based ranking and pattern matching , may not perform well . Deeper linguistic analysis has to be conducted . This paper proposes a statistical method which exploring correlation of dependency relation paths to rank candidate answers . It is motivated by the observation that relations between proper answers and question phrases in candidate sentences are always similar to the corresponding relations in question . For example , the question " What did Alfred Nobel invent ? " and the candidate sentence " ... in the will of Swedish industrialist Alfred Nobel , who invented dynamite . " For each question , firstly , dependency relation paths are defined and extracted from the question and each of its candidate sentences . Secondly , the paths from the question and the candidate sentence are paired according to question phrase mapping score . Thirdly , correlation between two paths of each pair is calculated by employing Dynamic Time Warping algorithm . The input of the calculation is correlations between dependency relations , which are estimated from a set of training path pairs . Lastly , a Maximum Entropy-based ranking model is proposed to incorporate the path correlations and rank candidate answers . Furthermore , sentence supportive measure are presented according to correlations of relation paths among question phrases . It is applied to re-rank the candidate answers extracted from the different candidate sentences . Considering phrases may provide more accurate information than individual words , we extract dependency relations on phrase level instead of word level . The experiment on TREC questions shows that our method significantly outperforms a densitybased method by 50 % in MRR and three stateof-the-art syntactic-based methods by up to 20 % in MRR . Furthermore , we classify questions by judging whether NER is used . We investigate how these methods perform on the two question sets . The results indicate that our method achieves better performance than the other syntactic-based methods on both question sets . Especially for more difficult questions , for which NER may not help , our method improves MRR by up to 31 % . The paper is organized as follows . Section 2 discusses related work and clarifies what is new in this paper . Section 3 presents relation path correlation in detail . Section 4 and 5 discuss how to incorporate the correlations for answer ranking and re-ranking . Section 6 reports experiment and results . In this paper , we propose a relation path correlation-based method to rank candidate answers in answer extraction . We extract and pair relation paths from questions and candidate sentences . Next , we measure the relation path correlation in each pair based on approximate phrase mapping score and relation sequence alignment , which is calculated by DTW algorithm . Lastly , a ME-based ranking model is proposed to incorporate the path correlations and rank candidate answers . The experiment on TREC questions shows that our method significantly outperforms a density-based method by 50 % in MRR and three state-of-the-art syntactic-based methods by up to 20 % in MRR . Furthermore , the method is especially effective for difficult questions , for which NER may not help . Therefore , it may be used to further enhance state-of-the-art QA systems even if they have a good NER . In the future , we are to further evaluate the method based on the overall performance of a QA system and adapt it to sentence retrieval task .

A generally accessible NER system for QA systems produces a larger answer candidate set which would be hard for current surface word-level ranking methods.

They propose a statistical method which takes correlations of dependency relation paths computed by the Dynamic Time Wrapping algorithm into account for ranking candidate answers.

The proposed method outperforms state-of-the-art syntactic relation-based methods by up to 20% and shows it works even better on harder questions where NER performs poorly.


Recently , many works have tried to augment the performance of Chinese named entity recognition ( NER ) using word lexicons . As a representative , Lattice-LSTM ( Zhang and Yang , 2018 ) has achieved new benchmark results on several public Chinese NER datasets . However , Lattice-LSTM has a complex model architecture . This limits its application in many industrial areas where real-time NER responses are needed . In this work , we propose a simple but effective method for incorporating the word lexicon into the character representations . This method avoids designing a complicated sequence modeling architecture , and for any neural NER model , it requires only subtle adjustment of the character representation layer to introduce the lexicon information . Experimental studies on four benchmark Chinese NER datasets show that our method achieves an inference speed up to 6.15 times faster than those of state-ofthe-art methods , along with a better performance . The experimental results also show that the proposed method can be easily incorporated with pre-trained models like BERT . 1 Named Entity Recognition ( NER ) is concerned with the identification of named entities , such as persons , locations , and organizations , in unstructured text . NER plays an important role in many downstream tasks , including knowledge base construction ( Riedel et al . , 2013 ) , information retrieval ( Chen et al . , 2015 ) , and question answering ( Diefenbach et al . , 2018 ) . In languages where words are naturally separated ( e.g. , English ) , NER has been conventionally formulated as a sequence labeling problem , and the state-of-the-art results have been achieved using neural-network-based models ( Huang et al . , 2015 ; Chiu and Nichols , 2016 ; Liu et al . , 2018 ) . Compared with NER in English , Chinese NER is more difficult since sentences in Chinese are not naturally segmented . Thus , a common practice for Chinese NER is to first perform word segmentation using an existing CWS system and then apply a word-level sequence labeling model to the segmented sentence ( Yang et al . , 2016 ; He and Sun , 2017b ) . However , it is inevitable that the CWS system will incorrectly segment query sentences . This will result in errors in the detection of entity boundary and the prediction of entity category in NER . Therefore , some approaches resort to performing Chinese NER directly at the character level , which has been empirically proven to be effective ( He and Wang , 2008 ; Liu et al . , 2010 ; Li et al . , 2014 ; Liu et al . , 2019 ; Sui et al . , 2019 ; Gui et al . , 2019b ; Ding et al . , 2019 ) . A drawback of the purely character-based NER method is that the word information is not fully exploited . With this consideration , Zhang and Yang , ( 2018 ) proposed Lattice-LSTM for incorporating word lexicons into the character-based NER model . Moreover , rather than heuristically choosing a word for the character when it matches multiple words in the lexicon , the authors proposed to preserve all words that match the character , leaving the subsequent NER model to determine which word to apply . To realize this idea , they introduced an elaborate modification to the sequence modeling layer of the LSTM-CRF model ( Huang et al . , 2015 ) . Experimental studies on four Chinese NER datasets have verified the effectiveness of Lattice-LSTM . However , the model architecture of Lattice-LSTM is quite complicated . In order to introduce lexicon information , Lattice-LSTM adds several additional edges between nonadjacent characters in the input sequence , which significantly slows its training and inference speeds . In addition , it is difficult to transfer the structure of Lattice-LSTM to other neural-network architectures ( e.g. , convolutional neural networks and transformers ) that may be more suitable for some specific tasks . In this work , we propose a simpler method to realize the idea of Lattice-LSTM , i.e. , incorporating all the matched words for each character to a character-based NER model . The first principle of our model design is to achieve a fast inference speed . To this end , we propose to encode lexicon information in the character representations , and we design the encoding scheme to preserve as much of the lexicon matching results as possible . Compared with Lattice-LSTM , our method avoids the need for a complicated model architecture , is easier to implement , and can be quickly adapted to any appropriate neural NER model by adjusting the character representation layer . In addition , ablation studies show the superiority of our method in incorporating more complete and distinct lexicon information , as well as introducing a more effective word-weighting strategy . The contributions of this work can be summarized as follows : • We propose a simple but effective method for incorporating word lexicons into the character representations for Chinese NER . • The proposed method is transferable to different sequence-labeling architectures and can be easily incorporated with pre-trained models like BERT ( Devlin et al . , 2018 ) . We performed experiments on four public Chinese NER datasets . The experimental results show that when implementing the sequence modeling layer with a single-layer Bi-LSTM , our method achieves considerable improvements over the state-of-theart methods in both inference speed and sequence labeling performance . In this work , we addressed the computational efficiency of utilizing word lexicons in Chinese NER . To obtain a high-performing Chinese NER system with a fast inference speed , we proposed a novel method to incorporate the lexicon information into the character representations . Experimental studies on four benchmark Chinese NER datasets reveal that our method can achieve a much faster inference speed and better performance than the compared state-of-the-art methods .

Named entity recognition in Chinese requires word segmentation causes errors or character-level model with lexical features that is complex and expensive.

They propose to encode lexicon features into character representations so it can keep the system simpler and achieve faster inference than previous models.

The proposed efficient character-based LSTM method with lexical features achieves 6.15 times faster inference speed and better performance than previous models.


Word embedding is central to neural machine translation ( NMT ) , which has attracted intensive research interest in recent years . In NMT , the source embedding plays the role of the entrance while the target embedding acts as the terminal . These layers occupy most of the model parameters for representation learning . Furthermore , they indirectly interface via a soft-attention mechanism , which makes them comparatively isolated . In this paper , we propose shared-private bilingual word embeddings , which give a closer relationship between the source and target embeddings , and which also reduce the number of model parameters . For similar source and target words , their embeddings tend to share a part of the features and they cooperatively learn these common representation units . Experiments on 5 language pairs belonging to 6 different language families and written in 5 different alphabets demonstrate that the proposed model provides a significant performance boost over the strong baselines with dramatically fewer model parameters . With the introduction of ever more powerful architectures , neural machine translation ( NMT ) has become the most promising machine translation method ( Kalchbrenner and Blunsom , 2013 ; Sutskever et al . , 2014 ; Bahdanau et al . , 2015 ) . For word representation , different architecturesincluding , but not limited to , recurrence-based ( Chen et al . , 2018 ) , convolution-based ( Gehring et al . , 2017 ) and transformation-based ( Vaswani et al . , 2017 ) NMT models-have been taking advantage of the distributed word embeddings to capture the syntactic and semantic properties of words ( Turian et al . , 2010 ) . Figure 1 : Comparison between ( a ) standard word embeddings and ( b ) shared-private word embeddings . In ( a ) , the English word " Long " and the German word " Lange " , which have similar lexical meanings , are represented by two private d-dimension vectors . While in ( b ) , the two word embeddings are made up of two parts , indicating the shared ( lined nodes ) and the private ( unlined nodes ) features . This enables the two words to make use of common representation units , leading to a closer relationship between them . NMT usually utilizes three matrices to represent source embeddings , target input embeddings , and target output embeddings ( also known as pre-softmax weight ) , respectively . These embeddings occupy most of the model parameters , which constrains the improvements of NMT because the recent methods become increasingly memory-hungry ( Vaswani et al . , 2017 ; Chen et al . , 2018 ) . 1 Even though converting words into subword units ( Sennrich et al . , 2016b ) , nearly 55 % of model parameters are used for word representation in the Transformer model ( Vaswani et al . , 2017 ) . To overcome this difficulty , several methods are proposed to reduce the parameters used for word representation of NMT . Press and Wolf ( 2017 ) propose two weight tying ( WT ) methods , called decoder WT and three-way WT , to substantially reduce the parameters of the word embeddings . Decoder WT ties the target input embedding and target output embedding , which has become the new de facto standard of practical NMT ( Sen- Figure 2 : Shared-private bilingual word embeddings perform between the source and target words or sub-words ( a ) with similar lexical meaning , ( b ) with same word form , and ( c ) without any relationship . Different sharing mechanisms are adapted into different relationship categories . This strikes the right balance between capturing monolingual and bilingual characteristics . The closeness of relationship decides the portion of features to be used for sharing . Words with similar lexical meaning tend to share more features , followed by the words with the same word form , and then the unrelated words , as illustrated by the lined nodes . nrich et al . , 2017 ) . Three-way WT uses only one matrix to represent the three word embeddings , where the source and target words that have the same word form tend to share a word vector . This method can also be adapted to sub-word NMT with a shared source-target sub-word vocabulary and it performs well in language pairs with many of the same characters , such as English-German and English-French ( Vaswani et al . , 2017 ) . Unfortunately , this method is not applicable to languages that are written in different alphabets , such as Chinese-English ( Hassan et al . , 2018 ) . Another challenge facing the source and target word embeddings of NMT is the lack of interactions . This degrades the attention performance , leading to some unaligned translations that hurt the translation quality . Hence , Kuang et al . ( 2018 ) propose to bridge the source and target embeddings , which brings better attention to the related source and target words . Their method is applicable to any language pairs , providing a tight interaction between the source and target word pairs . However , their method requires additional components and model parameters . In this work , we aim to enhance the word representations and the interactions between the source and target words , while using even fewer parameters . To this end , we present a languageindependent method , which is called sharedprivate bilingual word embeddings , to share a part of the embeddings of a pair of source and target words that have some common characteristics ( i.e. similar words should have similar vectors ) . Figure 1 illustrates the difference between the standard word embeddings and shared-private word embeddings of NMT . In the proposed method , each source ( or target ) word is represented by a word embedding that consists of the shared features and the private features . The shared features can also be regarded as the prior alignments connecting the source and target words . The private features allow the words to better learn the monolingual characteristics . Meanwhile , the features shared by the source and target embeddings result in a significant reduction of the number of parameters used for word representations . The experimental results on 6 translation datasets of different scales show that our model with fewer parameters yields consistent improvements over the strong Transformer baselines . In this work , we propose a novel sharing technique to improve the learning of word embeddings for NMT . Each word embedding is composed of shared and private features . The shared features act as a prior alignment guidance for the attention model to improve the quality of attention . Meanwhile , the private features enable the words to better capture the monolingual characteristics , result in an improvement of the overall translation quality . According to the degree of relevance between a parallel word pair , the word pairs are categorized into three different groups and the number of shared features is different . Our experimental results show that the proposed method outperforms the strong Transformer baselines while using fewer model parameters .

Word embeddings occupy a large amount of memory, and weight tying does not mitigate this issue for distant language pairs on translation tasks.

They propose a language independet method where a model shares embeddings between source and target only when words have some common characteristics.

Experiments on machine translation datasets involving multiple language families and scripts show that the proposed model outperforms baseline models while using fewer parameters.


This paper explores log-based query expansion ( QE ) models for Web search . Three lexicon models are proposed to bridge the lexical gap between Web documents and user queries . These models are trained on pairs of user queries and titles of clicked documents . Evaluations on a real world data set show that the lexicon models , integrated into a ranker-based QE system , not only significantly improve the document retrieval performance but also outperform two state-of-the-art log-based QE methods . Term mismatch is a fundamental problem in Web search , where queries and documents are composed using different vocabularies and language styles . Query expansion ( QE ) is an effective strategy to address the problem . It expands a query issued by a user with additional related terms , called expansion terms , so that more relevant documents can be retrieved . In this paper we explore the use of clickthrough data and translation models for QE . We select expansion terms for a query according to how likely it is that the expansion terms occur in the title of a document that is relevant to the query . Assuming that a query is parallel to the titles of documents clicked for that query ( Gao et al . 2010a ) , three lexicon models are trained on query-title pairs extracted from clickthrough data . The first is a word model that learns the translation probability between single words . The second model uses lexi-calized triplets to incorporate word dependencies for translation . The third is a bilingual topic model , which represents a query as a distribution of hidden topics and learns the translation between a query and a title term at the semantic level . We will show that the word model provides a rich set of expansion candidates while the triplet and topic models can effectively select good expansion terms , and that a ranker-based QE system which incorporates all three of these models not only significantly improves Web search result but outperforms other log-based QE methods that are stateof-the-art . There is growing interest in applying user logs to improve QE . A recent survey is due to Baeze-Yates and Ribeiro-Neto ( 2011 ) . Below , we briefly discuss two log-based QE methods that are closest to ours and are re-implemented in this study for comparison . Both systems use the same type of log data that we used to train the lexicon models . The term correlation model of Cui et al . ( 2002 ; 2003 ) is to our knowledge the first to explore querydocument relations for direct extraction of expansion terms for Web search . The method outperforms traditional QE methods that do not use log data e.g. the local analysis model of Xu and Croft ( 1996 ) . In addition , as pointed out by Cui et al . ( 2003 ) there are three important advantages that make log-based QE a promising technology to improve the performance of commercial search engines . First , unlike traditional QE methods that are based on relevance feedback , log-based QE derives expansion terms from search logs , allowing term correlations to be pre-computed offline . Compared to methods that are based on thesauri either compiled manually ( Prager et al . 2001 ) or derived au-tomatically from document collections ( Jing and Croft 1994 ) , the log-based method is superior in that it explicitly captures the correlation between query terms and document terms , and thus can bridge the lexical gap between them more effectively . Second , since search logs retrain querydocument pairs clicked by millions of users , the term correlations reflect the preference of the majority of users . Third , the term correlations evolve along with the accumulation of user logs , thus can reflect updated user interests at a specific time . However , as pointed out by Riezler et al . ( 2008 ) , Cui et al . 's correlation-based method suffers low precision of QE partly because the correlation model does not explicitly capture context information and is susceptible to noise . Riezler et al . developed a QE system by retraining a standard phrase-based statistical machine translation ( SMT ) system using query-snippet pairs extracted from clickthrough data ( Riezler et al . 2008 ; Riezler and Liu 2010 ) . The SMT-based system can produce cleaner , more relevant expansion terms because rich context information useful for filtering noisy expansions is captured by combining language model and phrase translation model in its decoder . Furthermore , in the SMT system all component models are properly smoothed using sophisticated techniques to avoid sparse data problems while the correlation model relies on pure counts of term frequencies . However , the SMT system is used as a black box in their experiments . So the relative contribution of different SMT components is not verified empirically . In this study we break this black box in order to build a better , simpler QE system . We will show that the proposed lexicon models outperform significantly the term correlation model , and that a simpler QE system that incorporates the lexicon models can beat the sophisticated , black-box SMT system . In this paper we extend the previous log-based QE methods in two directions . First , we formulate QE as the problem of translating a source language of queries into a target language of documents , represented as titles . This allows us to adapt the established techniques developed for SMT to QE . Specially , we propose three lexicon models based on terms , lexicalized triplets , and topics , respectively . These models are trained on pairs of user queries and the titles of clicked documents using EM . Second , we present a ranker-based QE system , the heart of which is a MRF-based ranker in which the lexicon models are incorporated as features . We perform experiments on the Web search task using a real world data set . Results show that the proposed system outperforms significantly other stateof-the-art QE systems . This study is part of a bigger , ongoing project , aiming to develop a real-time QE system for Web search , where simplicity is the key to the success . Thus , what we learned from this study is particularly encouraging . We demonstrate that with large amounts of clickthrough data for model training , simple lexicon models can achieve state-of-the-art QE performance , and that the MRF-based ranker provides a simple and flexible framework to incorporate a variety of features capturing different types of term dependencies in such an effective way that the Web search performance can be directly optimized .

Term mismatches between a query and documents hinder retrievals of relevant documents and black box statistical machine translation models are used to expand queries.

They propose to train lexicon query expansion models by using transaction logs that contain pairs of queries and titles of clicked documents.

The proposed query expansion model enables retrieval systems to significantly outperform models with previous expansion models while being more transparent.


Traditional noun phrase coreference resolution systems represent features only of pairs of noun phrases . In this paper , we propose a machine learning method that enables features over sets of noun phrases , resulting in a first-order probabilistic model for coreference . We outline a set of approximations that make this approach practical , and apply our method to the ACE coreference dataset , achieving a 45 % error reduction over a comparable method that only considers features of pairs of noun phrases . This result demonstrates an example of how a firstorder logic representation can be incorporated into a probabilistic model and scaled efficiently . Noun phrase coreference resolution is the problem of clustering noun phrases into anaphoric sets . A standard machine learning approach is to perform a set of independent binary classifications of the form " Is mention a coreferent with mention b ? " This approach of decomposing the problem into pairwise decisions presents at least two related difficulties . First , it is not clear how best to convert the set of pairwise classifications into a disjoint clustering of noun phrases . The problem stems from the transitivity constraints of coreference : If a and b are coreferent , and b and c are coreferent , then a and c must be coreferent . This problem has recently been addressed by a number of researchers . A simple approach is to perform the transitive closure of the pairwise decisions . However , as shown in recent work ( McCallum and Wellner , 2003 ; Singla and Domingos , 2005 ) , better performance can be obtained by performing relational inference to directly consider the dependence among a set of predictions . For example , McCallum and Wellner ( 2005 ) apply a graph partitioning algorithm on a weighted , undirected graph in which vertices are noun phrases and edges are weighted by the pairwise score between noun phrases . A second and less studied difficulty is that the pairwise decomposition restricts the feature set to evidence about pairs of noun phrases only . This restriction can be detrimental if there exist features of sets of noun phrases that can not be captured by a combination of pairwise features . As a simple example , consider prohibiting coreferent sets that consist only of pronouns . That is , we would like to require that there be at least one antecedent for a set of pronouns . The pairwise decomposition does not make it possible to capture this constraint . In general , we would like to construct arbitrary features over a cluster of noun phrases using the full expressivity of first-order logic . Enabling this sort of flexible representation within a statistical model has been the subject of a long line of research on first-order probabilistic models ( Gaifman , 1964 ; Halpern , 1990 ; Paskin , 2002 ; Poole , 2003 ; Richardson and Domingos , 2006 ) . Conceptually , a first-order probabilistic model can be described quite compactly . A configuration of the world is represented by a set of predi- cates , each of which has an associated real-valued parameter . The likelihood of each configuration of the world is proportional to a combination of these weighted predicates . In practice , however , enumerating all possible configurations , or even all the predicates of one configuration , can result in intractable combinatorial growth ( de Salvo Braz et al . , 2005 ; Culotta and McCallum , 2006 ) . In this paper , we present a practical method to perform training and inference in first-order models of coreference . We empirically validate our approach on the ACE coreference dataset , showing that the first-order features can lead to an 45 % error reduction . We have presented learning and inference procedures for coreference models using first-order features . By relying on sampling methods at training time and approximate inference methods at testing time , this approach can be made scalable . This results in a coreference model that can capture features over sets of noun phrases , rather than simply pairs of noun phrases . This is an example of a model with extremely flexible representational power , but for which exact inference is intractable . The simple approximations we have described here have enabled this more flexible model to outperform a model that is simplified for tractability . A short-term extension would be to consider features over entire clusterings , such as the number of clusters . This could be incorporated in a ranking scheme , as in Ng ( 2005 ) . Future work will extend our approach to a wider variety of tasks . The model we have described here is specific to clustering tasks ; however a similar formulation could be used to approach a number of language processing tasks , such as parsing and relation extraction . These tasks could benefit from first-order features , and the present work can guide the approximations required in those domains . Additionally , we are investigating more sophisticated inference algorithms that will reduce the greediness of the search procedures described here .

Existing approaches treat noun phrase coreference resolution as a set of independent binary classifications limiting the features to be only pairs of noun phrases.

They propose a machine learning method that uses sets of noun phrases as features that are coupled with a sampling method to enable scalability.

Evaluation on the ACE coreference dataset, the proposed method achieves a 45% error reduction over a previous method.


Bilingual lexicons map words in one language to their translations in another , and are typically induced by learning linear projections to align monolingual word embedding spaces . In this paper , we show it is possible to produce much higher quality lexicons with methods that combine ( 1 ) unsupervised bitext mining and ( 2 ) unsupervised word alignment . Directly applying a pipeline that uses recent algorithms for both subproblems significantly improves induced lexicon quality and further gains are possible by learning to filter the resulting lexical entries , with both unsupervised and semisupervised schemes . Our final model outperforms the state of the art on the BUCC 2020 shared task by 14 F 1 points averaged over 12 language pairs , while also providing a more interpretable approach that allows for rich reasoning of word meaning in context . Further analysis of our output and the standard reference lexicons suggests they are of comparable quality , and new benchmarks may be needed to measure further progress on this task . 1 Bilingual lexicons map words in one language to their translations in another , and can be automatically induced by learning linear projections to align monolingual word embedding spaces ( Artetxe et al . , 2016 ; Smith et al . , 2017 ; Lample et al . , 2018 , inter alia ) . Although very successful in practice , the linear nature of these methods encodes unrealistic simplifying assumptions ( e.g. all translations of a word have similar embeddings ) . In this paper , we show it is possible to produce much higher quality lexicons without these restrictions by introducing new methods that combine ( 1 ) unsupervised bitext mining and ( 2 ) unsupervised word alignment . We show that simply pipelining recent algorithms for unsupervised bitext mining ( Tran et al . , 2020 ) and unsupervised word alignment ( Sabet et al . , 2020 ) significantly improves bilingual lexicon induction ( BLI ) quality , and that further gains are possible by learning to filter the resulting lexical entries . Improving on a recent method for doing BLI via unsupervised machine translation ( Artetxe et al . , 2019 ) , we show that unsupervised mining produces better bitext for lexicon induction than translation , especially for less frequent words . These core contributions are established by systematic experiments in the class of bitext construction and alignment methods ( Figure 1 ) . Our full induction algorithm filters the lexicon found via the initial unsupervised pipeline . The filtering can be either fully unsupervised or weakly-supervised : for the former , we filter using simple heuristics and global statistics ; for the latter , we train a multi-layer perceptron ( MLP ) to predict the probability of a word pair being in the lexicon , where the features are global statistics of word alignments . In addition to BLI , our method can also be directly adapted to improve word alignment and reach competitive or better alignment accuracy than the state of the art on all investigated language pairs . We find that improved alignment in sentence representations ( Tran et al . , 2020 ) leads to better contextual word alignments using local similarity ( Sabet et al . , 2020 ) . Our final BLI approach outperforms the previous state of the art on the BUCC 2020 shared task ( Rapp et al . , 2020 ) by 14 F 1 points averaged over 12 language pairs . Manual analysis shows that most of our false positives are due to the incompleteness of the reference and that our lexicon is comparable to the reference lexicon and the output of a supervised system . Because both of our key building blocks make use of the pretrainined contextual representations from mBART ( Liu et al . , Word Alignment Statistical Feature Extraction cooccurrence(good , guten ) = 2 one-to-one align(good , guten ) = 2 many-to-one align(good , guten ) = 0 cosine_similarity(good , guten ) = 0.8 inner_product(good , guten ) = 1.8 count(good ) = 2 count(guten ) = 2 We present a direct and effective framework for BLI with unsupervised bitext mining and word alignment , which sets a new state of the art on the task . From the perspective of pretrained multilingual models ( Conneau et al . , 2019 ; Liu et al . , 2020 ; Tran et al . , 2020 , inter alia ) , our work shows that they have successfully captured information about word translation that can be extracted using similarity based alignment and refinement . Although BLI is only about word types , it strongly benefits from contextualized reasoning at the token level . umich.edu/ ˜mihalcea / wpt ( en-fr and ro-en ) ; https : //web.eecs.umich.edu/ ˜mihalcea / wpt05 ( enhi )

Existing methods to induce bilingual lexicons use linear projections to align word embeddings that are based on unrealistic simplifying assumptions.

They propose to use both unsupervised bitext mining and unsupervised word alignment methods to produce higher quality lexicons.

The proposed method achieves the state-of-the-art in the bilingual lexical induction task while keeping the interpretability of their pipeline.


In this paper we show that a simple beam approximation of the joint distribution between attention and output is an easy , accurate , and efficient attention mechanism for sequence to sequence learning . The method combines the advantage of sharp focus in hard attention and the implementation ease of soft attention . On five translation and two morphological inflection tasks we show effortless and consistent gains in BLEU compared to existing attention mechanisms . In structured input-output models as used in tasks like translation and image captioning , the attention variable decides which part of the input aligns to the current output . Many attention mechanisms have been proposed ( Xu et al . , 2015 ; Bahdanau et al . , 2014 ; Luong et al . , 2015 ; Martins and Astudillo , 2016 ) but the de facto standard is a soft attention mechanism that first assigns attention weights to input encoder states , then computes an attention weighted ' soft ' aligned input state , which finally derives the output distribution . This method is end to end differentiable and easy to implement . Another less popular variant is hard attention that aligns each output to exactly one input state but requires intricate training to teach the network to choose that state . When successfully trained , hard attention is often found to be more accurate ( Xu et al . , 2015 ; Zaremba and Sutskever , 2015 ) . In NLP , a recent success has been in a monotonic hard attention setting in morphological inflection tasks ( Yu et al . , 2016 ; Aharoni and Goldberg , 2017 ) . For general seq2seq learning , methods like Sparse-Max ( Martins and Astudillo , 2016 ) and local attention ( Luong et al . , 2015 ) were proposed to bridge the gap between soft and hard attention . * Both authors contributed equally to this work In this paper we propose a surprisingly simpler alternative based on the original joint distribution between output and attention , of which existing soft and hard attention mechanisms are approximations . The joint model couples input states individually to the output like in hard attention , but it combines the advantage of end-to-end trainability of soft attention . When the number of input states is large , we propose to use a simple approximation of the full joint distribution called Beam-joint . This approximation is also easily trainable and does not suffer from the high variance of Monte-Carlo sampling gradients of hard attention . We evaluated our model on five translation tasks and increased BLEU by 0.8 to 1.7 over soft attention , which in turn was better than hard and the recent Sparsemax ( Martins and Astudillo , 2016 ) attention . More importantly , the training process was as easy as soft attention . For further support , we also evaluate on two morphological inflection tasks and got gains over soft and hard attention . In this paper we showed a simple yet effective approximation of the joint attention-output distribution in sequence to sequence learning . Our joint model consistently provides higher accuracy without significant running time overheads in five translation and two morphological inflection tasks . An interesting direction for future work is to extend beam-joint to multi-head attention architectures as in ( Vaswani et al . , 2017 ; Xu Chen , 2018 ) .

Softmax attention models are popular because of their differentiable and easy to implement nature while hard attention models outperform them when successfully trained.

They propose a method to approximate the joint attention-output distribution which provides sharp attention as hard attention and easy implementation as soft attention.

The proposed approach outperforms soft attention models and recent hard attention and Sparsemax models on five translation tasks and also on morphological inflection tasks.


Contrastive learning has achieved impressive success in generation tasks to militate the " exposure bias " problem and discriminatively exploit the different quality of references . Existing works mostly focus on contrastive learning on the instance-level without discriminating the contribution of each word , while keywords are the gist of the text and dominant the constrained mapping relationships . Hence , in this work , we propose a hierarchical contrastive learning mechanism , which can unify hybrid granularities semantic meaning in the input text . Concretely , we first propose a keyword graph via contrastive correlations of positive-negative pairs to iteratively polish the keyword representations . Then , we construct intra-contrasts within instance-level and keyword-level , where we assume words are sampled nodes from a sentence distribution . Finally , to bridge the gap between independent contrast levels and tackle the common contrast vanishing problem , we propose an inter-contrast mechanism that measures the discrepancy between contrastive keyword nodes respectively to the instance distribution . Experiments demonstrate that our model outperforms competitive baselines on paraphrasing , dialogue generation , and storytelling tasks . Generation tasks such as storytelling , paraphrasing , and dialogue generation aim at learning a certain correlation between text pairs that maps an arbitrary-length input to another arbitrary-length output . Traditional methods are mostly trained with " teacher forcing " and lead to an " exposure bias " problem ( Schmidt , 2019 ) . Incorporating the generation method with contrastive learning achieved impressive performance on tackling such issues , which takes an extra consideration of synthetic negative samples contrastively ( Lee et al . , 2021 Existing contrastive mechanisms are mainly focused on the instance level ( Lee et al . , 2021 ; Cai et al . , 2020 ) . However , word-level information is also of great importance . Take the case shown in the upper part of Figure 1 for example , the keyword covers the gist of the input text and determines the embedding space of the text . The text representation will be significantly affected if adding a slight perturbation on the keyword , i.e. , changing " cosmology " to " astrophysics " . In addition , as shown on the bottom part , under some circumstances , it is too easy for the model to do the classification since the semantic gap between contrastive pairs is huge . Thus , the model fails to distinguish the actual discrepancy , which causes a " contrast vanishing " problem at both instance-level and keyword-level . Based on the above motivation , in this paper , we propose a hierarchical contrastive learning method built on top of the classic CVAE structure . We choose CVAE due to its ability in modeling global properties such as syntactic , semantic , and discourse coherence ( Li et al . , 2015 ; Yu et al . , 2020 ) . We first learn different granularity representations through two independent contrast , i.e. , instancelevel and keyword-level . Specifically , we use the universal and classic TextRank ( Mihalcea and Tarau , 2004 ) method to extract keywords from each text , which contain the most important information and need to be highlighted . On the instancelevel , we treat the keyword in the input text as an additional condition for a better prior semantic distribution . Then , we utilize Kullback-Leibler divergence ( Kullback and Leibler , 1951 ) to reduce the distance between prior distribution and positive posterior distribution , and increase the distance with the negative posterior distribution . While on the keyword-level , we propose a keyword graph via contrastive correlations of positive-negative pairs to learn informative and accurate keyword representations . By treating the keyword in the output text as an anchor , the imposter keyword is produced by neighboring nodes of the anchor keyword and forms the keyword-level contrast , where the similarity between the imposter keyword and the anchor keyword is poorer than the positive keyword . To unify individual intra-contrasts and tackle the " contrast vanishing " problem in independent contrastive granularities , we leverage an inter-contrast , the Mahalanobis contrast , to investigate the contrastive enhancement based on the Mahalanobis distance ( De Maesschalck et al . , 2000 ) , a measure of the distance between a point and a distribution , between the instance distribution and the keyword representation . Concretely , we ensure the distance from the anchor instance distribution to the groundtruth keyword vector is closer than to the imposter keyword vector . The Mahalanobis contrast plays an intermediate role that joins the different granularities contrast via incorporating the distribution of instance with the representation of its crucial part , and makes up a more comprehensive keyworddriven hierarchical contrastive mechanism , so as to ameliorate the generated results . We empirically show that our model outperforms CVAE and other baselines significantly on three generation tasks : paraphrasing , dialogue genera-tion , and storytelling . Our contributions can be summarized as follows : • To our best knowledge , we are the first to propose an inter-level contrastive learning method , which unifies instance-level and keyword-level contrasts in the CVAE framework . • We propose three contrastive learning measurements : KL divergence for semantic distribution , cosine distance for points , and Mahalanobis distance for points with distribution . • We introduce a global keyword graph to obtain polished keyword representations and construct imposter keywords for contrastive learning . In this paper , we propose a hierarchical contrastive learning mechanism , which consists of intra-contrasts within instance-level and keywordlevel and inter-contrast with Mahalanobis contrast . The experimental results yield significant out-performance over baselines when applied in the CVAE framework . In the future , we aim to extend the contrastive learning mechanism to different basic models , and will explore contrastive learning methods based on external knowledge .

Existing works on contrastive learning for text generation focus only on instance-level while word-level information such as keywords is also of great importance.

They propose a CVAE-based hierarchical contrastive learning within instance and keyword-level using a keyword graph which iteratively polishes the keyword representations.

The proposed model outperforms CVAE and baselines on storytelling, paraphrasing, and dialogue generation tasks.


Word embedding models are typically able to capture the semantics of words via the distributional hypothesis , but fail to capture the numerical properties of numbers that appear in a text . This leads to problems with numerical reasoning involving tasks such as question answering . We propose a new methodology to assign and learn embeddings for numbers . Our approach creates Deterministic , Independentof-Corpus Embeddings ( referred to as DICE ) for numbers , such that their cosine similarity reflects the actual distance on the number line . DICE outperforms a wide range of pre-trained word embedding models across multiple examples of two tasks : ( i ) evaluating the ability to capture numeration and magnitude ; and ( ii ) to perform list maximum , decoding , and addition . We further explore the utility of these embeddings in downstream applications by initializing numbers with our approach for the task of magnitude prediction . We also introduce a regularization approach to learn model-based embeddings of numbers in a contextual setting . Word embeddings capture semantic relationships between words by operationalizing the distributional hypothesis ( Harris , 1954 ; Firth , 1957 ) . They can be learned either non-contextually ( Mikolov et al . , 2013b ; Pennington et al . , 2014 ; Bojanowski et al . , 2017 ) or contextually ( Devlin et al . , 2018 ; Peters et al . , 2018 ) . Non-contextual embeddings have worked well on various language understanding and semantic tasks ( Rumelhart et al . , 1988 ; Mikolov et al . , 2013a , b ) . More recently , they have also been used as pre-trained word embeddings to aid more sophisticated contextual models for solving rigorous natural language processing ( NLP ) problems , including translation , paraphrasing , and sentence-similarity tasks ( Kiros et al . , 2015 ; Wieting et al . , 2015 ) . While word embeddings effectively capture semantic relationships between words , they are less effective at capturing numeric properties associated with numbers . Though numbers represent a significant percentage of tokens in a corpus , they are often overlooked . In non-contextual word embedding models , they are treated like any other word , which leads to misinterpretation . For instance , they exhibit unintuitive similarities with other words and do not contain strong prior information about the magnitude of the number they encode . In sentence similarity and reasoning tasks , failure to handle numbers causes as much as 29 % of contradictions ( De Marneffe et al . , 2008 ) . In other data-intensive tasks where numbers are abundant , like neural machine translation , they are masked to hide the translation models inefficiency in dealing with them ( Mitchell and Lapata , 2009 ) . There are a variety of tests proposed to measure the efficiency of number embeddings . For instance , Naik et al . ( 2019 ) shows that GloVe ( Pennington et al . , 2014 ) , word2vec ( Mikolov et al . , 2013b ) , and fastText ( Joulin et al . , 2016 ; Bojanowski et al . , 2017 ) fail to capture numeration and magnitude properties of a number . Numeration is the property of associating numbers with their corresponding word representations ( " 3 " and " three " ) while magnitude represents a number 's actual value ( 3 < 4 ) . Further , Wallace et al . ( 2019 ) proposes several tests for analyzing numerical reasoning of number embeddings that include list maximum , decoding , and addition . In this paper , we experimentally demonstrate that if the cosine similarity between word embeddings of two numbers reflects their actual distance on the number line , the resultant word embeddings are useful in downstream tasks . We first demonstrate how Deterministic , Independent-of-Corpus Embeddings ( DICE ) can be constructed such that they almost perfectly capture properties of numera-tion and magnitude . These non-contextual embeddings also perform well on related tests for numeracy ( Wallace et al . , 2019 ) . To demonstrate the efficacy of DICE for downstream tasks , we explore its utility in two experiments . First , we design a DICE embedding initialized Bi-LSTM network to classify the magnitude of masked numbers in the 600 K dataset ( Chen et al . , 2019 ) . Second , given the popularity of modern contextual model-based embeddings , we devise a regularization procedure that emulates the hypothesis proposed by DICE and can be employed in any task-based fine-tuning process . We demonstrate that adding such regularization helps the model internalize notions of numeracy while learning task-based contextual embeddings for the numbers present in the text . We find promising results in a numerical reasoning task that involves numerical question answering based on a sub-split of the popular SQuAD dataset ( Rajpurkar et al . , 2016 ) . Our contribution can be summarized as follows : • We propose a deterministic technique to learn numerical embeddings . DICE embeddings are learned independently of corpus and effectively capture properties of numeracy . • We prove experimentally that the resultant embeddings learned using the above methods improve a model 's ability to reason about numbers in a variety of tasks , including numeration , magnitude , list maximum , decoding , and addition . • We also demonstrate that properties of DICE can be adapted to contextual models , like BERT ( Devlin et al . , 2018 ) , through a novel regularization technique for solving tasks involving numerical reasoning . In this work , we methodologically assign and learn embeddings for numbers to reflect their numerical properties . We validate our proposed approach with several experiments that test number embeddings . The tests that evaluate the numeral embeddings are fundamentally applicable to all real numbers . Finally , we introduced an approach to jointly learn embeddings of numbers and words that preserve numerical properties and evaluated them on a contextual word embedding based model . In our future work , we would like to extend this idea to unseen numbers in vocabulary as a function of seen ones .

Existing word embeddings treat numbers like words failing to capture numeration and magnitude properties of numbers which is problematic for tasks such as question answering.

They propose a deterministic technique to learn numerical embeddings where cosine similarity reflects the actual distance and a regularization approach for a contextual setting.

A Bi-LSTM network initialized with the proposed embedding shows the ability to capture numeration and magnitude and to perform list maximum, decoding, and addition.


We propose a novel approach to improve SMT via paraphrase rules which are automatically extracted from the bilingual training data . Without using extra paraphrase resources , we acquire the rules by comparing the source side of the parallel corpus with the target-to-source translations of the target side . Besides the word and phrase paraphrases , the acquired paraphrase rules mainly cover the structured paraphrases on the sentence level . These rules are employed to enrich the SMT inputs for translation quality improvement . The experimental results show that our proposed approach achieves significant improvements of 1.6~3.6 points of BLEU in the oral domain and 0.5~1 points in the news domain . This work was done when the first author was visiting Baidu . The translation quality of the SMT system is highly related to the coverage of translation models . However , no matter how much data is used for training , it is still impossible to completely cover the unlimited input sentences . This problem is more serious for online SMT systems in real-world applications . Naturally , a solution to the coverage problem is to bridge the gaps between the input sentences and the translation models , either from the input side , which targets on rewriting the input sentences to the MT-favored expressions , or from the side of translation models , which tries to enrich the translation models to cover more expressions . In recent years , paraphrasing has been proven useful for improving SMT quality . The proposed methods can be classified into two categories according to the paraphrase targets : ( 1 ) enrich translation models to cover more bilingual expressions ; ( 2 ) paraphrase the input sentences to reduce OOVs or generate multiple inputs . In the first category , He et al . ( 2011 ) , Bond et al . ( 2008 ) and Nakov ( 2008 ) enriched the SMT models via paraphrasing the training corpora . Kuhn et al . ( 2010 ) and Max ( 2010 ) used paraphrases to smooth translation models . For the second category , previous studies mainly focus on finding translations for unknown terms using phrasal paraphrases . Callison-Burch et al . ( 2006 ) and Marton et al . ( 2009 ) paraphrase unknown terms in the input sentences using phrasal paraphrases extracted from bilingual and monolingual corpora . Mirkin et al . ( 2009 ) rewrite OOVs with entailments and paraphrases acquired from WordNet . Onishi et al . ( 2010 ) and Du et al . ( 2010 ) use phrasal paraphrases to build a word lattice to get multiple input candidates . In the above methods , only word or phrasal paraphrases are used for input sentence rewriting . No structured paraphrases on the sentence level have been investigated . However , the information in the sentence level is very important for disambiguation . For example , we can only substitute play with drama in a context related to stage or theatre . Phrasal paraphrase substitutions can hardly solve such kind of problems . In this paper , we propose a method that rewrites the input sentences of the SMT system using automatically extracted paraphrase rules which can capture structures on sentence level in addition to paraphrases on the word or phrase level . Without extra paraphrase resources , a novel approach is proposed to acquire paraphrase rules from the bilingual training corpus based on the results of Forward-Translation and Back-Translation . The rules target on rewriting the input sentences to an MT-favored expression to ensure a better translation . The paraphrase rules cover all kinds of paraphrases on the word , phrase and sentence levels , enabling structure reordering , word or phrase insertion , deletion and substitution . The experimental results show that our proposed approach achieves significant improvements of 1.6~3.6 points of BLEU in the oral domain and 0.5~1 points in the news domain . The remainder of the paper is organized as follows : Section 2 makes a comparison between the Forward-Translation and Back-Translation . Section 3 introduces our methods that extract paraphrase rules from the bilingual corpus of SMT . Section 4 describes the strategies for constructing word lattice with paraphrase rules . The experimental results and some discussions are presented in Section 5 and Section 6 . Section 7 compares our work to the previous researches . Finally , Section 8 concludes the paper and suggests directions for future work . In this paper , we propose a novel method for extracting paraphrase rules by comparing the source side of bilingual corpus to the target-tosource translation of the target side . The acquired paraphrase rules are employed to enrich the SMT inputs , which target on rewriting the input sentences to an MT-favored form . The paraphrase rules cover all kinds of paraphrases on the word , phrase and sentence levels , enabling structure reordering , word or phrase insertion , deletion and substitution . Experimental results show that the paraphrase rules can improve SMT quality in both the oral and news domains . The manual investigation on oral translation results indicate that the paraphrase rules capture four kinds of MTfavored transformation to ensure translation quality improvement .

Incorporating paraphrases improves statistical machine translation however no works investigate sentence level paraphrases.

They propose to use bilingual training data to obtain paraphrase rules on word, phrase and sentence levels to rewrite inputs to be MT-favored.

The acquired paraphrase rules improve translation qualities in oral and news domains.


Automatically extracting social meaning and intention from spoken dialogue is an important task for dialogue systems and social computing . We describe a system for detecting elements of interactional style : whether a speaker is awkward , friendly , or flirtatious . We create and use a new spoken corpus of 991 4-minute speed-dates . Participants rated their interlocutors for these elements of style . Using rich dialogue , lexical , and prosodic features , we are able to detect flirtatious , awkward , and friendly styles in noisy natural conversational data with up to 75 % accuracy , compared to a 50 % baseline . We describe simple ways to extract relatively rich dialogue features , and analyze which features performed similarly for men and women and which were gender-specific . How can we extract social meaning from speech , deciding if a speaker is particularly engaged in the conversation , is uncomfortable or awkward , or is particularly friendly and flirtatious ? Understanding these meanings and how they are signaled in language is an important sociolinguistic task in itself . Extracting them automatically from dialogue speech and text is crucial for developing socially aware computing systems for tasks such as detection of interactional problems or matching conversational style , and will play an important role in creating more natural dialogue agents ( Pentland , 2005 ; Nass and Brave , 2005 ; Brave et al . , 2005 ) . Cues for social meaning permeate speech at every level of linguistic structure . Acoustic cues such as low and high F0 or energy and spectral tilt are important in detecting emotions such as annoyance , anger , sadness , or boredom ( Ang et al . , 2002 ; Lee and Narayanan , 2002 ; Liscombe et al . , 2003 ) , speaker characteristics such as charisma ( Rosenberg and Hirschberg , 2005 ) , or personality features like extroversion ( Mairesse et al . , 2007 ; Mairesse and Walker , 2008 ) . Lexical cues to social meaning abound . Speakers with links to depression or speakers who are under stress use more first person singular pronouns ( Rude et al . , 2004 ; Pennebaker and Lay , 2002 ; Cohn et al . , 2004 ) , positive emotion words are cues to agreeableness ( Mairesse et al . , 2007 ) , and negative emotion words are useful cues to deceptive speech ( Newman et al . , 2003 ) . The number of words in a sentence can be a useful feature for extroverted personality ( Mairesse et al . , 2007 ) . Finally , dialog features such as the presence of disfluencies can inform listeners about speakers ' problems in utterance planning or about confidence ( Brennan and Williams , 1995 ; Brennan and Schober , 2001 ) . Our goal is to see whether cues of this sort are useful in detecting particular elements of conversational style and social intention ; whether a speaker in a speed-dating conversation is judged by the interlocutor as friendly , awkward , or flirtatious . The results presented here should be regarded with some caution . The sample is not a random sample of English speakers or American adults , and speed dating is not a natural context for expressing every conversational style . Therefore , a wider array of studies across populations and genres would be required before a more general theory of conversational styles is established . On the other hand , the presented results may under-reflect the relations being captured . The quality of recordings and coarse granularity ( 1 second ) of the time-stamps likely cloud the relations , and as the data is cleaned and improved , we expect the associations to only grow stronger . Caveats aside , we believe the evidence indicates that the perception of several types of conversational style have relatively clear signals across genders , but with some additional gender contextualization . Both genders convey flirtation by laughing more , speaking faster , and using higher and more variable pitch . Both genders convey friendliness by laughing more , and using collaborative completions . However , we do find gender differences ; men asl more questions when ( labeled as ) flirting , women ask fewer . Men labeled as flirting are softer , but women labeled as flirting are louder . Women flirt-ing swear more , while men are more likely to use sexual vocabulary . Gender differences exist as well for the other variables . Men labeled as friendly use you while women labeled as friendly use I. Friendly women are very disfluent ; friendly men are not . While the features for friendly and flirtatious speech overlap , there are clear differences . Men speaker faster and with higher f0 ( min ) in flirtatious speech , but not faster and with lower f0 ( min ) in friendly speech . For men , flirtatious speech involves more questions and repair questions , while friendly speech does not . For women , friendly speech is more disfluent than flirtatious speech , and has more collaborative style ( completions , repair questions , appreciations ) . We also seem to see a model of collaborative conversational style ( probably related to the collaborative floor of Edelsky ( 1981 ) and Coates ( 1996 ) ) , cued by the use of more collaborative completions , repair questions and other questions , you , and laughter . These collaborative techniques were used by both women and men who were labeled as friendly , and occurred less with men labeled as awkward . Women themselves displayed more of this collaborative conversational style when they labeled the men as friendly . For women only , collaborative style included appreciations ; while for men only , collaborative style included overlaps . In addition to these implications for social science , our work has implications for the extraction of meaning in general . A key focus of our work was on ways to extract useful dialog act and disfluency features ( repair questions , backchannels , appreciations , restarts , dispreferreds ) with very shallow methods . These features were indeed extractable and proved to be useful features in classification . We are currently extending these results to predict date outcomes including ' liking ' , extending work such as Madan and Pentland ( 2006 ) .

Methods to extract social meanings such as engagement from speech remain unknown while it is important in sociolinguistics and to develop socially aware computing systems.

They create a spoken corpus from conversations in speed-dating and perform analysis using extracted dialogue features with a focus on genders.

They found several gender dependent and independent phenomena in conversations related to the speed of speaking, laughing or asking questions.


We introduce the task of predicting adverbial presupposition triggers such as also and again . Solving such a task requires detecting recurring or similar events in the discourse context , and has applications in natural language generation tasks such as summarization and dialogue systems . We create two new datasets for the task , derived from the Penn Treebank and the Annotated English Gigaword corpora , as well as a novel attention mechanism tailored to this task . Our attention mechanism augments a baseline recurrent neural network without the need for additional trainable parameters , minimizing the added computational cost of our mechanism . We demonstrate that our model statistically outperforms a number of baselines , including an LSTM-based language model . In pragmatics , presuppositions are assumptions or beliefs in the common ground between discourse participants when an utterance is made ( Frege , 1892 ; Strawson , 1950 ; Stalnaker , 1973 Stalnaker , , 1998 ) ) , and are ubiquitous in naturally occurring discourses ( Beaver and Geurts , 2014 ) . Presuppositions underly spoken statements and written sentences and understanding them facilitates smooth communication . We refer to expressions that indicate the presence of presuppositions as presupposition triggers . These include definite descriptions , factive verbs and certain adverbs , among others . For example , consider the following statements : ( 1 ) John is going to the restaurant again . * Authors ( listed in alphabetical order ) contributed equally . ( 2 ) John has been to the restaurant . ( 1 ) is only appropriate in the context where ( 2 ) is held to be true because of the presence of the presupposition trigger again . One distinguishing characteristic of presupposition is that it is unaffected by negation of the presupposing context , unlike other semantic phenomena such as entailment and implicature . The negation of ( 1 ) , John is not going to the restaurant again . , also presupposes ( 2 ) . Our focus in this paper is on adverbial presupposition triggers such as again , also and still . Adverbial presupposition triggers indicate the recurrence , continuation , or termination of an event in the discourse context , or the presence of a similar event . In one study of presuppositional triggers in English journalistic texts ( Khaleel , 2010 ) , adverbial triggers were found to be the most commonly occurring presupposition triggers after existential triggers . 1 Despite their frequency , there has been little work on these triggers in the computational literature from a statistical , corpus-driven perspective . As a first step towards language technology systems capable of understanding and using presuppositions , we propose to investigate the detection of contexts in which these triggers can be used . This task constitutes an interesting testing ground for pragmatic reasoning , because the cues that are indicative of contexts containing recurring or similar events are complex and often span more than one sentence , as illustrated in Sentences ( 1 ) and ( 2 ) . Moreover , such a task has immediate practical consequences . For example , in language generation applications such as summarization and dialogue systems , adding presuppositional triggers in contextually appropriate loca-1 Presupposition of existence are triggered by possessive constructions , names or definite noun phrases . tions can improve the readability and coherence of the generated output . We create two datasets based on the Penn Treebank corpus ( Marcus et al . , 1993 ) and the English Gigaword corpus ( Graff et al . , 2007 ) , extracting contexts that include presupposition triggers as well as other similar contexts that do not , in order to form a binary classification task . In creating our datasets , we consider a set of five target adverbs : too , again , also , still , and yet . We focus on these adverbs in our investigation because these triggers are well known in the existing linguistic literature and commonly triggering presuppositions . We control for a number of potential confounding factors , such as class balance , and the syntactic governor of the triggering adverb , so that models can not exploit these correlating factors without any actual understanding of the presuppositional properties of the context . We test a number of standard baseline classifiers on these datasets , including a logistic regression model and deep learning methods based on recurrent neural networks ( RNN ) and convolutional neural networks ( CNN ) . In addition , we investigate the potential of attention-based deep learning models for detecting adverbial triggers . Attention is a promising approach to this task because it allows a model to weigh information from multiple points in the previous context and infer long-range dependencies in the data ( Bahdanau et al . , 2015 ) . For example , the model could learn to detect multiple instances involving John and restaurants , which would be a good indication that again is appropriate in that context . Also , an attention-based RNN has achieved success in predicting article definiteness , which involves another class of presupposition triggers ( Kabbara et al . , 2016 ) . As another contribution , we introduce a new weighted pooling attention mechanism designed for predicting adverbial presupposition triggers . Our attention mechanism allows for a weighted averaging of our RNN hidden states where the weights are informed by the inputs , as opposed to a simple unweighted averaging . Our model uses a form of self-attention ( Paulus et al . , 2018 ; Vaswani et al . , 2017 ) , where the input sequence acts as both the attention mechanism 's query and key / value . Unlike other attention models , instead of simply averaging the scores to be weighted , our approach aggregates ( learned ) attention scores by learning a reweighting scheme of those scores through another level ( dimension ) of attention . Additionally , our mechanism does not introduce any new parameters when compared to our LSTM baseline , reducing its computational impact . We compare our model using the novel attention mechanism against the baseline classifiers in terms of prediction accuracy . Our model outperforms these baselines for most of the triggers on the two datasets , achieving 82.42 % accuracy on predicting the adverb " also " on the Gigaword dataset . The contributions of this work are as follows : 1 . We introduce the task of predicting adverbial presupposition triggers . 2 . We present new datasets for the task of detecting adverbial presupposition triggers , with a data extraction method that can be applied to other similar pre-processing tasks . 3 . We develop a new attention mechanism in an RNN architecture that is appropriate for the prediction of adverbial presupposition triggers , and show that its use results in better prediction performance over a number of baselines without introducing additional parameters . 2 Related Work In this work , we have investigated the task of predicting adverbial presupposition triggers and introduced several datasets for the task . Additionally , we have presented a novel weighted-pooling attention mechanism which is incorporated into a recurrent neural network model for predicting the presence of an adverbial presuppositional trigger . Our results show that the model outperforms the CNN and LSTM , and does not add any additional parameters over the standard LSTM model . This shows its promise in classification tasks involving capturing and combining relevant information from multiple points in the previous context . In future work , we would like to focus more on designing models that can deal with and be optimized for scenarios with severe data imbalance . We would like to also explore various applications of presupposition trigger prediction in language generation applications , as well as additional attention-based neural network architectures .

Adverbaial triggers indicate the event recurrence, continuation, or termination in the discourse context and are frequently found in English but there are few related works.

They introduce an adverbial presupposition trigger prediction task and datasets and propose an attention mechanism that augments a recurrent neural network without additional trainable parameters.

The proposed model outperforms baselines including an LSTM-based language model on most of the triggers on the two datasets.


This paper proposes a novel method that exploits multiple resources to improve statistical machine translation ( SMT ) based paraphrasing . In detail , a phrasal paraphrase table and a feature function are derived from each resource , which are then combined in a log-linear SMT model for sentence-level paraphrase generation . Experimental results show that the SMT-based paraphrasing model can be enhanced using multiple resources . The phrase-level and sentence-level precision of the generated paraphrases are above 60 % and 55 % , respectively . In addition , the contribution of each resource is evaluated , which indicates that all the exploited resources are useful for generating paraphrases of high quality . Paraphrases are alternative ways of conveying the same meaning . Paraphrases are important in many natural language processing ( NLP ) applications , such as machine translation ( MT ) , question answering ( QA ) , information extraction ( IE ) , multidocument summarization ( MDS ) , and natural language generation ( NLG ) . This paper addresses the problem of sentencelevel paraphrase generation , which aims at generating paraphrases for input sentences . An example of sentence-level paraphrases can be seen below : S1 : The table was set up in the carriage shed . S2 : The table was laid under the cart-shed . Paraphrase generation can be viewed as monolingual machine translation ( Quirk et al . , 2004 ) , which typically includes a translation model and a language model . The translation model can be trained using monolingual parallel corpora . However , acquiring such corpora is not easy . Hence , data sparseness is a key problem for the SMT-based paraphrasing . On the other hand , various methods have been presented to extract phrasal paraphrases from different resources , which include thesauri , monolingual corpora , bilingual corpora , and the web . However , little work has been focused on using the extracted phrasal paraphrases in sentence-level paraphrase generation . In this paper , we exploit multiple resources to improve the SMT-based paraphrase generation . In detail , six kinds of resources are utilized , including : ( 1 ) an automatically constructed thesaurus , ( 2 ) a monolingual parallel corpus from novels , ( 3 ) a monolingual comparable corpus from news articles , ( 4 ) a bilingual phrase table , ( 5 ) word definitions from Encarta dictionary , and ( 6 ) a corpus of similar user queries . Among the resources , ( 1 ) , ( 2 ) , ( 3 ) , and ( 4 ) have been investigated by other researchers , while ( 5 ) and ( 6 ) are first used in this paper . From those resources , six phrasal paraphrase tables are extracted , which are then used in a log-linear SMTbased paraphrasing model . Both phrase-level and sentence-level evaluations were carried out in the experiments . In the former one , phrase substitutes occurring in the paraphrase sentences were evaluated . While in the latter one , the acceptability of the paraphrase sentences was evaluated . Experimental results show that : ( 1 ) The SMT-based paraphrasing is enhanced using multiple resources . The phrase-level and sentence-level precision of the generated paraphrases exceed 60 % and 55 % , respectively . ( 2 ) Although the contributions of the resources differ a lot , all the resources are useful . ( 3 ) The performance of the method varies greatly on different test sets and it performs best on the test set of news sentences , which are from the same source as most of the training data . The rest of the paper is organized as follows : Section 2 reviews related work . Section 3 introduces the log-linear model for paraphrase generation . Section 4 describes the phrasal paraphrase extraction from different resources . Section 5 presents the parameter estimation method . Section 6 shows the experiments and results . Section 7 draws the conclusion . This paper proposes a method that improves the SMT-based sentence-level paraphrase generation using phrasal paraphrases automatically extracted from different resources . Our contribution is that we combine multiple resources in the framework of SMT for paraphrase generation , in which the dic-tionary definitions and similar user queries are first used as phrasal paraphrases . In addition , we analyze and compare the contributions of different resources . Experimental results indicate that although the contributions of the exploited resources differ a lot , they are all useful to sentence-level paraphrase generation . Especially , the dictionary definitions and similar user queries are effective for paraphrasing some certain types of phrases . In the future work , we will try to use syntactic and context constraints in paraphrase generation to enhance the acceptability of the paraphrases . In addition , we will extract paraphrase patterns that contain more structural variation and try to combine the SMT-based and pattern-based systems for sentencelevel paraphrase generation .

Paraphrase generation requires monolingual parallel corpora which is not easily obtainable, and few works focus on using the extracted phrasal paraphrases in sentence-level paraphrase generation.

They propose to exploit six paraphrase resources to extract phrasal paraphrase tables that are further used to build a log-linear statistical machine translation-based paraphrasing model.

They show that using multiple resources enhances paraphrase generation quality in precision on phrase and sentence level especially when they are similar to user queries.


There are many possible different semantic relationships between nominals . Classification of such relationships is an important and difficult task ( for example , the well known noun compound classification task is a special case of this problem ) . We propose a novel pattern clusters method for nominal relationship ( NR ) classification . Pattern clusters are discovered in a large corpus independently of any particular training set , in an unsupervised manner . Each of the extracted clusters corresponds to some unspecified semantic relationship . The pattern clusters are then used to construct features for training and classification of specific inter-nominal relationships . Our NR classification evaluation strictly follows the ACL SemEval-07 Task 4 datasets and protocol , obtaining an f-score of 70.6 , as opposed to 64.8 of the best previous work that did not use the manually provided WordNet sense disambiguation tags . Automatic extraction and classification of semantic relationships is a major field of activity , of both practical and theoretical interest . A prominent type of semantic relationships is that holding between nominals1 . For example , in noun compounds many different semantic relationships are encoded by the same simple form ( Girju et al . , 2005 ) : ' dog food ' denotes food consumed by dogs , while ' summer morn-ing ' denotes a morning that happens in the summer . These two relationships are completely different semantically but are similar syntactically , and distinguishing between them could be essential for NLP applications such as question answering and machine translation . Relation classification usually relies on a training set in the form of tagged data . To improve results , some systems utilize additional manually constructed semantic resources such as WordNet ( WN ) ( Beamer et al . , 2007 ) . However , in many domains and languages such resources are not available . Furthermore , usage of such resources frequently requires disambiguation and connection of the data to the resource ( word sense disambiguation in the case of WordNet ) . Manual disambiguation is unfeasible in many practical tasks , and an automatic one may introduce errors and greatly degrade performance . It thus makes sense to try to minimize the usage of such resources , and utilize only corpus contexts in which the relevant words appear . A leading method for utilizing context information for classification and extraction of relationships is that of patterns ( Hearst , 1992 ; Pantel and Pennacchiotti , 2006 ) . The standard classification process is to find in an auxiliary corpus a set of patterns in which a given training word pair co-appears , and use pattern-word pair co-appearance statistics as features for machine learning algorithms . In this paper we introduce a novel approach , based on utilizing pattern clusters that are prepared separately and independently of the training set . We do not utilize any manually constructed resource or any manual tagging of training data beyond the cor-rect classification , thus making our method applicable to fully automated tasks and less domain and language dependent . Moreover , our pattern clustering algorithm is fully unsupervised . Our method is based on the observation that while each lexical pattern can be highly ambiguous , several patterns in conjunction can reliably define and represent a lexical relationship . Accordingly , we construct pattern clusters from a large generic corpus , each such cluster potentially representing some important generic relationship . This step is done without accessing any training data , anticipating that most meaningful relationships , including those in a given classification problem , will be represented by some of the discovered clusters . We then use the training set to label some of the clusters , and the labeled clusters to assign classes to tested items . One of the advantages of our method is that it can be used not only for classification , but also for further analysis and retrieval of the observed relationships2 . The semantic relationships between the components of noun compounds and between nominals in general are not easy to categorize rigorously . Several different relationship hierarchies have been proposed ( Nastase and Szpakowicz , 2003 ; Moldovan et al . , 2004 ) . Some classes , like Container-Contained , Time-Event and Product-Producer , appear in several classification schemes , while classes like Tool-Object are more vaguely defined and are subdivided differently . Recently , SemEval-07 Task 4 ( Girju et al . , 2007 ) proposed a benchmark dataset that includes a subset of 7 widely accepted nominal relationship ( NR ) classes , allowing consistent evaluation of different NR classification algorithms . In the SemEval event , 14 research teams evaluated their algorithms using this benchmark . Some of the teams have used the manually annotated WN labels provided with the dataset , and some have not . We evaluated our algorithm on SemEval-07 Task 4 data , showing superior results over participating algorithms that did not utilize WordNet disambiguation tags . We also show how pattern clusters can be used for a completely unsupervised classification of the test set . Since in this case no training data is used , this allows the automated discovery of a potentially unbiased classification scheme . Section 2 discusses related work , Section 3 outlines the pattern clustering algorithm , Section 4 details three classification methods , and Sections 5 and 6 describe the evaluation protocol and results . Relationship classification is known to improve many practical tasks , e.g. , textual entailment ( Tatu and Moldovan , 2005 ) . We have presented a novel framework for relationship classification , based on pattern clusters prepared as a standalone resource independently of the training set . Our method outperforms current state-of-the-art algorithms that do not utilize WordNet tags on Task 4 of SemEval-07 . In practical situations , it would not be feasible to provide a large amount of such sense disambiguation tags manually . Our method also shows competitive performance compared to the majority of task participants that do utilize WN tags . Our method can produce labeled pattern clusters , which can be potentially useful for automatic discovery of additional instances for a given relationship . We intend to pursue this promising direction in future work .

Using annotated data or semantic resources such as WordNet for relation classification introduces errors and such data is not available in many domains and languages.

They propose an unsupervised pattern clustering method for nominal relation classification using a large generic corpus enabling scale in domain and language.

Experiments on the ACL SemEval-07 dataset show the proposed method performs better than existing methods that do not use disambiguation tags.


Learning sentence embeddings from dialogues has drawn increasing attention due to its low annotation cost and high domain adaptability . Conventional approaches employ the siamese-network for this task , which obtains the sentence embeddings through modeling the context-response semantic relevance by applying a feed-forward network on top of the sentence encoders . However , as the semantic textual similarity is commonly measured through the element-wise distance metrics ( e.g. cosine and L2 distance ) , such architecture yields a large gap between training and evaluating . In this paper , we propose DialogueCSE , a dialogue-based contrastive learning approach to tackle this issue . DialogueCSE first introduces a novel matching-guided embedding ( MGE ) mechanism , which generates a contextaware embedding for each candidate response embedding ( i.e. the context-free embedding ) according to the guidance of the multi-turn context-response matching matrices . Then it pairs each context-aware embedding with its corresponding context-free embedding and finally minimizes the contrastive loss across all pairs . We evaluate our model on three multi-turn dialogue datasets : the Microsoft Dialogue Corpus , the Jing Dong Dialogue Corpus , and the E-commerce Dialogue Corpus . Evaluation results show that our approach significantly outperforms the baselines across all three datasets in terms of MAP and Spearman 's correlation measures , demonstrating its effectiveness . Further quantitative experiments show that our approach achieves better performance when leveraging more dialogue context and remains robust when less training data is provided . Sentence embeddings are used with success for a variety of NLP applications ( Cer et al . , 2018 ) and many prior methods have been proposed with different learning schemes . Kiros et al . ( 2015 ) ; Logeswaran and Lee ( 2018 ) ; Hill et al . ( 2016 ) train sentence encoders in a self-supervised manner with web pages and books . Conneau et al . ( 2017 ) ; Cer et al . ( 2018 ) ; Reimers and Gurevych ( 2019 ) propose to learn sentence embeddings on the supervised datasets such as SNLI ( Bowman et al . , 2015 ) and MNLI ( Williams et al . , 2018 ) . Although the supervised-learning approaches achieve better performance , they suffer from high cost of annotation in building the training dataset , which makes them hard to adapt to other domains or languages . Recently , learning sentence embeddings from dialogues has begun to attract increasing attention . Dialogues provide strong semantic relationships among conversational utterances and are usually easy to collect in large amounts . Such advantages make the dialogue-based self-supervised learning methods promising to achieve competitive or even superior performance against the supervised-learning methods , especially under the low-resource conditions . While promising , the issue of how to effectively exploit the dialogues for this task has not been sufficiently explored . Yang et al . ( 2018 ) propose to train an input-response prediction model on Reddit dataset ( Al-Rfou et al . , 2016 ) . Since they build their architecture based on the single-turn dialogue , the multi-turn dialogue history is not fully exploited . Henderson et al . ( 2020 ) demonstrate that introducing the multi-turn dialogue context can improve the sentence embedding performance . However , they concatenate the multi-turn dialogue context into a long token sequence , failing to model intersentence semantic relationships among the utterances . Recently , more advanced methods such as ( Reimers and Gurevych , 2019 ) achieve better performance by employing BERT ( Devlin et al . , 2019 ) as the sentence encoder . These works have in common that they employ a feed-forward network with a non-linear activation on top of the sentence en-coders to model the context-response semantic relevance , thereby learning the sentence embeddings . However , such architecture presents two limitations : ( 1 ) It yields a large gap between training and evaluating , since the semantic textual similarity is commonly measured by the element-wise distance metrics such as cosine and L2 distance . ( 2 ) Concatenating all the utterances in the dialogue context inevitably introduces the noise as well as the redundant information , resulting in a poor result . In this paper , we propose DialogueCSE , a dialogue-based contrastive learning approach to tackle these issues . We hold that the semantic matching relationships between the context and the response can be implicitly modeled through contrastive learning , thus making it possible to eliminate the gap between training and evaluating . To this end , we introduce a novel matching-guided embedding ( MGE ) mechanism . Specifically , MGE first pairs each utterance in the context with the response and performs a token-level dot-product operation across all the utterance-response pairs to obtain the multi-turn matching matrices . Then the multi-turn matching matrices are used as guidance to generate a context-aware embedding for the response embedding ( i.e. the context-free embedding ) . Finally , the context-aware embedding and the context-free embedding are paired as a training sample , whose label is determined by whether the context and the response are originally from the same dialogue . Our motivation is that once the context semantically matches the response , it has the ability to distill the context-aware information from the context-free embedding , which is exactly the learning objective of the sentence encoder that aims to produce context-aware sentence embeddings . We train our model on three multi-turn dialogue datasets : the Microsoft Dialogue Corpus ( MDC ) ( Li et al . , 2018 ) , the Jing Dong Dialogue Corpus ( JDDC ) ( Chen et al . , 2020 ) , and the E-commerce Dialogue Corpus ( ECD ) ( Zhang et al . , 2018 ) . To evaluate our model , we introduce two types of tasks : the semantic retrieval ( SR ) task and the dialogue-based semantic textual similarity ( D-STS ) task . Here we do not adopt the standard semantic textual similarity ( STS ) task ( Cer et al . , 2017 ) for two reasons : ( 1 ) As revealed in ( Zhang et al . , 2020 ) , the sentence embedding performance varies greatly as the domain of the training data changes . As a dialogue dataset is always about several certain domains , evaluating on the STS benchmark may mis-lead the evaluation of the model . ( 2 ) The dialoguebased sentence embeddings focus on context-aware rather than context-free semantic meanings , which may not be suitable to be evaluated through the context-free benchmarks . Since previous dialoguebased works have not set up a uniform benchmark , we construct two evaluation datasets for each dialogue corpus . A total of 18,964 retrieval samples and 4,000 sentence pairs are annotated by seven native speakers through the crowd-sourcing platform1 . The evaluation results indicate that DialogueCSE significantly outperforms the baselines on the three datasets in terms of both MAP and Spearman 's correlation metrics , demonstrating its effectiveness . Further quantitative experiments show that Dia-logueCSE achieves better performance when leveraging more dialogue context and remains robust when less training data is provided . To sum up , our contributions are threefold : • We propose DialogueCSE , a dialogue-based contrastive learning approach with MGE mechanism for learning sentence embeddings from dialogues . As far as we know , this is the first attempt to apply contrastive learning in this area . • We construct the dialogue-based sentence embedding evaluation benchmarks for three dialogue corpus . All of the datasets will be released to facilitate the follow-up researches . • Extensive experiments show that Dia-logueCSE significantly outperforms the baselines , establishing the state-of-the-art results . 2 Related Work In this work , we propose DialogueCSE , a dialoguebased contrastive learning approach to learn sentence embeddings from dialogues . We also propose uniform evaluation benchmarks for evaluating the quality of the dialogue-based sentence embeddings . Evaluation results show that DialogueCSE achieves the best result over the baselines while adding no additional parameters . In the next step , we will study how to introduce more interaction information to learn the sentence embeddings and try to incorporate the contrast learning method into the pre-training stage .

Existing methods to learn representations from dialogues have a similarity-measurement gap between training and evaluation time and do not exploit the multi-turn structure of data.

They propose a dialogue-based contrastive learning approach to learn sentence embeddings from dialogues by modelling semantic matching relationships between the context and response implicitly.

The proposed approach outperforms baseline methods on two newly introduced tasks coupled with three multi-turn dialogue datasets in terms of MAP and Spearman's correlation measures.