In this paper , we propose a neural network model for graph-based dependency parsing which utilizes Bidirectional LSTM ( BLSTM ) to capture richer contextual information instead of using high-order factorization , and enable our model to use much fewer features than previous work .

In addition , we propose an effective way to learn sentence segment embedding on sentence-level based on an extra forward LSTM network .

Although our model uses only first-order factorization , experiments on English Peen Treebank and Chinese Penn Treebank show that our model could be competitive with previous higher-order graph-based dependency parsing models and state-of-the-art models .

Dependency parsing is a fundamental task for language processing which has been investigated for decades .

It has been applied in a wide range of applications such as information extraction and machine translation .

Among a variety of dependency parsing models , graph-based models are attractive for their ability of scoring the parsing decisions on a whole-tree basis .

Typical graph-based models factor the dependency tree into subgraphs , including single arcs ( McDonald et al . , 2005 ) , sibling or grandparent arcs

( McDonald and Pereira , 2006 ; Carreras , 2007 ) or higher-order substructures ( Koo and Collins , 2010 ; Ma and Zhao , 2012 ) and then score the whole tree by summing scores of the subgraphs .

In these models , subgraphs are usually represented as high-dimensional feature vectors which are then fed into a linear model to learn the feature weights .

However , conventional graph-based models heavily rely on feature engineering and their performance is restricted by the design of features .

In addition , standard decoding algorithm ( Eisner , 2000 ) only works for the first-order model which limits the scope of feature selection .

To incorporate high-order features , Eisner algorithm must be somehow extended or modified , which is usually done at high cost in terms of efficiency .

The fourth-order graph-based model ( Ma and Zhao , 2012 ) , which seems the highest-order model so far to our knowledge , requires O(n 5 ) time and O(n 4 ) space .

Due to the high computational cost , highorder models are normally restricted to producing only unlabeled parses to avoid extra cost introduced by inclusion of arc-labels into the parse trees .

To alleviate the burden of feature engineering , Pei et al . ( 2015 ) presented an effective neural network model for graph-based dependency parsing .

They only use atomic features such as word unigrams and POS tag unigrams and leave the model to automatically learn the feature combinations .

However , their model requires many atomic features and still relies on high-order factorization strategy to further improve the accuracy .

Different from previous work , we propose an LSTM-based dependency parsing model in this paper and aim to use LSTM network to capture richer contextual information to support parsing decisions , instead of adopting a high-order factorization .

The main advantages of our model are as follows :

• By introducing Bidirectional LSTM , our model shows strong ability to capture potential long range contextual information and exhibits improved accuracy in recovering long distance dependencies .

It is different to previous work in which a similar effect is usually achieved by high-order factorization .

More-over , our model also eliminates the need for setting feature selection windows and reduces the number of features to a minimum level .

• We propose an LSTM-based sentence segment embedding method named LSTM-Minus , in which distributed representation of sentence segment is learned by using subtraction between LSTM hidden vectors .

Experiment shows this further enhances our model 's ability to access to sentence-level information .

• Last but important , our model is a first-order model using standard Eisner algorithm for decoding , the computational cost remains at the lowest level among graph-based models .

Our model does not trade-off efficiency for accuracy .

We evaluate our model on the English Penn Treebank and Chinese Penn Treebank , experiments show that our model achieves competitive parsing accuracy compared with conventional high-order models , however , with a much lower computational cost .

In this paper , we propose an LSTM-based neural network model for graph-based dependency parsing .

Utilizing Bidirectional LSTM and segment embeddings learned by LSTM-Minus allows our model access to sentence-level information , making our model more accurate in recovering longdistance dependencies with only first-order factorization .

Experiments on PTB and CTB show that our model could be competitive with conventional high-order models with a faster speed .


Text mining and data harvesting algorithms have become popular in the computational linguistics community .

They employ patterns that specify the kind of information to be harvested , and usually bootstrap either the pattern learning or the term harvesting process ( or both ) in a recursive cycle , using data learned in one step to generate more seeds for the next .

They therefore treat the source text corpus as a network , in which words are the nodes and relations linking them are the edges .

The results of computational network analysis , especially from the world wide web , are thus applicable .

Surprisingly , these results have not yet been broadly introduced into the computational linguistics community .

In this paper we show how various results apply to text mining , how they explain some previously observed phenomena , and how they can be helpful for computational linguistics applications .

Text mining / harvesting algorithms have been applied in recent years for various uses , including learning of semantic constraints for verb participants ( Lin and Pantel , 2002 ) related pairs in various relations , such as part-whole ( Girju et al . , 2003 ) , cause ( Pantel and Pennacchiotti , 2006 ) , and other typical information extraction relations , large collections of entities ( Soderland et al . , 1999 ; Etzioni et al . , 2005 ) , features of objects ( Pasca , 2004 ) and ontologies ( Carlson et al . , 2010 ) .

They generally start with one or more seed terms and employ patterns that specify the desired information as it relates to the seed(s ) .

Several approaches have been developed specifically for learning patterns , including guided pattern collection with manual filtering ( Riloff and Shepherd , 1997 ) automated surface-level pattern induction ( Agichtein and Gravano , 2000 ; Ravichandran and Hovy , 2002 ) probabilistic methods for taxonomy relation learning ( Snow et al . , 2005 ) and kernel methods for relation learning ( Zelenko et al . , 2003 ) .

Generally , the harvesting procedure is recursive , in which data ( terms or patterns ) gathered in one step of a cycle are used as seeds in the following step , to gather more terms or patterns .

This method treats the source text as a graph or network , consisting of terms ( words ) as nodes and inter-term relations as edges .

Each relation type induces a different network1 .

Text mining is a process of network traversal , and faces the standard problems of handling cycles , ranking search alternatives , estimating yield maxima , etc .

The computational properties of large networks and large network traversal have been studied intensively ( Sabidussi , 1966 ; Freeman , 1979 ; Watts and Strogatz , 1998 ) and especially , over the past years , in the context of the world wide web ( Page et al . , 1999 ; Broder et al . , 2000 ; Kleinberg and Lawrence , 2001 ; Li et al . , 2005 ; Clauset et al . , 2009 ) .

Surprisingly , except in ( Talukdar and Pereira , 2010 ) , this work has not yet been related to text mining research in the computational linguistics community .

The work is , however , relevant in at least two ways .

It sometimes explains why text mining algo-rithms have the limitations and thresholds that are empirically found ( or suspected ) , and it may suggest ways to improve text mining algorithms for some applications .

In Section 2 , we review some related work .

In Section 3 we describe the general harvesting procedure , and follow with an examination of the various statistical properties of implicit semantic networks in Section 4 , using our implemented harvester to provide illustrative statistics .

In Section 5 we discuss implications for computational linguistics research .

In this paper we describe the implicit ' hidden ' semantic network graph structure induced over the text of the web and other sources by the semantic relations people use in sentences .

We describe how term harvesting patterns whose seed terms are harvested and then applied recursively can be used to discover these semantic term networks .

Although these networks differ considerably from the web in relation density , type , and network size , we show , somewhat surprisingly , that the same power-law , smallworld effect , transitivity , and most other characteristics that apply to the web 's hyperlinked network structure hold also for the implicit semantic term graphs-certainly for the semantic relations and languages we have studied , and most probably for almost all semantic relations and human languages .

This rather interesting observation leads us to surmise that the hyperlinks people create in the web are of essentially the same type as the semantic relations people use in normal sentences , and that they form an extension of normal language that was not needed before because people did not have the ability within the span of a single sentence to ' embed ' structures larger than a clause-certainly not a whole other page 's worth of information .

The principal exception is the academic citation reference ( lexicalized as " see " ) , which is not used in modern webpages .

Rather , the ' lexicalization ' now used is a formatting convention : the hyperlink is colored and often underlined , facilities offered by computer screens but not available to speech or easy in traditional typesetting .


Question answering ( QA ) systems for large document collections typically use pipelines that ( i ) retrieve possibly relevant documents , ( ii ) re-rank them , ( iii ) rank paragraphs or other snippets of the top-ranked documents , and ( iv ) select spans of the top-ranked snippets as exact answers .

Pipelines are conceptually simple , but errors propagate from one component to the next , without later components being able to revise earlier decisions .

We present an architecture for joint document and snippet ranking , the two middle stages , which leverages the intuition that relevant documents have good snippets and good snippets come from relevant documents .

The architecture is general and can be used with any neural text relevance ranker .

We experiment with two main instantiations of the architecture , based on POSIT-DRMM ( PDRMM ) and a BERT-based ranker .

Experiments on biomedical data from BIOASQ show that our joint models vastly outperform the pipelines in snippet retrieval , the main goal for QA , with fewer trainable parameters , also remaining competitive in document retrieval .

Furthermore , our joint PDRMM-based model is competitive with BERT-based models , despite using orders of magnitude fewer parameters .

These claims are also supported by human evaluation on two test batches of BIOASQ .

To test our key findings on another dataset , we modified the Natural Questions dataset so that it can also be used for document and snippet retrieval .

Our joint PDRMM-based model again outperforms the corresponding pipeline in snippet retrieval on the modified Natural Questions dataset , even though it performs worse than the pipeline in document retrieval .

We make our code and the modified Natural Questions dataset publicly available .

Question answering ( QA ) systems that search large document collections ( Voorhees , 2001 ; Tsatsaro-nis et al . , 2015 ; Chen et al . , 2017 ) typically use pipelines operating at gradually finer text granularities .

A fully-fledged pipeline includes components that ( i ) retrieve possibly relevant documents typically using conventional information retrieval ( IR ) ; ( ii ) re-rank the retrieved documents employing a computationally more expensive document ranker ; ( iii ) rank the passages , sentences , or other ' snippets ' of the top-ranked documents ; and ( iv ) select spans of the top-ranked snippets as ' exact ' answers .

Recently , stages ( ii)-(iv ) are often pipelined neural models , trained individually ( Hui et al . , 2017 ; Pang et al . , 2017 ; Lee et al . , 2018 ; McDonald et al . , 2018 ;

Mackenzie et al . , 2020 ; Sekulić et al . , 2020 ) .

Although pipelines are conceptually simple , errors propagate from one component to the next ( Hosein et al . , 2019 ) , without later components being able to revise earlier decisions .

For example , once a document has been assigned a low relevance score , finding a particularly relevant snippet can not change the document 's score .

We propose an architecture for joint document and snippet ranking , i.e. , stages ( ii ) and ( iii ) , which leverages the intuition that relevant documents have good snippets and good snippets come from relevant documents .

We note that modern web search engines display the most relevant snippets of the top-ranked documents to help users quickly identify truly relevant documents and answers ( Sultan et al . , 2016 ; Xu et al . , 2019 ; Yang et al . , 2019a ) .

The top-ranked snippets can also be used as a starting point for multi-document query-focused summarization , as in the BIOASQ challenge ( Tsatsaronis et al . , 2015 ) .

Hence , methods that identify good snippets are useful in several other applications , apart from QA .

We also note that many neural models for stage ( iv ) have been proposed , often called QA or Machine Reading Comprehension ( MRC ) models ( Kadlec et al . , 2016 ; Cui et al . , 2017 ; Zhang et al . , 2020 ) , but they typically search for answers only in a particular , usually paragraph-sized snippet , which is given per question .

For QA systems that search large document collections , stages ( ii ) and ( iii ) are also important , if not more important , but have been studied much less in recent years , and not in a single joint neural model .

The proposed joint architecture is general and can be used in conjunction with any neural text relevance ranker ( Mitra and Craswell , 2018 ) .

Given a query and N possibly relevant documents from stage ( i ) , the neural text relevance ranker scores all the snippets of the N documents .

Additional neural layers re-compute the score ( ranking ) of each document from the scores of its snippets .

Other layers then revise the scores of the snippets taking into account the new scores of the documents .

The entire model is trained to jointly predict document and snippet relevance scores .

We experiment with two main instantiations of the proposed architecture , using POSIT-DRMM ( McDonald et al . , 2018 ) , hereafter called PDRMM , as the neural text ranker , or a BERT-based ranker ( Devlin et al . , 2019 ) .

We show how both PDRMM and BERT can be used to score documents and snippets in pipelines , then how our architecture can turn them into models that jointly score documents and snippets .

Experimental results on biomedical data from BIOASQ ( Tsatsaronis et al . , 2015 ) show the joint models vastly outperform the corresponding pipelines in snippet extraction , with fewer trainable parameters .

Although our joint architecture is engineered to favor retrieving good snippets ( as a near-final stage of QA ) , results show that the joint models are also competitive in document retrieval .

We also show that our joint version of PDRMM , which has the fewest parameters of all models and does not use BERT , is competitive to BERT-based models , while also outperforming the best system of BIOASQ 6 ( Brokos et al . , 2018 ) in both document and snippet retrieval .

These claims are also supported by human evaluation on two test batches of BIOASQ 7 ( 2019 ) .

To test our key findings on another dataset , we modified Natural Questions ( Kwiatkowski et al . , 2019 ) , which only includes questions and answer spans from a single document , so that it can be used for document and snippet retrieval .

Again , our joint PDRMMbased model largely outperforms the corresponding pipeline in snippet retrieval on the modified Natural Questions , though it does not perform better than the pipeline in document retrieval , since the joint model is geared towards snippet retrieval , i.e. , even though it is forced to extract snippets from fewer relevant documents .

Finally , we show that all the neural pipelines and joint models we considered improve the BM25 ranking of traditional IR on both datasets .

We make our code and the modified Natural Questions publicly available.1 2 Methods

Our contributions can be summarized as follows :

( 1 ) We proposed an architecture to jointly rank documents and snippets with respect to a question , two particularly important stages in QA for large document collections ; our architecture can be used with any neural text relevance model .

( 2 ) We instantiated the proposed architecture using a recent neural relevance model ( PDRMM ) and a BERTbased ranker .

( 3 ) Using biomedical data ( from BIOASQ ) , we showed that the two resulting joint models ( PDRMM-based and BERT-based ) vastly outperform the corresponding pipelines in snippet re-trieval , the main goal in QA for document collections , using fewer parameters , and also remaining competitive in document retrieval .

( 4 ) We showed that the joint model ( PDRMM-based ) that does not use BERT is competitive with BERT-based models , outperforming the best BIOASQ 6 system ; our joint models ( PDRMM-and BERT-based ) also outperformed all BIOASQ 7 competitors .

( 5 ) We provide a modified version of the Natural Questions dataset , suitable for document and snippet retrieval .

( 6 ) We showed that our joint PDRMM-based model also largely outperforms the corresponding pipeline on open-domain data ( Natural Questions ) in snippet retrieval , even though it performs worse than the pipeline in document retrieval .

( 7 ) We showed that all the neural pipelines and joint models we considered improve the traditional BM25 ranking on both datasets .

( 8) We make our code publicly available .

We hope to extend our models and datasets for stage ( iv ) , i.e. , to also identify exact answer spans within snippets ( paragraphs ) , similar to the answer spans of SQUAD ( Rajpurkar et al . , 2016 ( Rajpurkar et al . , , 2018 ) ) .

This would lead to a multi-granular retrieval task , where systems would have to retrieve relevant documents , relevant snippets , and exact answer spans from the relevant snippets .

BIOASQ already includes this multi-granular task , but exact answers are provided only for factoid questions and they are freely written by humans , as in MS-MARCO , with similar limitations .

Hence , appropriately modified versions of the BIOASQ datasets are needed .

Table 4 shows that further performance gains ( 6.80 to 7.85 document MAP , 15.42 to 17.34 snippet MAP ) are possible by tuning the weights of the two losses .

The best scores are obtained when using both the extra sentence and document features .

However , the model performs reasonably well even when one of the two types of extra features is removed , with the exception of λ snip = 10 .

The standard deviations of the MAP scores over the folds of the cross-validation indicate that the performance of the model is reasonably stable .

The architecture is general and can be used with any neural text relevance ranker .

These claims are also supported by human evaluation on two test batches of BIOASQ .


Query expansion techniques generally select new query terms from a set of top ranked documents .

Although a user 's manual judgment of those documents would much help to select good expansion terms , it is difficult to get enough feedback from users in practical situations .

In this paper we propose a query expansion technique which performs well even if a user notifies just a relevant document and a non-relevant document .

In order to tackle this specific condition , we introduce two refinements to a well-known query expansion technique .

One is application of a transductive learning technique in order to increase relevant documents .

The other is a modified parameter estimation method which laps the predictions by multiple learning trials and try to differentiate the importance of candidate terms for expansion in relevant documents .

Experimental results show that our technique outperforms some traditional query expansion methods in several evaluation measures .

Query expansion is a simple but very useful technique to improve search performance by adding some terms to an initial query .

While many query expansion techniques have been proposed so far , a standard method of performing is to use relevance information from a user ( Ruthven , 2003 ) .

If we can use more relevant documents in query expansion , the likelihood of selecting query terms achieving high search improvement increases .

However it is impractical to expect enough relevance information .

Some researchers said that a user usually notifies few relevance feedback or nothing ( Dumais and et al . , 2003 ) .

In this paper we investigate the potential performance of query expansion under the condition that we can utilize little relevance information , especially we only know a relevant document and a nonrelevant document .

To overcome the lack of relevance information , we tentatively increase the number of relevant documents by a machine learning technique called Transductive Learning .

Compared with ordinal inductive learning approach , this learning technique works even if there is few training examples .

In our case , we can use many documents in a hit-list , however we know the relevancy of few documents .

When applying query expansion , we use those increased documents as if they were true relevant ones .

When applying the learning , there occurs some difficult problems of parameter settings .

We also try to provide a reasonable resolution for the problems and show the effectiveness of our proposed method in experiments .

The point of our query expansion method is that we focus on the availability of relevance information in practical situations .

There are several researches which deal with this problem .

Pseudo relevance feedback which assumes top n documents as relevant ones is one example .

This method is simple and relatively effective if a search engine returns a hit-list which contains a certain number of relative documents in the upper part .

However , unless this assumption holds , it usually gives a worse ranking than the initial search .

Thus several researchers propose some specific procedure to make pseudo feedback be effective ( Yu and et al , 2003 ; Lam-Adesina and Jones , 2001 ) .

In another way , Onoda ( Onoda et al . , 2004 ) tried to apply one-class SVM ( Support Vector Machine ) to relevance feedback .

Their purpose is to improve search performance by using only nonrelevant documents .

Though their motivation is similar to ours in terms of applying a machine learning method to complement the lack of relevance information , the assumption is somewhat different .

Our assumption is to utilizes manual but the minimum relevance judgment .

Transductive leaning has already been applied in the field of image retrieval ( He and et al . , 2004 ) .

In this research , they proposed a transductive method called the manifold-ranking algorithm and showed its effectiveness by comparing with active learning based Support Vector Machine .

However , their setting of relevance judgment is not different from many other traditional researches .

They fix the total number of images that are marked by a user to 20 .

As we have already claimed , this setting is not practical because most users feel that 20 is too much for judgment .

We think none of research has not yet answered the question .

For relevance judgment , most of the researches have adopted either of the following settings .

One is the setting of " Enough relevant documents are available " , and the other is " No relevant document is available " .

In contrast to them , we adopt the setting of " Only one relevant document is available " .

Our aim is to achieve performance improvement with the minimum effort of judging relevancy of documents .

The reminder of this paper is structured as follows .

Section 2 describes two fundamental techniques for our query expansion method .

Section 3 explains a technique to complement the smallness of manual relevance judgment .

Section 4 introduces a whole procedure of our query expansion method step by step .

Section 5 shows empirical evidence of the effectiveness of our method compared with two traditional query expansion methods .

Section 6 investigates the experimental results more in detail .

Finally , Section 7 summarizes our findings .

In this paper we proposed a novel query expansion method which only use the minimum manual judgment .

To complement the lack of relevant documents , this method utilizes the SGT transductive learning algorithm to predict the relevancy of unjudged documents .

Since the performance of SGT much depends on an estimation of the fraction of relevant documents , we propose a method to sample some good fraction values .

We also propose a method to laps the predictions of multiple SGT trials with above sampled fraction values and try to differentiate the importance of candidate terms for expansion in relevant documents .

The experimental results showed our method outperforms other query expansion methods in the evaluations of several criteria .

However it is impractical to expect enough relevance information .


In this paper , we study differentiable neural architecture search ( NAS ) methods for natural language processing .

In particular , we improve differentiable architecture search by removing the softmax-local constraint .

Also , we apply differentiable NAS to named entity recognition ( NER ) .

It is the first time that differentiable NAS methods are adopted in NLP tasks other than language modeling .

On both the PTB language modeling and CoNLL-2003 English NER data , our method outperforms strong baselines .

It achieves a new state-ofthe-art on the NER task .

Neural architecture search ( NAS ) has become popular recently in machine learning for their ability to find new models and to free researchers from the hard work of designing network architectures .

The earliest of these approaches use reinforcement learning ( RL ) to learn promising architectures in a discrete space ( Zoph and Le , 2016 ) , whereas others have successfully modeled the problem in a continuous manner ( Liu et al . , 2019 ;

As an instance of the latter , differentiable architecture search ( DARTS ) employs continuous relaxation to architecture representation and makes gradient descent straightforwardly applicable to search .

This leads to an efficient search process that is orders of magnitude faster than the RL-based counterparts .

Like recent methods in NAS ( Xie and Yuille , 2017 ; Zoph and Le , 2016 ; Baker et al . , 2016 ) , DARTS represents networks as a directed acyclic graph for a given computation cell ( see Figure 1(a ) ) .

An edge between nodes performs a predefined operation to transform the input ( i.e. , tail ) Figure 1 : An overview of DARTS cell and our cell to the output ( i.e. , head ) .

For a continuous network space , DARTS uses the softmax trick to relax the categorical choice of edges to soft decisions .

Then , one can optimize over the graph using standard gradient descent methods .

The optimized network is inferred by choosing the edges with maximum weights in softmax .

However , DARTS is a " local " model because the softmax-based relaxation is imposed on each bundle of edges between two nodes .

This leads to a biased model in that edges coming from different nodes are not comparable .

Such a constraint limits the inference space to sub-graphs with one edge between each pair of nodes .

Also , the learned network might be redundant because every node has to receive edges from all predecessors no matter they are necessary or not .

This problem is similar to the bias problem in other graph-based models where local decisions make the model nonoptimal ( Lafferty et al . , 2001 ; Daphne Koller and Nir Friedman , 2009 ) .

Here we present an improvement of DARTS , called I-DARTS , that further relaxes the softmaxlocal constraint .

The idea is simple -we consider all incoming edges to a given node in a single softmax .

This offers a broader choice of edges and enlarges the space we infer the network from .

For example , one can simultaneously select multiple important edges between two nodes and leave some node pairs unlinked ( see Figure 1(b ) ) .

I-DARTS outperforms strong baselines on the PTB language modeling and CoNLL named entity recognition ( NER ) tasks .

This gives a new stateof-the-art on the NER dataset .

To our knowledge , it is the first time to apply differentiable architecture search methods to NLP tasks other than language modeling .

More interestingly , we observe that our method is 1.4X faster than DARTS for convergence of architecture search .

Also , we provide the architectures learned by I-DARTS , which can be referred for related tasks .

We improved the DARTS to address the bias problem by removing the softmax-local constraint .

Our method is search efficient and discovers several better architectures for PTB language modeling and CoNLL named entity recognition ( NER ) tasks .

We plan to consider the network density problem in search and apply I-DARTS to more tasks in our future study .


Generating open-domain conversational responses in the desired style usually suffers from the lack of parallel data in the style .

Meanwhile , using monolingual stylistic data to increase style intensity often leads to the expense of decreasing content relevance .

In this paper , we propose to disentangle the content and style in latent space by diluting sentence-level information in style representations .

Combining the desired style representation and a response content representation will then obtain a stylistic response .

Our approach achieves a higher BERT-based style intensity score and comparable BLEU scores , compared with baselines .

Human evaluation results show that our approach significantly improves style intensity and maintains content relevance .

Linguistic style is an essential aspect of natural language interaction and provides particular ways of using language to engage with the audiences ( Kabbara and Cheung , 2016 ) .

In human-bot conversations , it is crucial to generate stylistic responses for increasing user engagement to conversational systems ( Gan et al . , 2017 ) .

Currently , most of the existing parallel datasets are not stylistically consistent .

Samples in these datasets are usually contributed by a variety of users , resulting in an averaging effect across style characteristics ( Zhang et al . , 2018a ) .

Meanwhile , constructing a parallel stylistic dataset for training the open-domain conversational agents is both labor-intensive and time-consuming .

Recent studies show the effect of stylizing responses using a monolingual dataset in the desired style and a conventional conversational dataset ( Niu and Bansal , 2018 ; Gao et al . , 2019b ) .

However , increasing style intensity often leads to ( Niu and Bansal , 2018 ) , Style Fusion ( Gao et al . , 2019b ) , and our approach , targeting the Holmes style , which is quite formal and polite .

the expense of decreasing content relevance between dialogue history and response .

As an example in Figure 1 shows , Niu and Bansal ( 2018 ) independently train a response generation model and a stylistic language model and subsequently interpolates them in the inference phase .

Lacking the interaction between the stylistic language model and response generation encoder , it usually yields a trade-off between style intensity and content relevance .

Gao et al . ( 2019a , b ) fuse a structured latent space where the direction denotes the diversity , and the distance denotes style intensity and content relevance .

The main issue is that style intensity and content relevance are contradictory in measurement but are coupling to the same " distance " metric of the latent space .

To sum up , the key issue of the above studies is the improper entanglement of style and content .

To address the issue , we propose to disentangle the style and content of a response .

The disentanglement is conducted on the structured latent space , where each sentence ( dialogue history , response , and stylistic sentence ) is projected into a vector representation .

We further split the representation into two components : style and content representations .

The former is a corpus-level feature since sentences within a dataset have the same style .

In contrast , the content representation is a sentence-level feature decided by a sentence itself .

We thus disentangle the content and style by diluting sentence-level information in the style representation .

This encourages the encoding of content information into the content representation .

Otherwise , the content information will be corrupted in the style representation , making it hard to reconstruct the original content in the subsequent decoding process .

We conduct experiments on DailyDialogue conversational dataset ( Li et al . , 2017 ) and Holmes monolingual stylistic dataset ( Gao et al . , 2019b ) .

Experimental results show that our proposed approach improves style intensity and maintains content relevance .

• We propose a unified framework to simultaneously improve style intensity and maintain content relevance for neural stylistic response generation .

• We introduce a scheme of learning latent variables by a diluting strategy to disentangle the style and content .

• Experimental results show that our approach achieves higher performance in style intensity without decreasing content relevance , compared with previous approaches .

We propose a uniform framework to simultaneously improve the style intensity and maintain the content relevance for neural stylistic response generation .

In contrast to existing approaches , our approach disentangles the style and the content in the latent space by a diluting strategy .

Experiments show that our approach improves the style intensity of generated responses and maintains the content relevance at the same time , which demonstrates the effectiveness of this approach .


Large Transformers pretrained over clinical notes from Electronic Health Records ( EHR ) have afforded substantial gains in performance on predictive clinical tasks .

The cost of training such models ( and the necessity of data access to do so ) coupled with their utility motivates parameter sharing , i.e. , the release of pretrained models such as ClinicalBERT ( Alsentzer et al . , 2019 ) .

While most efforts have used deidentified EHR , many researchers have access to large sets of sensitive , nondeidentified EHR with which they might train a BERT model ( or similar ) .

Would it be safe to release the weights of such a model if they did ?

In this work , we design a battery of approaches intended to recover Personal Health Information ( PHI ) from a trained BERT .

Specifically , we attempt to recover patient names and conditions with which they are associated .

We find that simple probing methods are not able to meaningfully extract sensitive information from BERT trained over the MIMIC-III corpus of EHR .

However , more sophisticated " attacks " may succeed in doing so : To facilitate such research , we make our experimental setup and baseline probing models available .

Pretraining large ( masked ) language models such as BERT ( Devlin et al . , 2019 ) over domain specific corpora has yielded consistent performance gains across a broad range of tasks .

In biomedical NLP , this has often meant pretraining models over collections of Electronic Health Records ( EHRs ) ( Alsentzer et al . , 2019 ) .

For example , Huang et al . ( 2019 ) showed that pretraining models over EHR data improves performance on clinical predictive tasks .

Given their empirical utility , and the fact that pretraining large networks requires a nontrivial amount of compute , there is a natural desire to equal contribution .

1 https://github.com / elehman16/ exposing_patient_data_release .

share the model parameters for use by other researchers in the community .

However , in the context of pretraining models over patient EHR , this poses unique potential privacy concerns : Might the parameters of trained models leak sensitive patient information ?

In the United States , the Health Insurance Portability and Accountability Act ( HIPAA ) prohibits the sharing of such text if it contains any reference to Protected Health Information ( PHI ) .

If one removes all reference to PHI , the data is considered " deidentified " , and is therefore legal to share .

While researchers may not directly share nondeidentified text,2 it is unclear to what extent models pretrained on non-deidentified data pose privacy risks .

Further , recent work has shown that general purpose large language models are prone to memorizing sensitive information which can subsequently be extracted ( Carlini et al . , 2020 ) .

In the context of biomedical NLP , such concerns have been cited as reasons for withholding direct publication of trained model weights ( McKinney et al . , 2020 ) .

These uncertainties will continue to hamper dissemination of trained models among the broader biomedical NLP research community , motivating a need to investigate the susceptibility of such models to adversarial attacks .

This work is a first step towards exploring the potential privacy implications of sharing model weights induced over non-deidentified EHR text .

We propose and run a battery of experiments intended to evaluate the degree to which Transformers ( here , BERT ) pretrained via standard masked language modeling objectives over notes in EHR might reveal sensitive information ( Figure 1 ) .

We find that simple methods are able to recover associations between patients and conditions at rates better than chance , but not with performance beyond that achievable using baseline condition frequencies .

This holds even when we enrich clinical notes by explicitly inserting patient names into every sentence .

Our results using a recently proposed , more sophisticated attack based on generating text ( Carlini et al . , 2020 ) are mixed , and constitute a promising direction for future work .

We have performed an initial investigation into the degree to which large Transformers pretrained over EHR data might reveal sensitive personal health information ( PHI ) .

We ran a battery of experiments in which we attempted to recover such information from BERT model weights estimated over the MIMIC-III dataset ( into which we artificially reintroduced patient names , as MIMIC is deidentified ) .

Across these experiments , we found that we were mostly unable to meaningfully expose PHI using simple methods .

Moreover , even when we constructed a variant of data in which we prepended patient names to every sentence prior to pretraining BERT , we were still unable to recover sensitive information reliably .

Our initial results using more advanced techniques based on generation ( Carlini et al . 2020 ; Table 9 ) are intriguing but inconclusive at present .

Our results certainly do not rule out the possibility that more advanced methods might reveal PHI .

But , these findings do at least suggest that doing so is not trivial .

To facilitate further research , we make our experimental setup and baseline probing models available : https://github.com/ elehman16 / exposing_patient_data_release .


We investigate the influence of information status ( IS ) on constituent order in German , and integrate our findings into a loglinear surface realisation ranking model .

We show that the distribution of pairs of IS categories is strongly asymmetric .

Moreover , each category is correlated with morphosyntactic features , which can be automatically detected .

We build a loglinear model that incorporates these asymmetries for ranking German string realisations from input LFG F-structures .

We show that it achieves a statistically significantly higher BLEU score than the baseline system without these features .

There are many factors that influence word order , e.g. humanness , definiteness , linear order of grammatical functions , givenness , focus , constituent weight .

In some cases , it can be relatively straightforward to automatically detect these features ( i.e. in the case of definiteness , this is a syntactic property ) .

The more complex the feature , the more difficult it is to automatically detect .

It is common knowledge that information status1 ( henceforth , IS ) has a strong influence on syntax and word order ; for instance , in inversions , where the subject follows some preposed element , Birner ( 1994 ) reports that the preposed element must not be newer in the discourse than the subject .

We would like to be able to use information related to IS in the automatic generation of German text .

Ideally , we would automatically annotate text with IS labels and learn from this data .

Unfortunately , however , to date , there has been little success in automatically annotating text with IS .

We believe , however , that despite this shortcoming , we can still take advantage of some of the insights gained from looking at the influence of IS on word order .

Specifically , we look at the problem from a more general perspective by computing an asymmetry ratio for each pair of IS categories .

Results show that there are a large number of pairs exhibiting clear ordering preferences when co-occurring in the same clause .

The question then becomes , without being able to automatically detect these IS category pairs , can we , nevertheless , take advantage of these strong asymmetric patterns in generation .

We investigate the ( automatically detectable ) morphosyntactic characteristics of each asymmetric IS pair and integrate these syntactic asymmetric properties into the generation process .

The paper is structured as follows : Section 2 outlines the underlying realisation ranking system for our experiments .

Section 3 introduces information status and Section 4 describes how we extract and measure asymmetries in information status .

In Section 5 , we examine the syntactic characteristics of the IS asymmetries .

Section 6 outlines realisation ranking experiments to test the integration of IS into the system .

We discuss our findings in Section 7 and finally we conclude in Section 8 .

In this paper we presented a novel method of including IS into the task of generation ranking .

Since automatic annotation of IS labels themselves is not currently possible , we approximate the IS categories by their syntactic characteristics .

By calculating strong asymmetries between pairs of IS labels , and establishing the most frequent syntactic characteristics of these asymmetries , we designed a new set of features for a log-linear ranking model .

In comparison to a baseline model , we achieve statistically significant improvement in BLEU score .

We showed that these improvements were not only due to the effect of purely syntactic asymmetries , but that the IS asymmetries were what drove the improved model .


This paper explores the determinants of adjective-noun plausibility by using correlation analysis to compare judgements elicited from human subjects with five corpus-based variables : co-occurrence frequency of the adjective-noun pair , noun frequency , conditional probability of the noun given the adjective , the log-likelihood ratio , and Resnik 's ( 1993 ) selectional association measure .

The highest correlation is obtained with the co-occurrence frequency , which points to the strongly lexicalist and collocational nature of adjective-noun combinations .

Research on linguistic plausibility has focused mainly on the effects of argument plausibility during the processing of locally ambiguous sentences .

Psycholinguists have investigated whether the plausibility of the direct object affects reading times for sentences like ( 1 ) .

Here , argument plausibility refers to " pragmatic plausibility " or " local semantic fit " ( Holmes et al . , 1989 ) , and judgements of plausibility are typically obtained by asking subjects to rate sentence fragments containing verb-argument combinations ( as an example consider the bracketed parts of the sentences in ( 1 ) ) .

Such experiments typically use an ordinal scale for plausibility ( e.g. , from 1 to 7 ) .

[ The senior senator regretted the decision ] had ever been made public .

[ The senior senator regretted the reporter ] had ever seen the report .

The majority of research has focussed on investigating the effect of rated plausibility for verb-object combinations in human sentence processing ( Garnsey et al . , 1997 ; Pickering and Traxler , 1998 ) .

However , plausibility effects have also been observed for adjectivenoun combinations in a head-modifier relationship .

Murphy ( 1990 ) has shown that typical adjectivenoun phrases ( e.g. , salty olives ) are easier to interpret in comparison to atypical ones ( e.g. , sweet olives ) .

Murphy provides a schema-based explanation for this finding by postulating that in typical adjective-noun phrases , the adjective modifies part of the noun 's schema and consequently it is understood more quickly , whereas in atypical combinations , the adjective modifies non-schematic aspects of the noun , which leads to interpretation difficulties .

Smadja ( 1991 ) argues that the reason people prefer strong tea to powerful tea and powerful car to strong car is neither purely syntactic nor purely semantic , but rather lexical .

A similar argument is put forward by Cruse ( 1986 ) , who observes that the adjective spotless collocates well with the noun kitchen , relatively worse with the noun complexion and not all with the noun taste .

According to Cruse , words like spotless have idiosyncratic collocational restrictions : differences in the degree of acceptability of the adjective and its collocates do not seem to depend on the meaning of the individual words .

This paper explored the determinants of linguistic plausibility , a concept that is potentially relevant for lexical choice in natural language generation systems .

Adjective-noun plausibility served as a test bed for a number of corpus-based models of linguistic plausibility .

Plausibility judgements were obtained from human subjects for 90 randomly selected adjective-noun pairs .

The ratings revealed a clear effect of familiarity of the adjective-noun pair ( operationalised by corpus co-occurrence frequency ) .

In a correlation analysis we compared judged plausibility with the predictions of five corpus-based variables .

The highest correlation was obtained with the co-occurrence frequency of the adjective-noun pair .

Conditional probability , the log-likelihood ratio , and Resnik 's ( 1993 ) selectional association measure were also significantly correlated with plausibility ratings .

The correlation with Resnik 's measure was negative , contrary to the predictions of his model .

This points to a problem with his technique for estimating word class frequencies , which is aggravated by the collocational nature of noun-adjective combinations .

Overall , the results confirm the strongly lexicalist and collocational nature of adjective-noun combinations .

This fact could be exploited in a generation system by taking into account corpus co-occurrence counts for adjective-noun pairs ( which can be obtained straightforwardly ) during lexical choice .

Future research has to identify how this approach can be generalised to unseen data .


Unsupervised relation extraction ( URE ) extracts relations between named entities from raw text without manually-labelled data and existing knowledge bases ( KBs ) .

URE methods can be categorised into generative and discriminative approaches , which rely either on hand-crafted features or surface form .

However , we demonstrate that by using only named entities to induce relation types , we can outperform existing methods on two popular datasets .

We conduct a comparison and evaluation of our findings with other URE techniques , to ascertain the important features in URE .

We conclude that entity types provide a strong inductive bias for URE . 1

Relation extraction ( RE ) extracts semantic relations between entities from plain text .

For instance , " Jon Robin Baitz head , born in Los Angeles tail ... " expresses the relation /people / person / place of birth between the two head-tail entities .

Extracted relations are then used for several downstream tasks such as information retrieval ( Corcoglioniti et al . , 2016 ) and knowledge base construction ( Al-Zaidy and Giles , 2018 ) .

RE has been widely studied using fully supervised learning ( Nguyen and Grishman , 2015 ; Miwa and Bansal , 2016 ; Zhang et al . , 2017 Zhang et al . , , 2018 ) ) and distantly supervised approaches ( Mintz et al . , 2009 ; Riedel et al . , 2010 ; Lin et al . , 2016 ) .

Unsupervised relation extraction ( URE ) methods have not been explored as much as fully or distantly supervised learning techniques .

URE is promising , since it does not require manually annotated data nor human curated knowledge bases ( KBs ) , which are expensive to produce .

Therefore , it can be applied to domains and languages where annotated data and KBs are not available .

Moreover , URE can discover new relation types , since it is not restricted to specific relation types in the same way as fully and distantly supervised methods .

One might argue that Open Information Extraction ( OpenIE ) can also discover new relations .

However , OpenIE identifies relations based on textual surface information .

Thus , similar relations with different textual forms may not be recognised .

Unlike OpenIE , URE groups similar relations into clusters .

Despite these advantages , there are only a few attempts tackling URE using machine learning ( ML ) ( Hasegawa et al . , 2004 ; Banko et al . , 2007 ; Yao et al . , 2011 ; Marcheggiani and Titov , 2016 ; Simon et al . , 2019 ) .

Similarly to other unsupervised learning tasks , a challenge in URE is how to evaluate results .

Recent approaches ( Yao et al . , 2011 ; Marcheggiani and Titov , 2016 ; Simon et al . , 2019 ) employ a widely used data generation setting in distantly supervised RE , i.e. , aligning a large amount of raw text against triplets in a curated KB .

A standard metric score is computed by comparing the output relation clusters against the automatically annotated relations .

In particular , the NYT-FB dataset ( Marcheggiani and Titov , 2016 ) which is used for evaluation , has been created by mapping relation triplets in Freebase ( Bollacker et al . , 2008 ) against plain text articles in the New York Times ( NYT ) corpus ( Sandhaus , 2008 ) .

Standard clustering evaluation metrics for URE include B 3 ( Bagga and Baldwin , 1998 ) , V-measure ( Rosenberg and Hirschberg , 2007 ) , and ARI ( Hubert and Arabie , 1985 ) .

Although the above mentioned experimental setting can be created automatically , there are three challenges to overcome .

Firstly , the development and test sets are silver , i.e. , they include noisy labelled instances , since they are not human-curated .

Secondly , the development and test sentences are part of the training set , i.e. , a transductive setting .

It is thus unclear how well the existing models perform on unseen sentences .

Finally , NYT-FB can be considered highly imbalanced , since only 2.1 % of the training sentences can be aligned with Freebase 's triplets .

Due to the noisy nature of silver data ( NYT-FB ) , evaluation on silver data will not accurately reflect the system performance .

We also need unseen data during testing to examine the system generalisation .

To overcome these challenges , we will employ the test set of TACRED ( Zhang et al . , 2017 ) , a widely used manually annotated corpus .

Regarding the imbalanced data , we will demonstrate that in fact around 60 % ( instead of 2.1 % ) of instances in the training set express relation types defined in Freebase .

In this work , we present a simple URE approach relying only on entity types that can obtain improved performance compared to current methods .

Specifically , given a sentence consisting of two entities and their corresponding entity types , e.g. , PERSON and LOCATION , we induce relations as the combination of entity types , e.g. , PERSON-LOCATION .

It should be noted that we employ only entity types because their combinations form reasonably coarse relation types ( e.g. , PERSON-LOCATION covers /people / person / place of birth defined in Freebase ) .

We further discuss our improved performance in § 3 .

Our contributions are as follows : ( i ) We perform experiments on both automatically / manuallylabelled datasets , namely NYT-FB and TACRED , respectively .

We show that two methods using only entity types can outperform the state-of-theart models including both feature-engineering and deep learning approaches .

The surprising results raise questions about the current state of unsupervised relation extraction .

( ii ) For model design , we show that link predictor provides a good signal to train a URE model ( Fig 1 ) .

We also illustrate that entity types are a strong inductive bias for URE ( Table 1 ) .

We have shown the importance of entity types in URE .

Our methods use only entity types , yet they yield higher performance than previous work on both NYT-FB and TACRED .

We have investigated the current experimental setting , concluding that a strong inductive bias is required to train a relation extraction model without labelled data .

URE remains challenging , which requires improved methods to deal with silver data .

We also plan to use different types of labelled data , e.g. , domain specific data sets , to ascertain whether entity type information is more discriminative in sub-languages .


Although discriminative training guarantees to improve statistical machine translation by incorporating a large amount of overlapping features , it is hard to scale up to large data due to decoding complexity .

We propose a new algorithm to generate translation forest of training data in linear time with the help of word alignment .

Our algorithm also alleviates the oracle selection problem by ensuring that a forest always contains derivations that exactly yield the reference translation .

With millions of features trained on 519 K sentences in 0.03 second per sentence , our system achieves significant improvement by 0.84 BLEU over the baseline system on the NIST Chinese-English test sets .

Discriminative model ( Och and Ney , 2002 ) can easily incorporate non-independent and overlapping features , and has been dominating the research field of statistical machine translation ( SMT ) in the last decade .

Recent work have shown that SMT benefits a lot from exploiting large amount of features ( Liang et al . , 2006 ; Tillmann and Zhang , 2006 ; Watanabe et al . , 2007 ; Blunsom et al . , 2008 ; Chiang et al . , 2009 ) .

However , the training of the large number of features was always restricted in fairly small data sets .

Some systems limit the number of training examples , while others use short sentences to maintain efficiency .

Overfitting problem often comes when training many features on a small data ( Watanabe et al . , 2007 ; Chiang et al . , 2009 ) .

Obviously , using much more data can alleviate such problem .

Furthermore , large data also enables us to globally train millions of sparse lexical features which offer accurate clues for SMT .

Despite these advantages , to the best of our knowledge , no previous discriminative training paradigms scale up to use a large amount of training data .

The main obstacle comes from the complexity of packed forests or n-best lists generation which requires to search through all possible translations of each training example , which is computationally prohibitive in practice for SMT .

To make normalization efficient , contrastive estimation ( Smith and Eisner , 2005 ; Poon et al . , 2009 ) introduce neighborhood for unsupervised log-linear model , and has presented positive results in various tasks .

Motivated by these work , we use a translation forest ( Section 3 ) which contains both " reference " derivations that potentially yield the reference translation and also neighboring " non-reference " derivations that fail to produce the reference translation.1

However , the complexity of generating this translation forest is up to O(n 6 ) , because we still need biparsing to create the reference derivations .

Consequently , we propose a method to fast generate a subset of the forest .

The key idea ( Section 4 ) is to initialize a reference derivation tree with maximum score by the help of word alignment , and then traverse the tree to generate the subset forest in linear time .

Besides the efficiency improvement , such a forest allows us to train the model without resort- r 1 X ⇒ ⟨X 1 bei X 2 , X 1 was X 2 ⟩ e 2 r 2 X ⇒ ⟨qiangshou bei X 1 , the gunman was X 1 ⟩ e 3 r 3 X ⇒ ⟨jingfang X 1 , X 1 by the police⟩ e 4 r 4 X ⇒ ⟨jingfang X 1 , police X 1 ⟩ e 5 r 5 X ⇒ ⟨qiangshou , the gunman⟩ e 6 r 6 X ⇒ ⟨jibi , shot dead⟩ Figure 1 : A translation forest which is the running example throughout this paper .

The reference translation is " the gunman was killed by the police " .

( 1 ) Solid hyperedges denote a " reference " derivation tree t 1 which exactly yields the reference translation .

( 2 ) Replacing e 3 in t 1 with e 4 results a competing non-reference derivation t 2 , which fails to swap the order of X 3,4 .

( 3 ) Removing e 1 and e 5 in t 1 and adding e 2 leads to another reference derivation t 3 .

Generally , this is done by deleting a node X 0,1 .

ing to constructing the oracle reference ( Liang et al . , 2006 ; Watanabe et al . , 2007 ; Chiang et al . , 2009 ) , which is non-trivial for SMT and needs to be determined experimentally .

Given such forests , we globally learn a log-linear model using stochastic gradient descend ( Section 5 ) .

Overall , both the generation of forests and the training algorithm are scalable , enabling us to train millions of features on large-scale data .

To show the effect of our framework , we globally train millions of word level context features motivated by word sense disambiguation ( Chan et al . , 2007 ) together with the features used in traditional SMT system ( Section 6 ) .

Training on 519 K sentence pairs in 0.03 seconds per sentence , we achieve significantly improvement over the traditional pipeline by 0.84 BLEU .

We have presented a fast generation algorithm for translation forest which contains both reference derivations and neighboring non-reference derivations for large-scale SMT discriminative training .

We have achieved significantly improvement of 0.84 BLEU by incorporate 13.9 M feature trained on 519 K data in 0.03 second per sentence .

In this paper , we define the forest based on competing derivations which only differ in one rule .

There may be better classes of forest that can produce a better performance .

It 's interesting to modify the definition of forest , and use more local operators to increase the size of forest .

Furthermore , since the generation of forests is quite general , it 's straight to apply our forest on other learning algorithms .

Finally , we hope to exploit more features such as reordering features and syntactic features so as to further improve the performance .


Neural machine translation ( NMT ) with recurrent neural networks , has proven to be an effective technique for end-to-end machine translation .

However , in spite of its promising advances over traditional translation methods , it typically suffers from an issue of unbalanced outputs , that arise from both the nature of recurrent neural networks themselves , and the challenges inherent in machine translation .

To overcome this issue , we propose an agreement model for neural machine translation and show its effectiveness on large-scale Japaneseto-English and Chinese-to-English translation tasks .

Our results show the model can achieve improvements of up to 1.4 BLEU over the strongest baseline NMT system .

With the help of an ensemble technique , this new end-to-end NMT approach finally outperformed phrasebased and hierarchical phrase-based Moses baselines by up to 5.6 BLEU points .

Recurrent neural network ( RNN ) has achieved great successes on several structured prediction tasks ( Graves , 2013 ; Watanabe and Sumita , 2015 ; Dyer et al . , 2015 ) , in which RNNs are required to make a sequence of dependent predictions .

One of its advantages is that an unbounded history is available to enrich the context for the prediction at the current time-step .

Despite its successes , recently , ( Liu et al . , 2016 ) pointed out that the RNN suffers from a fundamental issue of generating unbalanced outputs : that is to say the suffixes of its outputs are typically worse than the prefixes .

This is due to the fact that later predictions directly depend on the accuracy of previous predictions .

They empirically demonstrated this issue on two simple sequence-to-sequence learning tasks : machine transliteration and grapheme-to-phoneme conversion .

On the more general sequence-to-sequence learning task of machine translation ( MT ) , neural machine translation ( NMT ) based on RNNs has recently become an active research topic ( Sutskever et al . , 2014 ; Bahdanau et al . , 2014 ) .

Compared to those two simple tasks , MT involves in much larger vocabulary and frequent reordering between input and output sequences .

This makes the prediction at each time-step far more challenging .

In addition , sequences in MT are much longer , with averaged length of 36.7 being about 5 times longer than that in grapheme-to-phoneme conversion .

Therefore , we believe that the history is more likely to contain incorrect predictions and the issue of unbalanced outputs may be more serious .

This hypothesis is supported later ( see Table 1 in § 4.1 ) , by an analysis that shows the quality of the prefixes of translation hypotheses is much higher than that of the suffixes .

To address this issue for NMT , in this paper we extend the agreement model proposed in ( Liu et al . , 2016 ) to the task of machine translation .

Its key idea is to encourage the agreement between a pair of target-directional ( left-to-right and right-to-left ) NMT models in order to produce more balanced translations and thus improve the overall translation quality .

• We introduce a simple and general method to address the issue of unbalanced outputs for NMT ( § 3 ) .

This method is robust without any extra hyperparameters to tune and is easy to implement .

In addition , it is general enough to be applied on top of any of the existing RNN translation models , although it was implemented on top of the model in ( Bahdanau et al . , 2014 ) in this paper .

• We provide an empirical evaluation of the technique on large scale Japanese-to-English and Chinese-to-English translation tasks .

The results show our model can generate more balanced translation results , and achieves substantial improvements ( of up to 1.4 BLEU points ) over the strongest NMT baseline ( § 4 ) .

With the help of an ensemble technique , our new end-to-end NMT gains up to 5.6 BLEU points over phrase-based and hierarchical phrasebased Moses ( Koehn et al . , 2007 )

2 Overview of Neural Machine Translation

Suppose x = x 1 , x 2 , • • • , x m denotes a source sentence , y = y 1 , y 2 , • • • , y n denotes a target sen- tence .

In addition , let x < t = x 1 , x 2 , • • • , x t-1 denote a prefix of x.

Neural Machine Translation ( NMT ) directly maps a source sentence into a target within a probabilistic framework .

Formally , it defines a conditional probability over a pair of sequences x and y via a recurrent neural network as follows :

where θ is the set of model parameters ; h t denotes a hidden state ( i.e. a vector ) of y at timestep t ; g is a transformation function from a hidden state to a vector with dimension of the target-side vocabulary size ; softmax is the softmax function , and [ i ] denotes the i th component in a vector .

) is defined by a recurrent function over both the previous hidden state h t-1 and the context c(x , y < t ) .3

Note that both h t and c(x , y < t ) have dimension d for all t.

In this paper , we develop our model on top of the neural machine translation approach of ( Bahdanau et al . , 2014 ) , and we refer the reader this paper for a complete description of the model , for example , the definitons of f and c. The proposed method could just as easily been implemented on top of any other RNN models such as that in ( Sutskever et al . , 2014 ) .

In this paper , we investigate the issue of unbalanced outputs suffered by recurrent neural networks , and empirically show its existence in the context of machine translation .

To address this issue , we propose an easy to implement agreement model that extends the method of ( Liu et al . , 2016 ) from simple sequence-to-sequence learning tasks to machine translation .

On two challenging JP-EN and CH-EN translation tasks , our approach was empirically shown to be effective in addressing the issue ; by generating balanced outputs , it was able to consistently outperform a respectable NMT baseline on all test sets , delivering gains of up to 1.4 BLEU points .

To put these results in the broader context of machine translation research , our approach ( even without special handling of unknown words ) achieved gains of up to 5.6 BLEU points over strong phrase-based and hierarchical phrase-based Moses baselines , with the help of an ensemble technique .


We approached the problem as learning how to order documents by estimated relevance with respect to a user query .

Our support vector machines based classifier learns from the relevance judgments available with the standard test collections and generalizes to new , previously unseen queries .

For this , we have designed a representation scheme , which is based on the discrete representation of the local ( lw ) and global ( gw ) weighting functions , thus is capable of reproducing and enhancing the properties of such popular ranking functions as tf.idf , BM25 or those based on language models .

Our tests with the standard test collections have demonstrated the capability of our approach to achieve the performance of the best known scoring functions solely from the labeled examples and without taking advantage of knowing those functions or their important properties or parameters .

Our work is motivated by the objective to bring closer numerous achievements in the domains of machine learning and classification to the classical task of ad-hoc information retrieval ( IR ) , which is ordering documents by the estimated degree of relevance to a given query .

Although used with striking success for text categorization , classification-based approaches ( e.g. those based on support vector machines , Joachims , 2001 ) have been relatively abandoned when trying to improve ad hoc retrieval in favor of empirical ( e.g. vector space , Salton & McGill , 1983 ) or generative ( e.g. language models ; Zhai & Lafferty 2001 ; Song & Croft ; 1999 ) , which produce a ranking function that gives each document a score , rather than trying to learn a classifier that would help to discriminate between relevant and irrelevant documents and order them accordingly .

A generative model needs to make assumptions that the query and document words are sampled from the same underlying distributions and that the distributions have certain forms , which entail specific smoothing techniques ( e.g. popular Dirichlet-prior ) .

A discriminative ( classifier-based ) model , on the other side , does not need to make any assumptions about the forms of the underlying distributions or the criteria for the relevance but instead , learns to predict to which class a certain pattern ( document ) belongs to based on the labeled training examples .

Thus , an important advantage of a discriminative approach for the information retrieval task , is its ability to explicitly utilize the relevance judgments existing with standard test collections in order to train the IR algorithms and possibly enhance retrieval accuracy for the new ( unseen ) queries .

Cohen , Shapire and Singer ( 1999 ) noted the differences between ordering and classification and presented a two-stage model to learn ordering .

The first stage learns a classifier for preference relations between objects using any suitable learning mechanism ( e.g. support vector machines ; Vapnik , 1998 ) .

The second stage converts preference relations into a rank order .

Although the conversion may be NP complete in a general case , they presented efficient approximations .

We limited our first study reported here to linear classifiers , in which conversion can be performed by simple ordering according to the score of each document .

However , approaching the problem as " learning how to order things " allowed us to design our sampling and training mechanisms in a novel and , we believe , more powerful way .

Our classifier learns how to compare every pair of documents with respect to a given query , based on the relevance indicating features that the documents may have .

As it is commonly done in information retrieval , the features are derived from the word overlap between the query and documents .

According to Nallapati ( 2004 ) , the earliest formulation of the classic IR problem as a classification ( discrimination ) problem was suggested by Robertson and Sparck Jones ( 1976 ) , however performed well only when the relevance judgments were available for the same query but not generalizing well to new queries .

Fuhr and Buckley ( 1991 ) used polynomial regression to estimate the coefficients in a linear scoring function combining such well-known features as a weighted term frequency , document length and query length .

They tested their " description-oriented " approach on the standard small-scale collections ( Cranfield , NPL , INSPEC , CISI , CACM ) to achieve the relative change in the average precision ranging from -17 % to + 33 % depending on the collection tested and the implementation parameters .

Gey ( 1994 ) applied logistic regression in a similar setting with the following results : Cranfield +12 % , CACM

-4.4 % , however he did not test them on new ( unseen by the algorithm ) queries , hypothesizing that splitting documents into training and testing collections would not be possible since " a large number of queries is necessary in order to train for a decent logistic regression approach to document retrieval . "

Instead , he applied a regression trained on Cranfield to CISI collection but with a negative effect .

Recently , the approaches based on learning have reported several important breakthroughs .

Fan et al . ( 2004 ) applied genetic programming in order to learn how to combine various terms into the optimal ranking function that outperformed the popular Okapi formula on robust retrieval test collection .

Nallapati ( 2004 ) made a strong argument in favor of discriminative models and trained an SVM-based classifier to combine 6 different components ( terms ) from the popular ranking functions ( such as tf.idf and language models ) to achieve better than the language model performance in 2 out of 16 test cases ( figure 3 in Nallapati , 2004 ) , not statistically distinguishable in 8 cases and only 80 % of the best performance in 6 cases .

There have been studies using past relevance judgements to optimize retrieval .

For example , Joachims ( 2002 ) applied Support Vector Machines to learn linear ranking function from user click-throughs while interfacing with a search engine .

In this study , we have developed a representation scheme , which is based on the discretization of the global ( corpus statistics ) and local ( document statistics ) weighting of term overlaps between queries and documents .

We have empirically shown that this representation is flexible enough to learn the properties of the popular ranking functions : tf.idf , BM25 and the language models .

The major difference of our work from Fan et al . ( 2004 ) or Nallapati ( 2004 ) or works on fusion ( e.g. Vogt & Cottrell , 1999 ) is that we did not try to combine several known ranking functions ( or their separate terms ) into one , but rather we learn the weighting functions directly through discretization .

Discretization allows representing a continuous function by a set of values at certain points .

These values are learned by a machine learning technique to optimize certain criteria , e.g. average precision .

Another important motivation behind using discretization was to design a representation with high dimensionality of features in order to combine our representation scheme with Support Vector Machines ( SVM ) ( Vapnik , 1998 ) , which are known to work well with a large number of features .

SVM contains a large class of neural nets , radial margin separation ( RBF ) nets , and polynomial classifiers as special cases .

They have been delivering superior performance in classification tasks in general domains , e.g. in face recognition ( Hearst , 1998 ) , and in text categorization ( Joachims , 2001 ) .

Another important distinction of this work from the prior research is that we train our classifier not to predict the absolute relevance of a document d with respect to a query q , but rather to predict which of the two documents d1 , d2 is more relevant to the query q.

The motivation for this distinction was that all the popular evaluation metrics in information retrieval ( e.g. average precision ) are based on document ranking rather than classification accuracy .

This affected our specially designed sampling procedure which we empirically discovered to be crucial for successful learning .

We have also empirically established that our combination of the representation scheme , learning mechanism and sampling allows learning from the past relevance judgments in order to successfully generalize to the new ( unseen ) queries .

When the representation was created without any knowledge of the top ranking functions and their parameters , our approach reached the known top performance solely through the learning process .

When our representation was taking advantage of functions that are known to perform well and their parameters , the resulting combination was able to slightly exceed the top performance on large test collections .

The next section formalizes our Discretization Based Learning ( DBL ) approach to Information Retrieval , followed by empirical results and conclusions .

We explored learning how to rank documents with respect to a given query using linear Support Vector Machines and discretization-based representation .

Our approach represents a family of discriminative approaches , currently studied much less than heuristic ( tf.idf , bm25 ) or generative approaches ( language models ) .

Our experiments indicate that learning from relevant judgments available with the standard test collections and generalizing to new queries is not only feasible but can be a source of improvement .

When tested with a popular standard collection , our approach achieved the performance of the best well-known techniques ( BM25 and language models ) , which have been developed as a result of extensive past experiments and elaborate theoretical modeling .

When combined with the best performing ranking functions , our approach added a small ( 2 - 3 % ) , but statistically significant , improvement .

Although practical significance of this study may be limited at the moment since it does not demonstrate a dramatic increase in retrieval performance in large test collections , we believe our findings have important theoretical contributions since they indicate that the power of discriminative approach is comparable to the best known analytical or heuristic apporaches .

This work also lays the foundation for extending the discriminative approach to " richer " representations , such as those using word n-grams , grammatical relations between words , and the structure of documents .

Our results also indicate that gw.lw family , which includes practically all the popular " bag of words " ranking formulas such as tf.idf , BM25 or language models , has almost reached its upper limit and other classes of representations and ranking formulas need to be explored in order to accomplish significant performance break-troughs .

Of course , using only few test cases ( topics sets and collections ) is a limitation of this current study , which we are going to address in our future research .

We view our approach as a complement , rather than competitive , to the analytical approaches such as language models .

Our approach can be also used as an explorative tool in order to identify important relevance-indicating features , which can be later modeled analytically .

We believe that our work and the ones referred in this paper may bring many of the achievements made in a more general area of classification and machine learning closer to the task of rank ordered information retrieval , thus making retrieval engines more helpful in reducing the information overload and meeting people 's needs .


Conventional solutions to automatic related work summarization rely heavily on humanengineered features .

In this paper , we develop a neural data-driven summarizer by leveraging the seq2seq paradigm , in which a joint context-driven attention mechanism is proposed to measure the contextual relevance within full texts and a heterogeneous bibliography graph simultaneously .

Our motivation is to maintain the topic coherency between a related work section and its target document , where both the textual and graphic contexts play a big role in characterizing the relationship among scientific publications accurately .

Experimental results on a large dataset show that our approach achieves a considerable improvement over a typical seq2seq summarizer and five classical summarization baselines .

In scientific fields , scholars need to contextualize their contribution to help readers acquire an understanding of their research papers .

For this purpose , the related work section of an article serves as a pivot to connect prior domain knowledge , in which the innovation and superiority of current work are displayed by a comparison with previous studies .

While citation prediction can assist in drafting a reference collection ( Nallapati et al . , 2008 ) , consuming all these papers is still a laborious job , where authors must read every source document carefully and locate the most relevant content cautiously .

As a solution in saving authors ' efforts , automatic related work summarization is essentially a topic-biased multi-document problem ( Cong and Kan , 2010 ) , which relies heavily on humanengineered features to retrieve snippets from the references .

Most recently , neural networks enable a data-driven architecture sequence-to-sequence ( seq2seq ) for natural language generation ( Bahdanau et al . , 2014 ( Bahdanau et al . , , 2016 ) ) , where an encoder reads a sequence of words / sentences into a context vector , from which a decoder yields a sequence of specific outputs .

Nonetheless , compared to scenarios like machine translation with an end-to-end nature , aligning a related work section to its source documents is far more challenging .

To address the summarization alignment , former studies try to apply an attention mechanism to measure the saliency / novelty of each candidate word / sentence ( Tan et al . , 2017 ) , with the aim of locating the most representative content to retain primary coverage .

However , toward summarizing a related work section , authors should be more creative when organizing text streams from the reference collection , where the selected content ought to highlight the topic bias of current work , rather than retell each reference in a compressed but balanced fashion .

This motivates us to introduce the contextual relevance and characterize the relationship among scientific publications accurately .

Generally speaking , for a pair of documents , a larger lexical overlap often implies a higher similarity in their research backgrounds .

Yet such a hypothesis is not always true when sampling content from multiple relevant topics .

Take " DSSM " 1 as an example , from viewpoint of the abstract similarity , those references investigating " Information Retrieval " , " Latent Semantic Model " or " Clickthrough Data Mining " could be of more importance in correlation and should be greatly sampled for the related work section .

But in reality , this article spends a bit larger chunk of texts ( about 58 % ) to elaborate " Deep Learning " during the literature review , which is quite difficult for machines to grasp the contextual relevance therein .

In addi-tion , other situations like emerging new concepts also suffer from the terminology variation or paraphrasing in varying degrees .

In this study , we utilize a heterogeneous bibliography graph to embody the relationship within a scalable scholarly database .

Over the recent past , there is a surge of interest in exploiting diverse relations to analyze bibliometrics , ranging from literature recommendation ( Yu et al . , 2015 ) to topic evolvement ( Jensen et al . , 2016 ) .

In a graphical sense , interconnected papers transfer the credit among each other directly / indirectly through various patterns , such as paper citation , author collaboration , keyword association and releasing on series of venues , which constitutes the graphic context for outlining concerned topics .

Unfortunately , a variety of edge types may pollute the information inquiry , where a slice of edges are not so important as the others on sampling content .

Meanwhile , most existing solutions in mining heterogeneous graphs depend on the human supervision , e.g. , hyperedge ( Bu et al . , 2010 ) and metapath ( Swami et al . , 2017 ) .

This is usually not easy to access due to the complexity of graph schemas .

First , we explore the edge-type usefulness distribution ( EUD ) on a heterogeneous bibliography graph , which enables the relationship discovery ( between any pair of papers ) for sampling the interested information .

Second , we develop a novel seq2seq summarizer for the automatic related work summarization , where a joint context-driven attention mechanism is proposed to measure the contextual relevance within both textual and graphic contexts .

Third , we conduct experiments on 8,080 papers with native related work sections , and experimental results show that our approach outperforms a typical seq2seq summarizer and five classical summarization baselines significantly .

In this paper , we highlight the contextual relevance for the automatic related work summarization , and analyze the graphic context to characterize the relationship among scientific publications accurately .

We develop a neural data-driven summarizer by leveraging the seq2seq paradigm , where a joint context-driven attention mechanism is proposed to measure the contextual relevance within full texts and a heterogeneous bibliography graph simultaneously .

Extensive experiments demonstrate the validity of the proposed attention mechanism , and the superiority of our approach over six representative summarization baselines .

In future work , an appealing direction is to organize the selected sentences in a logical fashion , e.g. , by leveraging a topic hierarchy tree to determine the arrangement of the related work section ( Cong and Kan , 2010 ) .

We also would like to take the citation sentences of each reference into consideration , which is another concise and universal data source for scientific summarization ( Chen and Hai , 2016 ;

At the end of this paper , we believe that extractive methods are by no means the final solutions for literature review generation due to plagiarism concerns , and we are going to put forward a fully abstractive version in further studies .


Obtaining human-like performance in NLP is often argued to require compositional generalisation .

Whether neural networks exhibit this ability is usually studied by training models on highly compositional synthetic data .

However , compositionality in natural language is much more complex than the rigid , arithmeticlike version such data adheres to , and artificial compositionality tests thus do not allow us to determine how neural models deal with more realistic forms of compositionality .

In this work , we re-instantiate three compositionality tests from the literature and reformulate them for neural machine translation ( NMT ) .

Our results highlight that : i ) unfavourably , models trained on more data are more compositional ; ii ) models are sometimes less compositional than expected , but sometimes more , exemplifying that different levels of compositionality are required , and models are not always able to modulate between them correctly ; iii ) some of the non-compositional behaviours are mistakes , whereas others reflect the natural variation in data .

Apart from an empirical study , our work is a call to action : we should rethink the evaluation of compositionality in neural networks and develop benchmarks using real data to evaluate compositionality on natural language , where composing meaning is not as straightforward as doing the math .

Although the successes of deep neural networks in natural language processing ( NLP ) are astounding and undeniable , they are still regularly criticised for lacking the powerful generalisation capacities that characterise human intelligence .

A frequently mentioned concept in such critiques is compositionality : the ability to build up the meaning of a complex expression by combining the meanings of its parts ( e.g. Partee , 1984 ) .

Compositionality is assumed to play an essential role in how humans understand language , but whether neural networks also exhibit this property has since long been a topic of vivid debate ( e.g. Fodor and Pylyshyn , 1988 ; Smolensky , 1990 ; Marcus , 2003 ; Nefdt , 2020 ) .

Studies about the compositional abilities of neural networks consider almost exclusively models trained on synthetic datasets , in which compositionality can be ensured and isolated ( e.g. Lake and Baroni , 2018 ; Hupkes et al . , 2020 ) .

In such tests , the interpretation of expressions is computed completely locally : every subpart is evaluated independently -without taking into account any external context -and the meaning of the whole expression is then formed by combining the meanings of its parts in a bottom-up fashion .

This protocol matches the type of compositionality observed in arithmetic : the meaning of ( 3 + 5 ) is always 8 , independent of the context it occurs in .

However , as exemplified by the sub-par performance of symbolic models that allow only strict , local protocols , compositionality in natural domains is far more intricate than this rigid , arithmeticlike variant of compositionality .

Natural language seems very compositional , but at the same time , it is riddled with cases that are difficult to interpret with a strictly local interpretation of compositionality .

Sometimes , the meaning of an expression does not derive from its parts ( e.g. for idioms ) , but the parts themselves are used compositionally in other contexts .

In other cases , the meaning of an expression does depend on its parts in a compositional way , but arriving at this meaning requires a more global approach because the meanings of the parts need to be disambiguated by information from elsewhere .

For instance , consider the meaning of homonyms ( " these dates are perfect for our dish / wedding " ) , potentially idiomatic expressions ( " the child kicked the bucket off the pavement " ) , or scope ambiguities ( " every human likes a cat " ) .

This paradoxical tension between local and global forms of compositionality inspired many debates on the compositionality of natural language .

Likewise , it impacts the evaluation of compositionality in NLP models .

On the one hand , local compositionality seems necessary for robust and reliable generalisation .

Yet , at the same time , global compositionality is needed to appropriately address the full complexity of language , which makes evaluating compositionality of state-of-the-art models ' in the wild ' a complicated endeavour .

In this work , we face this challenge head-on .

We concentrate on the domain of neural machine translation ( NMT ) , which is paradigmatically close to the tasks typically considered for compositionality tests , where the target represents the ' meaning ' of the input .

Furthermore , MT is an important domain of NLP , for which compositional generalisation is important to produce more robust translations and train adequate models for low-resource languages ( see , e.g. Chaabouni et al . , 2021 ) .

As an added advantage , compositionality is traditionally well studied and motivated for MT ( Rosetta , 1994 ; Janssen and Partee , 1997 ; Janssen , 1998 ) .

We reformulate three theoretically grounded tests from Hupkes et al . ( 2020 ) : systematicity , substitutivity and overgeneralisation .

Since accuracycommonly used in artificial compositionality testsis not a suitable evaluation metric for MT , we base our evaluations on the extent to which models behave consistently , rather than correctly .

In our tests for systematicity and substitutivity , we consider whether processing is local ; in our overgeneralisation test , we consider how models treat idioms that are assumed to require global processing .

Our results indicate that models often do not behave compositionally under the local interpretation , but exhibit behaviour that is too local in other cases .

In other words , models have the ability to process phrases both locally and globally but do not always correctly modulate between them .

We further show that some inconsistencies reflect variation in natural language , whereas others are true compositional mistakes , exemplifying the need for both local and global compositionality as well as illustrating the need for tests that encompass them both .

With our study , we contribute to ongoing questions about the compositional abilities of neural networks , and we provide nuance to the nature of this question when natural language is concerned : how local should the compositionality of models for natural language actually be ?

Aside from an empirical study , our work is also a call to action : we should rethink the evaluation of compositionality in neural networks and develop benchmarks using real data to evaluate compositionality on natural language , where composing meaning is not as straightforward as doing the math .

In conclusion , with this work , we contribute to the question of how compositional models trained on natural data are , and we argue that MT is a suitable and relevant testing ground to ask this question .

Focusing on the balance between local and global forms of compositionality , we formulate three different compositionality tests and discuss the issues and considerations that come up when considering compositionality in the context of natural data .

Our tests indicate that models show both local and global processing , but not necessarily for the right samples .

Furthermore , they underscore the difficulty of separating helpful and harmful types of non-compositionality , stressing the need to rethink the evaluation of compositionality using natural language , where composing meaning is not as straightforward as doing the math .

The Npeople Vtransitive the N sl people .