We present a technique which complements Hidden Markov Models by incorporating some lexicalized states representing syntactically uncommon words . ' Our approach examines the distribution of transitions , selects the uncommon words , and makes lexicalized states for the words .

We perfor'med a part-of-speech tagging experiment on the Brown corpus to evaluate the resultant language model and discovered that this technique improved the tagging accuracy by 0.21 % at the 95 % level of confidence .

Hidden Markov ' Models are widely used for statistical language modelling in various fields , e.g. , part-of-speech tagging or speech recognition ( Rabiner and Juang , 1986 ) .

The models are based on Markov assumptions , which make it possible to view the language prediction as a Markov process . ' In general , we make the firstorder Markov ass'umptions that the current tag is only dependant on the previous tag and that the current word is only dependant on the current tag .

These are very ' strong ' assumptions , so that the first-order Hidden Markov Models have the advantage of drastically reducing the number of its parameters .

On the other hand , the assumptions restrict the model from utilizing enough constraints provided by the local context and the resultant model consults only a single category ' as the contex .

A lot of effort has been devoted in the past to make up for the insufficient contextual information of the first-order probabilistic model .

The second order Hidden Markov Models with " The research underlying this paper was supported t ) 3 " research grants fl'om Korea Science and Engineering Foundation . appropriate smoothing techniques show better performance than the first order models and is considered a state-of-the-art technique ( Merialdo , 1994 ; Brants , 1996 ) .

The complexity of the model is however relatively very high considering the small improvement of the performance .

Garside describes IDIOMTAG ( Garside et al . , 1987 ) which is a component of a part-ofspeech tagging system named CLAWS .

ID-IOMTAG serves as a front-end to the tagger and modifies some initially assigned tags in order to reduce the amount of ambiguity to be dealt with by the tagger .

IDIOMTAG can look at any combination of words and tags , with or without intervening words .

By using the IDIOMTAG , CLAWS system improved tagging accuracy from 94 % to 96 - 97 % .

However , the manual-intensive process of producing idiom tags is very expensive although IDIOMTAG proved fruitful .

Kupiec ( Kupiec , 1992 ) describes a technique of augmenting the Hidden Markov Models for part-of-speech tagging by the use of networks .

Besides the original states representing each part-of-speech , the network contains additional states to reduce the noun / adjective confusion , and to extend the context for predicting past participles from preceding auxiliary verbs when they are separated by adverbs .

By using these additional states , the tagging system improved the accuracy from 95.7 % to 96.0 % .

However , the additional context is chosen by analyzing the tagging errors manually .

An automatic refining technique for Hidden Markov Models has been proposed by Brants ( Brants , 1996 ) .

It starts with some initial first order Markov Model .

Some states of the model are selected to be split or merged to take into account their predecessors .

As a result , each of new states represents a extended context .

With this technique , Brants reported a performance cquivalent to the second order Hidden Markov Models .

In this paper , we present an automatic refining technique for statistical language models .

First , we examine the distribution of transitions of lexicalized categories .

Next , we break out the uncommon ones from their categories and make new states for them .

All processes are automated and the user has only to determine the extent of the breaking-out .

In this paper , we present a method for complementing the Hidden Markov Models .

With this method , we lexicalize the Hidden Markov Model seletively and automatically by examining the transition distribution of each state relating to certain words .

Experimental results showed that the selective lexicalization improved the tagging accurary from about 95.79 % to about 96.00 % .

Using normal tests for statistical significance we found that the improvement is significant at the 95 % level of confidence .

Tile cost for this imt~rovenmnt is minimal .

The resulting network contains 210 additional lexicalized states which are found automatically .

Moreover , the lexicalization will not decrease the tagging speed 2 , because the lexicalized states and their corresponding original states are exclusive in our lexicalized network , and thus the rate of ambiguity is not increased even if the lexicalized states are included .

Our approach leaves much room for improvement .

We have so far considered only the outgoing transitions from the target states .

As a result , we have discriminated only the words with right-associativity .

We could also discriminate the words with left-associativity by examining the incoming transitions to the state .

Furthermore , we could extend the context by using the second-order context as represented in Figure l(c ) .

We believe that the same technique presented in this paper could be applied to the proposed extensions .

Hidden Markov Models used for language modeling are limited by the Markov assumptions and existing workarounds either make models complex or require expensive manual efforts.

They propose to complement Hidden Markov Models by first examining the transition distribution and breaking out new lexicalized states for uncommon words.

The proposed model improves on a part-of-speech tagging task from Brown corpus by finding 210 lexicalized states automatically while keeping the additional cost minimal.


An ideal summarization system should produce summaries that have high content coverage and linguistic quality .

Many state-ofthe-art summarization systems focus on content coverage by extracting content-dense sentences from source articles .

A current research focus is to process these sentences so that they read fluently as a whole .

The current AE-SOP task encourages research on evaluating summaries on content , readability , and overall responsiveness .

In this work , we adapt a machine translation metric to measure content coverage , apply an enhanced discourse coherence model to evaluate summary readability , and combine both in a trained regression model to evaluate overall responsiveness .

The results show significantly improved performance over AESOP 2011 submitted metrics .

Research and development on automatic and manual evaluation of summarization systems have been mainly focused on content coverage ( Lin and Hovy , 2003 ; Nenkova and Passonneau , 2004 ; Hovy et al . , 2006 ; Zhou et al . , 2006 ) .

However , users may still find it difficult to read such high-content coverage summaries as they lack fluency .

To promote research on automatic evaluation of summary readability , the Text Analysis Conference ( TAC ) ( Owczarzak and Dang , 2011 ) introduced a new subtask on readability to its Automatically Evaluating Summaries of Peers ( AESOP ) task .

Most of the state-of-the-art summarization systems ( Ng et al . , 2011 ; Zhang et al . , 2011 ; Conroy et al . , 2011 ) are extraction-based .

They extract the most content-dense sentences from source articles .

If no post-processing is performed to the generated summaries , the presentation of the extracted sentences may confuse readers .

Knott ( 1996 ) argued that when the sentences of a text are randomly ordered , the text becomes difficult to understand , as its discourse structure is disturbed .

Lin et al .

( 2011 ) validated this argument by using a trained model to differentiate an original text from a randomlyordered permutation of its sentences by looking at their discourse structures .

This

leads us to believe that we can apply such discourse models to evaluate the readability of extract-based summaries .

We will discuss the application of Lin et al . 's discourse coherence model to evaluate readability of machine generated summaries .

We also introduce two new feature sources to enhance the model with hierarchical and Explicit / Non-Explicit information , and demonstrate that they improve the original model .

There are parallels between evaluations of machine translation ( MT ) and summarization with respect to textual content .

For instance , the widely used ROUGE ( Lin and Hovy , 2003 ) metrics are influenced by BLEU ( Papineni et al . , 2002 ) : both look at surface n-gram overlap for content coverage .

Motivated by this , we will adapt a state-of-theart , linear programming-based MT evaluation metric , TESLA ( Liu et al . , 2010 ) , to evaluate the content coverage of summaries .

TAC 's overall responsiveness metric evaluates the quality of a summary with regard to both its content and readability .

Given this , we combine our two component coherence and content models into an SVM-trained regression model as our surrogate to overall responsiveness .

Our experiments show that the coherence model significantly outperforms all AESOP 2011 submissions on both initial and update tasks , while the adapted MT evaluation metric and the combined model significantly outperform all submissions on the initial task .

To the best of our knowledge , this is the first work that applies a discourse coherence model to measure the readability of summaries in the AESOP task .

2 Nenkova and Passonneau ( 2004 ) proposed a manual evaluation method that was based on the idea that there is no single best model summary for a collection of documents .

Human annotators construct a pyramid to capture important Summarization Content Units ( SCUs ) and their weights , which is used to evaluate machine generated summaries .

Lin and Hovy ( 2003 ) introduced an automatic summarization

metric , called ROUGE , which was motivated by the MT evaluation metric , BLEU ( Papineni et al . , 2002 ) .

It automatically determines the content quality of a summary by comparing it to the model summaries and counting the overlapping n-gram units .

Two configurations -ROUGE-2 , which counts bigram overlaps , and ROUGE-SU4 , which counts unigram and bigram overlaps in a word window of four -have been found to correlate well with human evaluations .

Hovy et al .

( 2006 ) pointed out that automated methods such as ROUGE , which match fixed length n-grams , face two problems of tuning the appropriate fragment lengths and matching them properly .

They introduced an method that makes use of small units of content , called Basic Elements ( BEs ) .

Their method automatically segments a text into BEs , matches similar BEs , and finally scores them .

Both ROUGE and BE have been implemented and included in the ROUGE / BE toolkit 1 , which has been used as the default evaluation tool in the summarization track in the Document Un-1 http://berouge.com / default.aspx derstanding Conference ( DUC ) and Text Analysis Conference ( TAC ) .

DUC and TAC also manually evaluated machine generated summaries by adopting the Pyramid method .

Besides evaluating with ROUGE / BE and Pyramid , DUC and TAC also asked human judges to score every candidate summary with regard to its content , readability , and overall responsiveness .

DUC and TAC defined linguistic quality to cover several aspects : grammaticality , non-redundancy , referential clarity , focus , and structure / coherence .

Recently , Pitler et al .

( 2010 ) conducted on various metrics designed to capture these aspects .

Their experimental results on DUC 2006 and 2007 show that grammaticality can be measured by a set of syntactic features , while the last three aspects are best evaluated by local coherence .

Conroy and Dang ( 2008 ) combined two manual linguistic scores -grammaticality and focus -with various ROUGE / BE metrics , and showed this helps better predict the responsiveness of the summarizers .

Since 2009 , TAC introduced the task of Automatically Evaluating Summaries of Peers ( AESOP ) .

AESOP 2009 and 2010 focused on two summary qualities : content and overall responsiveness .

Summary content is measured by comparing the output of an automatic metric with the manual Pyramid score .

Overall responsiveness measures a combination of content and linguistic quality .

In AESOP 2011 ( Owczarzak and Dang , 2011 ) , automatic metrics are also evaluated for their ability to assess summary readability , i.e. , to measure how linguistically readable a machine generated summary is .

Submitted metrics that perform consistently well on the three aspects include Giannakopoulos and Karkaletsis ( 2011 ) , Conroy et al .

( 2011 ) , and de Oliveira ( 2011 ) .

Giannakopoulos and Karkaletsis ( 2011 ) created two character-based n-gram graph representations for both the model and candidate summaries , and applied graph matching algorithm to assess their similarity .

Conroy et al .

( 2011 ) extended the model in ( Conroy and Dang , 2008 ) to include shallow linguistic features such as term overlap , redundancy , and term and sentence entropy . de Oliveira ( 2011 ) modeled the similarity between the model and candidate summaries as a maximum bipartite matching problem , where the two summaries are represented as two sets of nodes and precision and recall are cal- culated from the matched edges .

However , none of the AESOP metrics currently apply deep linguistic analysis , which includes discourse analysis .

Motivated by the parallels between summarization and MT , we will adapt a state-ofthe-art MT evaluation metric to measure summary content quality .

To apply deep linguistic analysis , we also enhance an existing discourse coherence model to evaluate summary readability .

We focus on metrics that measure the average quality of machine summarizers , i.e. , metrics that can rank a set of machine summarizers correctly ( human summarizers are not included in the list ) .

We proposed TESLA-S by adapting an MT metric to measure summary content coverage , and introduced DICOMER by applying a dis- course coherence model with newly introduced features to evaluate summary readability .

We combined these two metrics in the CREMER metric -an SVM-trained regression model -for automatic summarization overall responsiveness .

Experimental results on AESOP 2011 show that DICOMER significantly outperforms all submitted metrics on both initial and update tasks with large gaps , while TESLA-S and CREMER significantly outperform all metrics on the initial task.3

Existing evaluation metrics for summarization models focus on content coverage while fluency also impacts readability.

They propose to use a machine translation metric for content coverage, a discourse coherence model for readability, and combine them for overall responsiveness.

The proposed metric significantly improves over all the metrics submitted to AESOP 2011.


In this paper , we define the task of gender rewriting in contexts involving two users ( I and/or You ) -first and second grammatical persons with independent grammatical gender preferences .

We focus on Arabic , a gendermarking morphologically rich language .

We develop a multi-step system that combines the positive aspects of both rule-based and neural rewriting models .

Our results successfully demonstrate the viability of this approach on a recently created corpus for Arabic gender rewriting , achieving 88.42 M 2 F 0.5 on a blind test set .

Our proposed system improves over previous work on the first-person-only version of this task , by 3.05 absolute increase in M 2 F 0.5 .

We demonstrate a use case of our gender rewriting system by using it to post-edit the output of a commercial MT system to provide personalized outputs based on the users ' grammatical gender preferences .

We make our code , data , and pretrained models publicly available .

1 Gender bias is a fundamental problem in natural language processing ( NLP ) and it has been receiving an increasing attention across a variety of core tasks such as machine translation ( MT ) , co-reference resolution , and dialogue systems .

Research has shown that NLP systems have the ability to embed and amplify gender bias ( Sun et al . , 2019 ) , which not only degrades users ' experiences but also creates representational harm ( Blodgett et al . , 2020 ) .

The embedded bias within NLP systems is usually attributed to training models on biased data that reflects the social inequalities of the world we live in .

However , even the most balanced of models can still exhibit and amplify bias if they are designed to produce a single text output without taking their users ' gender preferences into consideration .

Therefore , to provide the correct user-aware output , NLP systems should be designed to produce outputs that are as gender-specific as the users information they have access to .

Users information could be either embedded as part of the input or provided externally by the users themselves .

In cases where this information is unavailable to the system , generating all gender-specific forms or a gender-neutral form is more appropriate .

Producing user-aware outputs becomes more challenging for systems targeting multi-user contexts ( first and second persons , with independent grammatical gender preferences ) , particularly when dealing with gender-marking morphologically rich languages .

In this paper , we define the task of gender rewriting in contexts involving two users ( I and/or You ) -first and second grammatical persons with independent grammatical gender preferences and we focus on Arabic , a gender-marking morphologically rich language .

The main contributions of our work are as follows : 1 .

We introduce a multi-step gender rewriting system that combines the positive aspects of rule-based and neural models .

2 .

We demonstrate our approach 's effectiveness by establishing a strong benchmark on a publicly available multi-user Arabic gender rewriting corpus .

3 .

We show that our best system yields state-ofthe-art results on the first-person-only version of this task , beating previous work .

4 .

We demonstrate a use case of our system by post-editing the output of an MT system to match users ' grammatical gender preferences .

This paper is organized as follows .

We first discuss

( § 2 ) as well as relevant Arabic linguistic facts ( § 3 ) .

We then define the gender rewriting task in § 4 and describe the data we use and the gender rewriting model we build in § 5 and § 6 .

Lastly , we present our

( § 7 ) and results ( § 8) and conclude in § 9 .

We defined the task of gender rewriting in contexts involving two users ( I and/or You ) , and developed a multi-step system that combines the positive aspects of both rule-based and neural rewriting models .

Our best models establish the benchmark for this newly defined task and the SOTA for a previously defined first person version of it .

We further demonstrated a use case of our gender rewriting system by post-editing the output of a commercial MT system to provide personalized outputs based on the users ' grammatical gender preferences .

In future work , we plan to explore the use of other pretrained models , and to work on the problem of gender rewriting in other languages and dialectal varieties .

Systems which can produce user-aware outputs to avoid embedding and amplifying gender bias remain a challenge, particularly with gender-marking morphologically rich languages.

They define the task of gender rewriting in contexts involving two users and propose a multi-step system that combines the rule-based and neural models.

The proposed approach is shown to be a strong baseline on the Arabic gender rewriting corpus and outperforms the state-of-the-art of the first-person-only version.


The de-facto standard decoding method for semantic parsing in recent years has been to autoregressively decode the abstract syntax tree of the target program using a top-down depthfirst traversal .

In this work , we propose an alternative approach : a Semi-autoregressive Bottom-up Parser ( SMBOP ) that constructs at decoding step t the top-K sub-trees of height ≤ t.

Our parser enjoys several benefits compared to top-down autoregressive parsing .

From an efficiency perspective , bottom-up parsing allows to decode all sub-trees of a certain height in parallel , leading to logarithmic runtime complexity rather than linear .

From a modeling perspective , a bottom-up parser learns representations for meaningful semantic sub-programs at each step , rather than for semantically-vacuous partial trees .

We apply SMBOP on SPIDER , a challenging zero-shot semantic parsing benchmark , and show that SMBOP leads to a 2.2x speed-up in decoding time and a ∼5x speed-up in training time , compared to a semantic parser that uses autoregressive decoding .

SMBOP obtains 71.1 denotation accuracy on SPIDER , establishing a new state-of-the-art , and 69.5 exact match , comparable to the 69.6 exact match of the autoregressive RAT-SQL+GRAPPA .

Semantic parsing , the task of mapping natural language utterances into programs ( Zelle and Mooney , 1996 ; Zettlemoyer and Collins , 2005 ; Clarke et al . ; Liang et al . , 2011 ) , has converged in recent years on a standard encoder-decoder architecture .

Recently , meaningful advances emerged on the encoder side , including developments in Transformer-based architectures ( Wang et al . , 2020a ) and new pretraining techniques ( Yin et al . , 2020 ; Herzig et al . , 2020 ; Yu et al . , 2020 ; Deng et al . , 2020 ; Shi et al . , 2021 ) .

Conversely , the decoder has remained roughly constant for years , where the abstract syntax tree of the target program is autoregressively decoded in a top-down manner ( Yin and Neubig , 2017 ; Krishnamurthy et al . , 2017 ; Rabinovich et al . , 2017 ) .

Bottom-up decoding in semantic parsing has received little attention ( Cheng et al . , 2019 ; Odena et al . , 2020 ) .

In this work , we propose a bottom-up semantic parser , and demonstrate that equipped with recent developments in Transformer-based ( Vaswani et al . , 2017 ) architectures , it offers several advantages .

From an efficiency perspective , bottom-up parsing can naturally be done semiautoregressively : at each decoding step t , the parser generates in parallel the top-K program sub-trees of depth ≤ t ( akin to beam search ) .

This leads to runtime complexity that is logarithmic in the tree size , rather than linear , contributing to the rocketing interest in efficient and greener artificial intelligence technologies ( Schwartz et al . , 2020 ) .

From a modeling perspective , neural bottom-up parsing provides learned representations for meaningful ( and executable ) sub-programs , which are sub-trees computed during the search procedure , in contrast to top-down parsing , where hidden states represent partial trees without clear semantics .

Figure 1 illustrates a single decoding step of our parser .

Given a beam Z t with K = 4 trees of height t ( blue vectors ) , we use cross-attention to contextualize the trees with information from the input question ( orange ) .

Then , we score the frontier , that is , the set of all trees of height t + 1 that can be constructed using a grammar from the current beam , and the top-K trees are kept ( purple ) .

Last , a representation for each of the new K trees is generated and placed in the new beam Z t+1 .

After T decoding steps , the parser returns the highest-scoring tree in Z T that corresponds to a full program .

Because we have gold trees at training time , the entire model is trained jointly using maximum likelihood .

We evaluate our model , SMBOP1 ( SeMiautoregressive Bottom-up semantic Parser ) , on SPI- What are the names of actors over 60 ?

Prune frontier In this work we present the first semiautoregressive bottom-up semantic parser that enjoys logarithmic theoretical runtime , and show that it leads to a 2.2x speed-up in decoding and ∼5x faster taining , while maintaining state-of-the-art performance .

Our work shows that bottom-up parsing , where the model learns representations for semantically meaningful sub-trees is a promising research direction , that can contribute in the future to setups such as contextual semantic parsing , where sub-trees often repeat , and can enjoy the benefits of execution at training time .

Future work can also leverage work on learning tree representations ( Shiv and Quirk , 2019 ) to further improve parser performance .

Although bottom-up approaches for semantic parsing can theoretically achieve efficiency and produce meaningful sub-trees, they have not been studied well.

They propose a transformer-based semi-autoregressive bottom-up parser for semantic parsing and couple it with a state-of-the-art encoder.

The proposed bottom-up parser can substantially improve over existing models in performance and also in decoding speed.


Recent advances in language and vision push forward the research of captioning a single image to describing visual differences between image pairs .

Suppose there are two images , I 1 and I 2 , and the task is to generate a description W 1,2 comparing them , existing

directly model ⟨I 1 , I 2 ⟩ → W 1,2 mapping without the semantic understanding of individuals .

In this paper , we introduce a Learningto-Compare ( L2C ) model , which learns to understand the semantic structures of these two images and compare them while learning to describe each one .

We demonstrate that L2C benefits from a comparison between explicit semantic representations and singleimage captions , and generalizes better on the new testing image pairs .

It outperforms the baseline on both automatic

and human evaluation for the Birds-to-Words dataset .

The task of generating textual descriptions of images tests a machine 's ability to understand visual data and interpret it in natural language .

It is a fundamental research problem lying at the intersection of natural language processing , computer vision , and cognitive science .

For example , single-image captioning ( Farhadi et al . , 2010 ; Kulkarni et al . , 2013 ; Vinyals et al . , 2015 ; Xu et al . , 2015 ) has been extensively studied .

Recently , a new intriguing task , visual comparison , along with several benchmarks ( Jhamtani and Berg-Kirkpatrick , 2018 ; Tan et al . , 2019 ; Park et al . , 2019 ; Forbes et al . , 2019 ) has drawn increasing attention in the community .

To complete the task and generate comparative descriptions , a machine should understand the visual differences between a pair of images ( see Figure 1 ) .

Previous methods ( Jhamtani and Berg-Kirkpatrick , 2018 ) as the ResNet features ( He et al . , 2016 ) as a whole , and build end-to-end neural networks to predict the description of visual comparison directly .

In contrast , humans can easily reason about the visual components of a single image and describe the visual differences between two images based on their semantic understanding of each one .

Humans do not need to look at thousands of image pairs to describe the difference of new image pairs , as they can leverage their understanding of single images for visual comparison .

Therefore , we believe that visual differences should be learned by understanding and comparing every single image 's semantic representation .

A most recent work ( Zhang et al . , 2020 ) conceptually supports this argument , where they show that low-level ResNet visual features lead to poor generalization in vision-and-language navigation , and high-level semantic segmentation helps the agent Image I 1 Image I 2 V 1 V 2 Relation- enhanced features V g 1 V g 2 LSTM h t In this paper , we present a learning-to-compare framework for generating visual comparisons .

Our segmentation encoder with semantic pooling and graph reasoning could construct structured image representations .

We also show that learning to describe visual differences benefits from understanding the semantics of each image .

To generate a different description between two images, in contrast to how humans reason, existing models do not understand the semantics of each image.

They propose a Learning-to-Compare model that can consider the semantic structures of each image using graph convolutional networks before comparing them and generating a description.

The proposed graph-based learning-to-compare model outperforms the baseline models using automatic and human evaluations and also generalizes better.


In this paper , we propose a neural network model for graph-based dependency parsing which utilizes Bidirectional LSTM ( BLSTM ) to capture richer contextual information instead of using high-order factorization , and enable our model to use much fewer features than previous work .

In addition , we propose an effective way to learn sentence segment embedding on sentence-level based on an extra forward LSTM network .

Although

our model uses only first-order factorization , experiments on English Peen Treebank and Chinese Penn Treebank show that our model could be competitive with previous higher-order graph-based dependency parsing models and state-of-the-art models .

Dependency parsing is a fundamental task for language processing which has been investigated for decades .

It has been applied in a wide range of applications such as information extraction and machine translation .

Among a variety of dependency parsing models , graph-based models are attractive for their ability of scoring the parsing decisions on a whole-tree basis .

Typical graph-based models factor the dependency tree into subgraphs , including single arcs ( McDonald et al . , 2005 ) , sibling or grandparent arcs ( McDonald and Pereira , 2006 ; Carreras , 2007 ) or higher-order substructures ( Koo and Collins , 2010 ; Ma and Zhao , 2012 ) and then score the whole tree by summing scores of the subgraphs .

In these models , subgraphs are usually represented as high-dimensional feature vectors which are then fed into a linear model to learn the feature weights .

However , conventional graph-based models heavily rely on feature engineering and their performance is restricted by the design of features .

In addition , standard decoding algorithm ( Eisner , 2000 ) only works for the first-order model which limits the scope of feature selection .

To incorporate high-order features , Eisner algorithm must be somehow extended or modified , which is usually done at high cost in terms of efficiency .

The fourth-order graph-based model ( Ma and Zhao , 2012 ) , which seems the highest-order model so far to our knowledge , requires O(n 5 ) time and O(n 4 ) space .

Due to the high computational cost , highorder models are normally restricted to producing only unlabeled parses to avoid extra cost introduced by inclusion of arc-labels into the parse trees .

To alleviate the burden of feature engineering , Pei et al .

( 2015 ) presented an effective neural network model for graph-based dependency parsing .

They only use atomic features such as word unigrams and POS tag unigrams and leave the model to automatically learn the feature combinations .

However , their model requires many atomic features and still relies on high-order factorization strategy to further improve the accuracy .

Different from previous work , we propose an LSTM-based dependency parsing model in this paper and aim to use LSTM network to capture richer contextual information to support parsing decisions , instead of adopting a high-order factorization .

The main advantages of our model are as follows : • By introducing Bidirectional LSTM , our model shows strong ability to capture potential long range contextual information and exhibits improved accuracy in recovering long distance dependencies .

It is different to previous work in which a similar effect is usually achieved by high-order factorization .

More-over , our model also eliminates the need for setting feature selection windows and reduces the number of features to a minimum level . • We propose an LSTM-based sentence segment embedding named LSTM-Minus , in which distributed representation of sentence segment is learned by using subtraction between LSTM hidden vectors .

shows this further enhances our model 's ability to access to sentence-level information . • Last but important , our model is a first-order model using standard Eisner algorithm for decoding , the computational cost remains at the lowest level among graph-based models .

Our model does not trade-off efficiency for accuracy .

We evaluate our model on the English Penn Treebank and Chinese Penn Treebank , show that our model achieves competitive parsing accuracy compared with conventional high-order models , however , with a much lower computational cost .

In this paper , we propose an LSTM-based neural network model for graph-based dependency parsing .

Utilizing Bidirectional LSTM and segment embeddings learned by LSTM-Minus allows our model access to sentence-level information , making our model more accurate in recovering longdistance dependencies with only first-order factorization . on PTB and CTB show that our model could be competitive with conventional high-order models with a faster speed .

Neural graph-based dependency parsers can alleviate feature engineering but require atomic features and use a high-order factorization strategy which is incompatible with standard decoding algorithms.

They propose to use Bidirectional LSTM to capture contextual information instead of high-order factorization and learn sentence segment embeddings on the sentence-level by another LSTM.

While the proposed model uses first-order factorization, it performs competitively with state-of-the-art models on English and Chinese Penn Treebank by capturing long range contextual information.


Text mining and data harvesting algorithms have become popular in the computational linguistics community .

They employ patterns that specify the kind of information to be harvested , and usually bootstrap either the pattern learning or the term harvesting process ( or both ) in a recursive cycle , using data learned in one step to generate more seeds for the next .

They therefore treat the source text corpus as a network , in which words are the nodes and relations linking them are the edges .

The results of computational network analysis , especially from the world wide web , are thus applicable .

Surprisingly , these results have not yet been broadly introduced into the computational linguistics community .

In this paper we show how various results apply to text mining , how they explain some previously observed phenomena , and how they can be helpful for computational linguistics applications .

Text mining / harvesting algorithms have been applied in recent years for various uses , including learning of semantic constraints for verb participants ( Lin and Pantel , 2002 ) related pairs in various relations , such as part-whole ( Girju et al . , 2003 ) , cause ( Pantel and Pennacchiotti , 2006 ) , and other typical information extraction relations , large collections of entities ( Soderland et al . , 1999 ; Etzioni et al . , 2005 ) , features of objects ( Pasca , 2004 ) and ontologies ( Carlson et al . , 2010 ) .

They generally start with one or more seed terms and employ patterns that specify the desired information as it relates to the seed(s ) .

Several approaches have been developed specifically for learning patterns , including guided pattern collection with manual filtering ( Riloff and Shepherd , 1997 ) automated surface-level pattern induction ( Agichtein and Gravano , 2000 ; Ravichandran and Hovy , 2002 ) probabilistic methods for taxonomy relation learning ( Snow et al . , 2005 ) and kernel methods for relation learning ( Zelenko et al . , 2003 ) .

Generally , the harvesting procedure is recursive , in which data ( terms or patterns ) gathered in one step of a cycle are used as seeds in the following step , to gather more terms or patterns .

This method treats the source text as a graph or network , consisting of terms ( words ) as nodes and inter-term relations as edges .

Each relation type induces a different network1 .

Text mining is a process of network traversal , and faces the standard problems of handling cycles , ranking search alternatives , estimating yield maxima , etc .

The computational properties of large networks and large network traversal have been studied intensively ( Sabidussi , 1966 ; Freeman , 1979 ; Watts and Strogatz , 1998 ) and especially , over the past years , in the context of the world wide web ( Page et al . , 1999 ; Broder et al . , 2000 ; Kleinberg and Lawrence , 2001 ; Li et al . , 2005 ; Clauset et al . , 2009 ) .

Surprisingly , except in ( Talukdar and Pereira , 2010 ) , this work has not yet been related to text mining research in the computational linguistics community .

The work is , however , relevant in at least two ways .

It sometimes explains why text mining algo-rithms have the limitations and thresholds that are empirically found ( or suspected ) , and it may suggest ways to improve text mining algorithms for some applications .

In Section 2 , we review some related work .

In Section 3 we describe the general harvesting procedure , and follow with an examination of the various statistical properties of implicit semantic networks in Section 4 , using our implemented harvester to provide illustrative statistics .

In Section 5 we discuss implications for computational linguistics research .

In this paper we describe the implicit ' hidden ' semantic network graph structure induced over the text of the web and other sources by the semantic relations people use in sentences .

We describe how term harvesting patterns whose seed terms are harvested and then applied recursively can be used to discover these semantic term networks .

Although these networks differ considerably from the web in relation density , type , and network size , we show , somewhat surprisingly , that the same power-law , smallworld effect , transitivity , and most other characteristics that apply to the web 's hyperlinked network structure hold also for the implicit semantic term graphs-certainly for the semantic relations and languages we have studied , and most probably for almost all semantic relations and human languages .

This rather interesting observation leads us to surmise that the hyperlinks people create in the web are of essentially the same type as the semantic relations people use in normal sentences , and that they form an extension of normal language that was not needed before because people did not have the ability within the span of a single sentence to ' embed ' structures larger than a clause-certainly not a whole other page 's worth of information .

The principal exception is the academic citation reference ( lexicalized as " see " ) , which is not used in modern webpages .

Rather , the ' lexicalization ' now used is a formatting convention : the hyperlink is colored and often underlined , facilities offered by computer screens but not available to speech or easy in traditional typesetting .

Corpora can be seen as a network in text mining and data harvesting however how recent network analysis methods would perform is unknown.

They study if recent network analysis tools can provide insights for computational linguistics by connecting two hidden semantic network graph structures in texts.

They show that there are similarities between networks on the Web and ones induced from texts such as in power-law, small-world effect and transitivity.


Question answering ( QA ) systems for large document collections typically use pipelines that ( i ) retrieve possibly relevant documents , ( ii ) re-rank them , ( iii ) rank paragraphs or other snippets of the top-ranked documents , and ( iv ) select spans of the top-ranked snippets as exact answers .

Pipelines are conceptually simple , but errors propagate from one component to the next , without later components being able to revise earlier decisions .

We present an architecture for joint document and snippet ranking , the two middle stages , which leverages the intuition that relevant documents have good snippets and good snippets come from relevant documents .

The architecture is general and can be used with any neural text relevance ranker .

We experiment with two main instantiations of the architecture , based on POSIT-DRMM ( PDRMM ) and a BERT-based ranker .

Experiments on biomedical data from BIOASQ show that our joint models vastly outperform the pipelines in snippet retrieval , the main goal for QA , with fewer trainable parameters , also remaining competitive in document retrieval .

Furthermore , our joint PDRMM-based model is competitive with BERT-based models , despite using orders of magnitude fewer parameters .

These claims are also supported by human evaluation on two test batches of BIOASQ .

To test our key findings on another dataset , we modified the Natural Questions dataset so that it can also be used for document and snippet retrieval .

Our joint PDRMM-based model again outperforms the corresponding pipeline in snippet retrieval on the modified Natural Questions dataset , even though it performs worse than the pipeline in document retrieval .

We make our code and the modified Natural Questions dataset publicly available .

Question answering ( QA ) systems that search large document collections ( Voorhees , 2001 ; Tsatsaro-nis et al . , 2015 ; Chen et al . , 2017 ) typically use pipelines operating at gradually finer text granularities .

A fully-fledged pipeline includes components that ( i ) retrieve possibly relevant documents typically using conventional information retrieval ( IR ) ; ( ii ) re-rank the retrieved documents employing a computationally more expensive document ranker ; ( iii ) rank the passages , sentences , or other ' snippets ' of the top-ranked documents ; and ( iv ) select spans of the top-ranked snippets as ' exact ' answers .

Recently , stages ( ii)-(iv ) are often pipelined neural models , trained individually ( Hui et al . , 2017 ; Pang et al . , 2017 ; Lee et al . , 2018 ; McDonald et al . , 2018 ; Pandey et al . , 2019 ; Mackenzie et al . , 2020 ; Sekulić et al . , 2020 ) .

Although pipelines are conceptually simple , errors propagate from one component to the next ( Hosein et al . , 2019 ) , without later components being able to revise earlier decisions .

For example , once a document has been assigned a low relevance score , finding a particularly relevant snippet can not change the document 's score .

We propose an architecture for joint document and snippet ranking , i.e. , stages ( ii ) and ( iii ) , which leverages the intuition that relevant documents have good snippets and good snippets come from relevant documents .

We note that modern web search engines display the most relevant snippets of the top-ranked documents to help users quickly identify truly relevant documents and answers ( Sultan et al . , 2016 ; Xu et al . , 2019 ; Yang et al . , 2019a ) .

The top-ranked snippets can also be used as a starting point for multi-document query-focused summarization , as in the BIOASQ challenge ( Tsatsaronis et al . , 2015 ) .

Hence ,

that identify good snippets are useful in several other applications , apart from QA .

We also note that many neural models for stage ( iv ) have been proposed , often called QA or Machine Reading Comprehension ( MRC ) models ( Kadlec et al . , 2016 ; Cui et al . , 2017 ; Zhang et al . , 2020 ) , but they typically search for answers only in a particular , usually paragraph-sized snippet , which is given per question .

For QA systems that search large document collections , stages ( ii ) and ( iii ) are also important , if not more important , but have been studied much less in recent years , and not in a single joint neural model .

The proposed joint architecture is general and can be used in conjunction with any neural text relevance ranker ( Mitra and Craswell , 2018 ) .

Given a query and N possibly relevant documents from stage ( i ) , the neural text relevance ranker scores all the snippets of the N documents .

Additional neural layers re-compute the score ( ranking ) of each document from the scores of its snippets .

Other layers then revise the scores of the snippets taking into account the new scores of the documents .

The entire model is trained to jointly predict document and snippet relevance scores .

We

with two main instantiations of the proposed architecture , using POSIT-DRMM ( McDonald et al . , 2018 ) , hereafter called PDRMM , as the neural text ranker , or a BERT-based ranker ( Devlin et al . , 2019 ) .

We show how both PDRMM and BERT can be used to score documents and snippets in pipelines , then how our architecture can turn them into models that jointly score documents and snippets .

Experimental results on biomedical data from BIOASQ ( Tsatsaronis et al . , 2015 ) show the joint models vastly outperform the corresponding pipelines in snippet extraction , with fewer trainable parameters .

Although our joint architecture is engineered to favor retrieving good snippets ( as a near-final stage of QA ) , results show that the joint models are also competitive in document retrieval .

We also show that our joint version of PDRMM , which has the fewest parameters of all models and does not use BERT , is competitive to BERT-based models , while also outperforming the best system of BIOASQ 6 ( Brokos et al . , 2018 ) in both document and snippet retrieval .

These claims are also supported by human on two test batches of BIOASQ 7 ( 2019 ) .

To test our key findings on another dataset , we modified Natural Questions ( Kwiatkowski et al . , 2019 ) , which only includes questions and answer spans from a single document , so that it can be used for document and snippet retrieval .

Again , our joint PDRMMbased model largely outperforms the corresponding pipeline in snippet retrieval on the modified Natural Questions , though it does not perform better than the pipeline in document retrieval , since the joint model is geared towards snippet retrieval , i.e. , even though it is forced to extract snippets from fewer relevant documents .

Finally , we show that all the neural pipelines and joint models we considered improve the BM25 ranking of traditional IR on both datasets .

We make our code and the modified Natural Questions publicly available.1 2 Methods Our contributions can be summarized as follows : ( 1 ) We proposed an architecture to jointly rank documents and snippets with respect to a question , two particularly important stages in QA for large document collections ; our architecture can be used with any neural text relevance model .

( 2 ) We instantiated the proposed architecture using a recent neural relevance model ( PDRMM ) and a BERTbased ranker .

( 3 ) Using biomedical data ( from BIOASQ ) , we showed that the two resulting joint models ( PDRMM-based and BERT-based ) vastly outperform the corresponding pipelines in snippet re-trieval , the main goal in QA for document collections , using fewer parameters , and also remaining competitive in document retrieval .

( 4 ) We showed that the joint model ( PDRMM-based ) that does not use BERT is competitive with BERT-based models , outperforming the best BIOASQ 6 system ; our joint models ( PDRMM-and BERT-based ) also outperformed all BIOASQ 7 competitors .

( 5 ) We provide a modified version of the Natural Questions dataset , suitable for document and snippet retrieval .

( 6 ) We showed that our joint PDRMM-based model also largely outperforms the corresponding pipeline on open-domain data ( Natural Questions ) in snippet retrieval , even though it performs worse than the pipeline in document retrieval .

( 7 ) We showed that all the neural pipelines and joint models we considered improve the traditional BM25 ranking on both datasets .

( 8) We make our code publicly available .

We hope to extend our models and datasets for stage ( iv ) , i.e. , to also identify exact answer spans within snippets ( paragraphs ) , similar to the answer spans of SQUAD ( Rajpurkar et al . , 2016 ( Rajpurkar et al . , , 2018 ) ) .

This would lead to a multi-granular retrieval task , where systems would have to retrieve relevant documents , relevant snippets , and exact answer spans from the relevant snippets .

BIOASQ already includes this multi-granular task , but exact answers are provided only for factoid questions and they are freely written by humans , as in MS-MARCO , with similar limitations .

Hence , appropriately modified versions of the BIOASQ datasets are needed .

Table 4 shows that further performance gains ( 6.80 to 7.85 document MAP , 15.42 to 17.34 snippet MAP ) are possible by tuning the weights of the two losses .

The best scores are obtained when using both the extra sentence and document features .

However , the model performs reasonably well even when one of the two types of extra features is removed , with the exception of λ snip = 10 .

The standard deviations of the MAP scores over the folds of the cross-validation indicate that the performance of the model is reasonably stable .

Pipeline systems for question answering systems for large document collections suffer from error propagation from one stage to the following.

They propose to jointly model document and snippet rankings in a generic way to make work with any neural text relevance rankers.

The proposed systems outperform existing BERT-based models by using much fewer parameters on biomedical data and also on the Natural Questions dataset.


Query expansion techniques generally select new query terms from a set of top ranked documents .

Although a user 's manual judgment of those documents would much help to select good expansion terms , it is difficult to get enough feedback from users in practical situations .

In this paper we propose a query expansion technique which performs well even if a user notifies just a relevant document and a non-relevant document .

In order to tackle this specific condition , we introduce two refinements to a well-known query expansion technique .

One is application of a transductive learning technique in order to increase relevant documents .

The other is a modified parameter estimation method which laps the predictions by multiple learning trials and try to differentiate the importance of candidate terms for expansion in relevant documents .

Experimental results show that our technique outperforms some traditional query expansion methods in several evaluation measures .

Query expansion is a simple but very useful technique to improve search performance by adding some terms to an initial query .

While many query expansion techniques have been proposed so far , a standard method of performing is to use relevance information from a user ( Ruthven , 2003 ) .

If we can use more relevant documents in query expansion , the likelihood of selecting query terms achieving high search improvement increases .

However it is impractical to expect enough relevance information .

Some researchers said that a user usually notifies few relevance feedback or nothing ( Dumais and et al . , 2003 ) .

In this paper we investigate the potential performance of query expansion under the condition that we can utilize little relevance information , especially we only know a relevant document and a nonrelevant document .

To overcome the lack of relevance information , we tentatively increase the number of relevant documents by a machine learning technique called Transductive Learning .

Compared with ordinal inductive learning approach , this learning technique works even if there is few training examples .

In our case , we can use many documents in a hit-list , however we know the relevancy of few documents .

When applying query expansion , we use those increased documents as if they were true relevant ones .

When applying the learning , there occurs some difficult problems of parameter settings .

We also try to provide a reasonable resolution for the problems and show the effectiveness of our proposed

in experiments .

The point of our query expansion method is that we focus on the availability of relevance information in practical situations .

There are several researches which deal with this problem .

Pseudo relevance feedback which assumes top n documents as relevant ones is one example .

This method is simple and relatively effective if a search engine returns a hit-list which contains a certain number of relative documents in the upper part .

However , unless this assumption holds , it usually gives a worse ranking than the initial search .

Thus several researchers propose some specific procedure to make pseudo feedback be effective ( Yu and et al , 2003 ; Lam-Adesina and Jones , 2001 ) .

In another way , Onoda ( Onoda et al . , 2004 ) tried to apply one-class SVM ( Support Vector Machine ) to relevance feedback .

Their purpose is to improve search performance by using only nonrelevant documents .

Though their motivation is similar to ours in terms of applying a machine learning method to complement the lack of relevance information , the assumption is somewhat different .

Our assumption is to utilizes manual but the minimum relevance judgment .

Transductive leaning has already been applied in the field of image retrieval ( He and et al . , 2004 ) .

In this research , they proposed a transductive called the manifold-ranking algorithm and showed its effectiveness by comparing with active learning based Support Vector Machine .

However , their setting of relevance judgment is not different from many other traditional researches .

They fix the total number of images that are marked by a user to 20 .

As we have already claimed , this setting is not practical because most users feel that 20 is too much for judgment .

We think none of research has not yet answered the question .

For relevance judgment , most of the researches have adopted either of the following settings .

One is the setting of " Enough relevant documents are available " , and the other is " No relevant document is available " .

In contrast to them , we adopt the setting of " Only one relevant document is available " .

Our aim is to achieve performance improvement with the minimum effort of judging relevancy of documents .

The reminder of this paper is structured as follows .

Section 2 describes two fundamental techniques for our query expansion method .

Section 3 explains a technique to complement the smallness of manual relevance judgment .

Section 4 introduces a whole procedure of our query expansion method step by step .

Section 5 shows empirical evidence of the effectiveness of our method compared with two traditional query expansion methods .

Section 6 investigates the experimental results more in detail .

Finally , Section 7 summarizes our findings .

In this paper we proposed a novel query expansion method which only use the minimum manual judgment .

To complement the lack of relevant documents , this method utilizes the SGT transductive learning algorithm to predict the relevancy of unjudged documents .

Since the performance of SGT much depends on an estimation of the fraction of relevant documents , we propose a method to sample some good fraction values .

We also propose a method to laps the predictions of multiple SGT trials with above sampled fraction values and try to differentiate the importance of candidate terms for expansion in relevant documents .

The experimental results showed our method outperforms other query expansion methods in the evaluations of several criteria .

Although existing works on query expansion assume some relevant documents at their disposal, it is impractical because users provide few or no feedback.

They consider an experimental setup where there is only one relevant document provided by a user and extend a transductive learning technique to compensate.

The proposed technique outperforms traditional query expansion methods in several evaluation metrics.


In this paper , we study differentiable neural architecture search ( NAS ) methods for natural language processing .

In particular , we improve differentiable architecture search by removing the softmax-local constraint .

Also , we apply differentiable NAS to named entity recognition ( NER ) .

It is the first time that differentiable NAS

are adopted in NLP tasks other than language modeling .

On both the PTB language modeling and CoNLL-2003 English NER data , our method outperforms strong baselines .

It achieves a new state-ofthe-art on the NER task .

Neural architecture search ( NAS ) has become popular recently in machine learning for their ability to find new models and to free researchers from the hard work of designing network architectures .

The earliest of these approaches use reinforcement learning ( RL ) to learn promising architectures in a discrete space ( Zoph and Le , 2016 ) , whereas others have successfully modeled the problem in a continuous manner ( Liu et al . , 2019 ; Xie et al . , 2019b ; Huang and Xiang , 2019 ) .

As an instance of the latter , differentiable architecture search ( DARTS ) employs continuous relaxation to architecture representation and makes gradient descent straightforwardly applicable to search .

This leads to an efficient search process that is orders of magnitude faster than the RL-based counterparts .

Like recent methods in NAS ( Xie and Yuille , 2017 ; Zoph and Le , 2016 ; Baker et al . , 2016 ) , DARTS represents networks as a directed acyclic graph for a given computation cell ( see Figure 1(a ) ) .

An edge between nodes performs a predefined operation to transform the input ( i.e. , tail ) Figure 1 : An overview of DARTS cell and our cell to the output ( i.e. , head ) .

For a continuous network space , DARTS uses the softmax trick to relax the categorical choice of edges to soft decisions .

Then , one can optimize over the graph using standard gradient descent .

The optimized network is inferred by choosing the edges with maximum weights in softmax .

However , DARTS is a " local " model because the softmax-based relaxation is imposed on each bundle of edges between two nodes .

This leads to a biased model in that edges coming from different nodes are not comparable .

Such a constraint limits the inference space to sub-graphs with one edge between each pair of nodes .

Also , the learned network might be redundant because every node has to receive edges from all predecessors no matter they are necessary or not .

This problem is similar to the bias problem in other graph-based models where local decisions make the model nonoptimal ( Lafferty et al . , 2001 ; Daphne Koller and Nir Friedman , 2009 ) .

Here we present an improvement of DARTS , called I-DARTS , that further relaxes the softmaxlocal constraint .

The idea is simple -we consider all incoming edges to a given node in a single softmax .

This offers a broader choice of edges and enlarges the space we infer the network from .

For example , one can simultaneously select multiple important edges between two nodes and leave some node pairs unlinked ( see Figure 1(b ) ) .

I-DARTS outperforms strong baselines on the PTB language modeling and CoNLL named entity recognition ( NER ) tasks .

This gives a new stateof-the-art on the NER dataset .

To our knowledge , it is the first time to apply differentiable architecture search to NLP tasks other than language modeling .

More interestingly , we observe that our method is 1.4X faster than DARTS for convergence of architecture search .

Also , we provide the architectures learned by I-DARTS , which can be referred for related tasks .

We improved the DARTS to address the bias problem by removing the softmax-local constraint .

Our method is search efficient and discovers several better architectures for PTB language modeling and CoNLL named entity recognition ( NER ) tasks .

We plan to consider the network density problem in search and apply I-DARTS to more tasks in our future study .

Softmax-based relaxation used for differentiable architecture search makes edges coming from different nodes incomparable limiting the inference space to sub-graphs.

They propose removing the softmax constraint from the existing differentiable architecture search method to enable a broader choice of edges and enlarge the search space.

The proposed method outperforms an existing method on the CoNLL NER dataset and also achieves 1.4 times faster convergence time.


Generating open-domain conversational responses in the desired style usually suffers from the lack of parallel data in the style .

Meanwhile , using monolingual stylistic data to increase style intensity often leads to the expense of decreasing content relevance .

In this paper , we propose to disentangle the content and style in latent space by diluting sentence-level information in style representations .

Combining the desired style representation and a response content representation will then obtain a stylistic response .

Our approach achieves a higher BERT-based style intensity score and comparable BLEU scores , compared with baselines .

Human

results show that our approach significantly improves style intensity and maintains content relevance .

Linguistic style is an essential aspect of natural language interaction and provides particular ways of using language to engage with the audiences ( Kabbara and Cheung , 2016 ) .

In human-bot conversations , it is crucial to generate stylistic responses for increasing user engagement to conversational systems ( Gan et al . , 2017 ) .

Currently , most of the existing parallel datasets are not stylistically consistent .

Samples in these datasets are usually contributed by a variety of users , resulting in an averaging effect across style characteristics ( Zhang et al . , 2018a ) .

Meanwhile , constructing a parallel stylistic dataset for training the open-domain conversational agents is both labor-intensive and time-consuming .

Recent studies show the effect of stylizing responses using a monolingual dataset in the desired style and a conventional conversational dataset ( Niu and Bansal , 2018 ; Gao et al . , 2019b ) .

However , increasing style intensity often leads to ( Niu and Bansal , 2018 ) , Style Fusion ( Gao et al . , 2019b ) , and our approach , targeting the Holmes style , which is quite formal and polite . the expense of decreasing content relevance between dialogue history and response .

As an example in Figure 1 shows , Niu and Bansal ( 2018 ) independently train a response generation model and a stylistic language model and subsequently interpolates them in the inference phase .

Lacking the interaction between the stylistic language model and response generation encoder , it usually yields a trade-off between style intensity and content relevance .

Gao et al .

( 2019a , b ) fuse a structured latent space where the direction denotes the diversity , and the distance denotes style intensity and content relevance .

The main issue is that style intensity and content relevance are contradictory in measurement but are coupling to the same " distance " metric of the latent space .

To sum up , the key issue of the above studies is the improper entanglement of style and content .

To address the issue , we propose to disentangle the style and content of a response .

The disentanglement is conducted on the structured latent space , where each sentence ( dialogue history , response , and stylistic sentence ) is projected into a vector representation .

We further split the representation into two components : style and content representations .

The former is a corpus-level feature since sentences within a dataset have the same style .

In contrast , the content representation is a sentence-level feature decided by a sentence itself .

We thus disentangle the content and style by diluting sentence-level information in the style representation .

This encourages the encoding of content information into the content representation .

Otherwise , the content information will be corrupted in the style representation , making it hard to reconstruct the original content in the subsequent decoding process .

We conduct on DailyDialogue conversational dataset ( Li et al . , 2017 ) and Holmes monolingual stylistic dataset ( Gao et al . , 2019b ) .

Experimental results show that our proposed approach improves style intensity and maintains content relevance .

Our contributions are listed below : • We propose a unified framework to simultaneously improve style intensity and maintain content relevance for neural stylistic response generation . • We introduce a scheme of learning latent variables by a diluting strategy to disentangle the style and content . • Experimental results show that our approach achieves higher performance in style intensity without decreasing content relevance , compared with previous approaches .

We propose a uniform framework to simultaneously improve the style intensity and maintain the content relevance for neural stylistic response generation .

In contrast to existing approaches , our approach disentangles the style and the content in the latent space by a diluting strategy . show that our approach improves the style intensity of generated responses and maintains the content relevance at the same time , which demonstrates the effectiveness of this approach .

Because existing studies for open-domain conversational response generation have improper entanglement of style and content, they fail to achieve the desired styles for content relevance.

They propose to disentangle the content and style in latent space by diluting sentence-level information, then combine style and content representations to produce stylistic responses.

The proposed method achieves a higher BERT-based style intensity score and comparable BLEU scores, and human evaluation shows improvements in style intensity and content relevance.


Large Transformers pretrained over clinical notes from Electronic Health Records ( EHR ) have afforded substantial gains in performance on predictive clinical tasks .

The cost of training such models ( and the necessity of data access to do so ) coupled with their utility motivates parameter sharing , i.e. , the release of pretrained models such as ClinicalBERT ( Alsentzer et al . , 2019 ) .

While most efforts have used deidentified EHR , many researchers have access to large sets of sensitive , nondeidentified EHR with which they might train a BERT model ( or similar ) .

Would it be safe to release the weights of such a model if they did ?

In this work , we design a battery of approaches intended to recover Personal Health Information ( PHI ) from a trained BERT .

Specifically , we attempt to recover patient names and conditions with which they are associated .

We find that simple probing

are not able to meaningfully extract sensitive information from BERT trained over the MIMIC-III corpus of EHR .

However , more sophisticated " attacks " may succeed in doing so : To facilitate such research , we make our

and baseline probing models available .

1 Pretraining large ( masked ) language models such as BERT ( Devlin et al . , 2019 ) over domain specific corpora has yielded consistent performance gains across a broad range of tasks .

In biomedical NLP , this has often meant pretraining models over collections of Electronic Health Records ( EHRs ) ( Alsentzer et al . , 2019 ) .

For example , Huang et al .

( 2019 ) showed that pretraining models over EHR data improves performance on clinical predictive tasks .

Given their empirical utility , and the fact that pretraining large networks requires a nontrivial amount of compute , there is a natural desire to equal contribution .

1 https://github.com / elehman16/ exposing_patient_data_release . share the model parameters for use by other researchers in the community .

However , in the context of pretraining models over patient EHR , this poses unique potential privacy concerns : Might the parameters of trained models leak sensitive patient information ?

In the United States , the Health Insurance Portability and Accountability Act ( HIPAA ) prohibits the sharing of such text if it contains any reference to Protected Health Information ( PHI ) .

If one removes all reference to PHI , the data is considered " deidentified " , and is therefore legal to share .

While researchers may not directly share nondeidentified text,2 it is unclear to what extent models pretrained on non-deidentified data pose privacy risks .

Further , recent work has shown that general purpose large language models are prone to memorizing sensitive information which can subsequently be extracted ( Carlini et al . , 2020 ) .

In the context of biomedical NLP , such concerns have been cited as reasons for withholding direct publication of trained model weights ( McKinney et al . , 2020 ) .

These uncertainties will continue to hamper dissemination of trained models among the broader biomedical NLP research community , motivating a need to investigate the susceptibility of such models to adversarial attacks .

This work is a first step towards exploring the potential privacy implications of sharing model weights induced over non-deidentified EHR text .

We propose and run a battery of intended to evaluate the degree to which Transformers ( here , BERT ) pretrained via standard masked language modeling objectives over notes in EHR might reveal sensitive information ( Figure 1 ) .

We find that simple methods are able to recover associations between patients and conditions at rates better than chance , but not with performance beyond that achievable using baseline condition frequencies .

This holds even when we enrich clinical notes by explicitly inserting patient names into every sentence .

Our results using a recently proposed , more sophisticated attack based on generating text ( Carlini et al . , 2020 ) are mixed , and constitute a promising direction for future work .

We have performed an initial investigation into the degree to which large Transformers pretrained over EHR data might reveal sensitive personal health information ( PHI ) .

We ran a battery of in which we attempted to recover such information from BERT model weights estimated over the MIMIC-III dataset ( into which we artificially reintroduced patient names , as MIMIC is deidentified ) .

Across these , we found that we were mostly unable to meaningfully expose PHI using simple methods .

Moreover , even when we constructed a variant of data in which we prepended patient names to every sentence prior to pretraining BERT , we were still unable to recover sensitive information reliably .

Our initial results using more advanced techniques based on generation ( Carlini et al .

2020 ; Table 9 ) are intriguing but inconclusive at present .

Our results certainly do not rule out the possibility that more advanced methods might reveal PHI .

But , these findings do at least suggest that doing so is not trivial .

To facilitate further research , we make our and baseline probing models available : https://github.com/ elehman16 / exposing_patient_data_release .

Whether distributing a language model such as BERT pre-trained on non-deidentified data like Electronic Health Records can leak sensitive patient information remains unknown.

They design a set of approaches aiming to recover Personal Health Information such as patient names and associated conditions from a trained BERT.

They find that simple probing methods are unable to extract sensitive information better than change even by using patient names explicitly.


We investigate the influence of information status ( IS ) on constituent order in German , and integrate our findings into a loglinear surface realisation ranking model .

We show that the distribution of pairs of IS categories is strongly asymmetric .

Moreover , each category is correlated with morphosyntactic features , which can be automatically detected .

We build a loglinear model that incorporates these asymmetries for ranking German string realisations from input LFG F-structures .

We show that it achieves a statistically significantly higher BLEU score than the baseline system without these features .

There are many factors that influence word order , e.g. humanness , definiteness , linear order of grammatical functions , givenness , focus , constituent weight .

In some cases , it can be relatively straightforward to automatically detect these features ( i.e. in the case of definiteness , this is a syntactic property ) .

The more complex the feature , the more difficult it is to automatically detect .

It is common knowledge that information status1 ( henceforth , IS ) has a strong influence on syntax and word order ; for instance , in inversions , where the subject follows some preposed element , Birner ( 1994 ) reports that the preposed element must not be newer in the discourse than the subject .

We would like to be able to use information related to IS in the automatic generation of German text .

Ideally , we would automatically annotate text with IS labels and learn from this data .

Unfortunately , however , to date , there has been little success in automatically annotating text with IS .

We believe , however , that despite this shortcoming , we can still take advantage of some of the insights gained from looking at the influence of IS on word order .

Specifically , we look at the problem from a more general perspective by computing an asymmetry ratio for each pair of IS categories .

Results show that there are a large number of pairs exhibiting clear ordering preferences when co-occurring in the same clause .

The question then becomes , without being able to automatically detect these IS category pairs , can we , nevertheless , take advantage of these strong asymmetric patterns in generation .

We investigate the ( automatically detectable ) morphosyntactic characteristics of each asymmetric IS pair and integrate these syntactic asymmetric properties into the generation process .

The paper is structured as follows : Section 2 outlines the underlying realisation ranking system for our experiments .

Section 3 introduces information status and Section 4 describes how we extract and measure asymmetries in information status .

In Section 5 , we examine the syntactic characteristics of the IS asymmetries .

Section 6 outlines realisation ranking experiments to test the integration of IS into the system .

We discuss our findings in Section 7 and finally we conclude in Section 8 .

In this paper we presented a novel method of including IS into the task of generation ranking .

Since automatic annotation of IS labels themselves is not currently possible , we approximate the IS categories by their syntactic characteristics .

By calculating strong asymmetries between pairs of IS labels , and establishing the most frequent syntactic characteristics of these asymmetries , we designed a new set of features for a log-linear ranking model .

In comparison to a baseline model , we achieve statistically significant improvement in BLEU score .

We showed that these improvements were not only due to the effect of purely syntactic asymmetries , but that the IS asymmetries were what drove the improved model .

While it is known that information status has a strong influence on syntax and word order, its automatic annotation has not been successful.

They propose to integrate syntactic asymmetric properties of information status into a log-linear model to generate German by approximating their category pairs by syntactic characteristics.

The proposed approach achieves a higher BLEU score than the baseline system, not only by the effect of purely syntactic asymmetries but of information status.


This paper explores the determinants of adjective-noun plausibility by using correlation analysis to compare judgements elicited from human subjects with five corpus-based variables : co-occurrence frequency of the adjective-noun pair , noun frequency , conditional probability of the noun given the adjective , the log-likelihood ratio , and Resnik 's ( 1993 ) selectional association measure .

The highest correlation is obtained with the co-occurrence frequency , which points to the strongly lexicalist and collocational nature of adjective-noun combinations .

Research on linguistic plausibility has focused mainly on the effects of argument plausibility during the processing of locally ambiguous sentences .

Psycholinguists have investigated whether the plausibility of the direct object affects reading times for sentences like ( 1 ) .

Here , argument plausibility refers to " pragmatic plausibility " or " local semantic fit " ( Holmes et al . , 1989 ) , and judgements of plausibility are typically obtained by asking subjects to rate sentence fragments containing verb-argument combinations ( as an example consider the bracketed parts of the sentences in ( 1 ) ) .

Such experiments typically use an ordinal scale for plausibility ( e.g. , from 1 to 7 ) .

( 1 ) a. [ The senior senator regretted the decision ] had ever been made public . b. [ The senior senator regretted the reporter ] had ever seen the report .

The majority of research has focussed on investigating the effect of rated plausibility for verb-object combinations in human sentence processing ( Garnsey et al . , 1997 ; Pickering and Traxler , 1998 ) .

However , plausibility effects have also been observed for adjectivenoun combinations in a head-modifier relationship .

Murphy ( 1990 ) has shown that typical adjectivenoun phrases ( e.g. , salty olives ) are easier to interpret in comparison to atypical ones ( e.g. , sweet olives ) .

Murphy provides a schema-based explanation for this finding by postulating that in typical adjective-noun phrases , the adjective modifies part of the noun 's schema and consequently it is understood more quickly , whereas in atypical combinations , the adjective modifies non-schematic aspects of the noun , which leads to interpretation difficulties .

Smadja ( 1991 ) argues that the reason people prefer strong tea to powerful tea and powerful car to strong car is neither purely syntactic nor purely semantic , but rather lexical .

A similar argument is put forward by Cruse ( 1986 ) , who observes that the adjective spotless collocates well with the noun kitchen , relatively worse with the noun complexion and not all with the noun taste .

According to Cruse , words like spotless have idiosyncratic collocational restrictions : differences in the degree of acceptability of the adjective and its collocates do not seem to depend on the meaning of the individual words .

This paper explored the determinants of linguistic plausibility , a concept that is potentially relevant for lexical choice in natural language generation systems .

Adjective-noun plausibility served as a test bed for a number of corpus-based models of linguistic plausibility .

Plausibility judgements were obtained from human subjects for 90 randomly selected adjective-noun pairs .

The ratings revealed a clear effect of familiarity of the adjective-noun pair ( operationalised by corpus co-occurrence frequency ) .

In a correlation analysis we compared judged plausibility with the predictions of five corpus-based variables .

The highest correlation was obtained with the co-occurrence frequency of the adjective-noun pair .

Conditional probability , the log-likelihood ratio , and Resnik 's ( 1993 ) selectional association measure were also significantly correlated with plausibility ratings .

The correlation with Resnik 's measure was negative , contrary to the predictions of his model .

This points to a problem with his technique for estimating word class frequencies , which is aggravated by the collocational nature of noun-adjective combinations .

Overall , the results confirm the strongly lexicalist and collocational nature of adjective-noun combinations .

This fact could be exploited in a generation system by taking into account corpus co-occurrence counts for adjective-noun pairs ( which can be obtained straightforwardly ) during lexical choice .

Future research has to identify how this approach can be generalised to unseen data .

Existing studies on the plausibility of texts focus on verb-object combinations while there could be some impacts from adjective-noun pairs as well.

They perform correlational analysis to explore the determinants of adjective-noun plausibility against human judgements.

They show that co-occurrence frequency correlates play the biggest role as the determinans of adjective-noun plausibility.


Unsupervised relation extraction ( URE ) extracts relations between named entities from raw text without manually-labelled data and existing knowledge bases ( KBs ) .

URE

can be categorised into generative and discriminative approaches , which rely either on hand-crafted features or surface form .

However , we demonstrate that by using only named entities to induce relation types , we can outperform existing on two popular datasets .

We conduct a comparison and evaluation of our findings with other URE techniques , to ascertain the important features in URE .

We conclude that entity types provide a strong inductive bias for URE .

1 Relation extraction ( RE ) extracts semantic relations between entities from plain text .

For instance , " Jon Robin Baitz head , born in Los Angeles tail ... " expresses the relation /people / person / place of birth between the two head-tail entities .

Extracted relations are then used for several downstream tasks such as information retrieval ( Corcoglioniti et al . , 2016 ) and knowledge base construction ( Al-Zaidy and Giles , 2018 ) .

RE has been widely studied using fully supervised learning ( Nguyen and Grishman , 2015 ; Miwa and Bansal , 2016 ; Zhang et al . , 2017 Zhang et al . , , 2018 ) ) and distantly supervised approaches ( Mintz et al . , 2009 ; Riedel et al . , 2010 ; Lin et al . , 2016 ) .

Unsupervised relation extraction ( URE ) methods have not been explored as much as fully or distantly supervised learning techniques .

URE is promising , since it does not require manually annotated data nor human curated knowledge bases ( KBs ) , which are expensive to produce .

Therefore , it can be applied to domains and languages where annotated data and KBs are not available .

Moreover , URE can discover new relation types , since it is not restricted to specific relation types in the same way as fully and distantly supervised .

One might argue that Open Information Extraction ( OpenIE ) can also discover new relations .

However , OpenIE identifies relations based on textual surface information .

Thus , similar relations with different textual forms may not be recognised .

Unlike OpenIE , URE groups similar relations into clusters .

Despite these advantages , there are only a few attempts tackling URE using machine learning ( ML ) ( Hasegawa et al . , 2004 ; Banko et al . , 2007 ; Yao et al . , 2011 ; Marcheggiani and Titov , 2016 ; Simon et al . , 2019 ) .

Similarly to other unsupervised learning tasks , a challenge in URE is how to evaluate results .

Recent approaches ( Yao et al . , 2011 ; Marcheggiani and Titov , 2016 ; Simon et al . , 2019 ) employ a widely used data generation setting in distantly supervised RE , i.e. , aligning a large amount of raw text against triplets in a curated KB .

A standard metric score is computed by comparing the output relation clusters against the automatically annotated relations .

In particular , the NYT-FB dataset ( Marcheggiani and Titov , 2016 ) which is used for

, has been created by mapping relation triplets in Freebase ( Bollacker et al . , 2008 ) against plain text articles in the New York Times ( NYT ) corpus ( Sandhaus , 2008 ) .

Standard clustering metrics for URE include B 3 ( Bagga and Baldwin , 1998 ) , V-measure ( Rosenberg and Hirschberg , 2007 ) , and ARI ( Hubert and Arabie , 1985 ) .

Although the above mentioned experimental setting can be created automatically , there are three challenges to overcome .

Firstly , the development and test sets are silver , i.e. , they include noisy labelled instances , since they are not human-curated .

Secondly , the development and test sentences are part of the training set , i.e. , a transductive setting .

It is thus unclear how well the existing models perform on unseen sentences .

Finally , NYT-FB can be considered highly imbalanced , since only 2.1 % of the training sentences can be aligned with Freebase 's triplets .

Due to the noisy nature of silver data ( NYT-FB ) , on silver data will not accurately reflect the system performance .

We also need unseen data during testing to examine the system generalisation .

To overcome these challenges , we will employ the test set of TACRED ( Zhang et al . , 2017 ) , a widely used manually annotated corpus .

Regarding the imbalanced data , we will demonstrate that in fact around 60 % ( instead of 2.1 % ) of instances in the training set express relation types defined in Freebase .

In this work , we present a simple URE approach relying only on entity types that can obtain improved performance compared to current methods .

Specifically , given a sentence consisting of two entities and their corresponding entity types , e.g. , PERSON and LOCATION , we induce relations as the combination of entity types , e.g. , PERSON-LOCATION .

It should be noted that we employ only entity types because their combinations form reasonably coarse relation types ( e.g. , PERSON-LOCATION covers /people / person / place of birth defined in Freebase ) .

We further discuss our improved performance in § 3 .

Our contributions are as follows : ( i ) We perform on both automatically / manuallylabelled datasets , namely NYT-FB and TACRED , respectively .

We show that two methods using only entity types can outperform the state-of-theart models including both feature-engineering and deep learning approaches .

The surprising results raise questions about the current state of unsupervised relation extraction .

( ii ) For model design , we show that link predictor provides a good signal to train a URE model ( Fig 1 ) .

We also illustrate that entity types are a strong inductive bias for URE ( Table 1 ) .

We have shown the importance of entity types in URE .

Our methods use only entity types , yet they yield higher performance than previous work on both NYT-FB and TACRED .

We have investigated the current experimental setting , concluding that a strong inductive bias is required to train a relation extraction model without labelled data .

URE remains challenging , which requires improved methods to deal with silver data .

We also plan to use different types of labelled data , e.g. , domain specific data sets , to ascertain whether entity type information is more discriminative in sub-languages .

Unsupervised relation extraction has not been explored although it can be applied to languages and domains where annotated data or knowledge bases are not available.

They propose an approach which only relies on entity types and performs a comparison with other unsupervised approaches to find important features.

They find that only using entity types can outperform existing methods on two datasets indicating that they provide a strong inductive bias.


Although discriminative training guarantees to improve statistical machine translation by incorporating a large amount of overlapping features , it is hard to scale up to large data due to decoding complexity .

We propose a new algorithm to generate translation forest of training data in linear time with the help of word alignment .

Our algorithm also alleviates the oracle selection problem by ensuring that a forest always contains derivations that exactly yield the reference translation .

With millions of features trained on 519 K sentences in 0.03 second per sentence , our system achieves significant improvement by 0.84 BLEU over the baseline system on the NIST Chinese-English test sets .

Discriminative model ( Och and Ney , 2002 ) can easily incorporate non-independent and overlapping features , and has been dominating the research field of statistical machine translation ( SMT ) in the last decade .

Recent work have shown that SMT benefits a lot from exploiting large amount of features ( Liang et al . , 2006 ; Tillmann and Zhang , 2006 ; Watanabe et al . , 2007 ; Blunsom et al . , 2008 ; Chiang et al . , 2009 ) .

However , the training of the large number of features was always restricted in fairly small data sets .

Some systems limit the number of training examples , while others use short sentences to maintain efficiency .

Overfitting problem often comes when training many features on a small data ( Watanabe et al . , 2007 ; Chiang et al . , 2009 ) .

Obviously , using much more data can alleviate such problem .

Furthermore , large data also enables us to globally train millions of sparse lexical features which offer accurate clues for SMT .

Despite these advantages , to the best of our knowledge , no previous discriminative training paradigms scale up to use a large amount of training data .

The main obstacle comes from the complexity of packed forests or n-best lists generation which requires to search through all possible translations of each training example , which is computationally prohibitive in practice for SMT .

To make normalization efficient , contrastive estimation ( Smith and Eisner , 2005 ; Poon et al . , 2009 ) introduce neighborhood for unsupervised log-linear model , and has presented positive results in various tasks .

Motivated by these work , we use a translation forest ( Section 3 ) which contains both " reference " derivations that potentially yield the reference translation and also neighboring " non-reference " derivations that fail to produce the reference translation.1 However , the complexity of generating this translation forest is up to O(n 6 ) , because we still need biparsing to create the reference derivations .

Consequently , we propose a method to fast generate a subset of the forest .

The key idea ( Section 4 ) is to initialize a reference derivation tree with maximum score by the help of word alignment , and then traverse the tree to generate the subset forest in linear time .

Besides the efficiency improvement , such a forest allows us to train the model without resort- r 1 X ⇒ ⟨X 1 bei X 2 , X 1 was X 2 ⟩ e 2 r 2 X ⇒ ⟨qiangshou bei X 1 , the gunman was X 1 ⟩ e 3 r 3 X ⇒ ⟨jingfang X 1 , X 1 by the police⟩ e 4 r 4 X ⇒ ⟨jingfang X 1 , police X 1 ⟩ e 5 r 5 X ⇒ ⟨qiangshou , the gunman⟩ e 6 r 6 X ⇒ ⟨jibi , shot dead⟩ Figure 1 : A translation forest which is the running example throughout this paper .

The reference translation is " the gunman was killed by the police " .

( 1 ) Solid hyperedges denote a " reference " derivation tree t 1 which exactly yields the reference translation .

( 2 ) Replacing e 3 in t 1 with e 4 results a competing non-reference derivation t 2 , which fails to swap the order of X 3,4 .

( 3 ) Removing e 1 and e 5 in t 1 and adding e 2 leads to another reference derivation t 3 .

Generally , this is done by deleting a node X 0,1 . ing to constructing the oracle reference ( Liang et al . , 2006 ; Watanabe et al . , 2007 ; Chiang et al . , 2009 ) , which is non-trivial for SMT and needs to be determined experimentally .

Given such forests , we globally learn a log-linear model using stochastic gradient descend ( Section 5 ) .

Overall , both the generation of forests and the training algorithm are scalable , enabling us to train millions of features on large-scale data .

To show the effect of our framework , we globally train millions of word level context features motivated by word sense disambiguation ( Chan et al . , 2007 ) together with the features used in traditional SMT system ( Section 6 ) .

Training on 519 K sentence pairs in 0.03 seconds per sentence , we achieve significantly improvement over the traditional pipeline by 0.84 BLEU .

We have presented a fast generation algorithm for translation forest which contains both reference derivations and neighboring non-reference derivations for large-scale SMT discriminative training .

We have achieved significantly improvement of 0.84 BLEU by incorporate 13.9 M feature trained on 519 K data in 0.03 second per sentence .

In this paper , we define the forest based on competing derivations which only differ in one rule .

There may be better classes of forest that can produce a better performance .

It 's interesting to modify the definition of forest , and use more local operators to increase the size of forest .

Furthermore , since the generation of forests is quite general , it 's straight to apply our forest on other learning algorithms .

Finally , we hope to exploit more features such as reordering features and syntactic features so as to further improve the performance .

Applying discriminative models to statistical machine translation systems is expensive, however; reducing training set size leads to overfitting.

They propose a faster algorithm for translation forest generation which runs in linear time.

The proposed forest generation method can process a sentence in 0.03 seconds using a large feature space and also significantly outperforms the existing pipeline system.