We describe a syntax-based algorithm that automatically builds Finite State Automata ( word lattices ) from semantically equivalent translation sets . These FSAs are good representations of paraphrases . They can be used to extract lexical and syntactic paraphrase pairs and to generate new , unseen sentences that express the same meaning as the sentences in the input sets . Our FSAs can also predict the correctness of alternative semantic renderings , which may be used to evaluate the quality of translations . In the past , paraphrases have come under the scrutiny of many research communities . Information retrieval researchers have used paraphrasing techniques for query reformulation in order to increase the recall of information retrieval engines ( Sparck Jones and Tait , 1984 ) . Natural language generation researchers have used paraphrasing to increase the expressive power of generation systems ( Iordanskaja et al . , 1991 ; Lenke , 1994 ; Stede , 1999 ) . And researchers in multi-document text summarization ( Barzilay et al . , 1999 ) , information extraction ( Shinyama et al . , 2002 ) , and question answering ( Lin and Pantel , 2001 ; Hermjakob et al . , 2002 ) have focused on identifying and exploiting paraphrases in the context of recognizing redundancies , alternative formulations of the same meaning , and improving the performance of question answering systems . In previous work ( Barzilay and McKeown , 2001 ; Lin and Pantel , 2001 ; Shinyama et al . , 2002 ) , paraphrases are represented as sets or pairs of semantically equivalent words , phrases , and patterns . Although this is adequate in the context of some applications , it is clearly too weak from a generative perspective . Assume , for example , that we know that text pairs ( stock market rose , stock market gained ) and ( stock market rose , stock prices rose ) have the same meaning . If we memorized only these two pairs , it would be impossible to infer that , in fact , consistent with our intuition , any of the following sets of phrases are also semantically equivalent : { stock market rose , stock market gained , stock prices rose , stock prices gained } and { stock market , stock prices } in the context of rose or gained ; { market rose } , { market gained } , { prices rose } and { prices gained } in the context of stock ; and so on . In this paper , we propose solutions for two problems : the problem of paraphrase representation and the problem of paraphrase induction . We propose a new , finite-statebased representation of paraphrases that enables one to encode compactly large numbers of paraphrases . We also propose algorithms that automatically derive such representations from inputs that are now routinely released in conjunction with large scale machine translation evaluations ( DARPA , 2002 ) : multiple English translations of many foreign language texts . For instance , when given as input the 11 semantically equivalent English translations in Figure 1 , our algorithm automatically induces the FSA in Figure 2 , which represents compactly 49 distinct renderings of the same semantic meaning . Our FSAs capture both lexical paraphrases , such as { fighting , bat-tle } , { died , were killed } and structural paraphrases such as { last week 's fighting , the battle of last week } . The contexts in which these are correct paraphrases are also conveniently captured in the representation . In previous work , Langkilde and Knight ( 1998 ) used word lattices for language generation , but their method involved hand-crafted rules . Bangalore et al . ( 2001 ) and Barzilay and Lee ( 2002 ) both applied the technique of multi-sequence alignment ( MSA ) to align parallel corpora and produced similar FSAs . For their purposes , they mainly need to ensure the correctness of consensus among different translations , so that different constituent orderings in input sentences do not pose a serious prob- lem . In contrast , we want to ensure the correctness of all paths represented by the FSAs , and direct application of MSA in the presence of different constituent orderings can be problematic . For example , when given as input the same sentences in Figure 1 , one instantiation of the MSA algorithm produces the FSA in Figure 3 , which contains many " bad " paths such as the battle of last week 's fighting took at least 12 people lost their people died in the fighting last week 's fighting ( See Section 4.2.2 for a more quantitative analysis . ) . It 's still possible to use MSA if , for example , the input is pre-clustered to have the same constituent ordering ( Barzilay and Lee ( 2003 ) ) . But we chose to approach this problem from another direction . As a result , we propose a new syntax-based algorithm to produce FSAs . In this paper , we first introduce the multiple translation corpus that we use in our experiments ( see Section 2 ) . We then present the algorithms that we developed to induce finite-state paraphrase representations from such data ( see Section 3 ) . An important part of the paper is dedicated to evaluating the quality of the finite-state representations that we derive ( see Section 4 ) . Since our representations encode thousands and sometimes millions of equivalent verbalizations of the same meaning , we use both manual and automatic evaluation techniques . Some of the automatic evaluations we perform are novel as well . In this paper , we presented a new syntax-based algorithm that learns paraphrases from a newly available dataset . The multiple translation corpus that we use in this paper is the first instance in a series of similar corpora that are built and made publicly available by LDC in the context of a series of DARPA-sponsored MT evaluations . The algorithm we proposed constructs finite state representations of paraphrases that are useful in many contexts : to induce large lists of lexical and structural paraphrases ; to generate semantically equivalent renderings of a given meaning ; and to estimate the quality of machine translation systems . More experiments need to be carried out in order to assess extrinsically whether the FSAs we produce can be used to yield higher agreement scores between human and automatic assessments of translation quality . In our future work , we wish to experiment with more flexible merging algorithms and to integrate better the top-down and bottom-up processes that are used to in-duce FSAs . We also wish to extract more abstract paraphrase patterns from the current representation . Such patterns are more likely to get reused -which would help us get reliable statistics for them in the extraction phase , and also have a better chance of being applicable to unseen data .

Existing approaches to represent paraphrases as sets or pairs of semantically equivalent words, phrases and patterns that are weak for text generation purposes.

They propose a syntax-based algorithm that builds Finite State Automata from translation sets which are good representations of paraphrases.

Manual and automatic evaluations show that the representations extracted by the proposed method can be used for automatic translation evaluations.


While sophisticated neural-based techniques have been developed in reading comprehension , most approaches model the answer in an independent manner , ignoring its relations with other answer candidates . This problem can be even worse in open-domain scenarios , where candidates from multiple passages should be combined to answer a single question . In this paper , we formulate reading comprehension as an extract-then-select twostage procedure . We first extract answer candidates from passages , then select the final answer by combining information from all the candidates . Furthermore , we regard candidate extraction as a latent variable and train the two-stage process jointly with reinforcement learning . As a result , our approach has improved the state-ofthe-art performance significantly on two challenging open-domain reading comprehension datasets . Further analysis demonstrates the effectiveness of our model components , especially the information fusion of all the candidates and the joint training of the extract-then-select procedure . Teaching machines to read and comprehend human languages is a long-standing objective in natural language processing . In order to evaluate this ability , reading comprehension ( RC ) is designed to answer questions through reading relevant passages . In recent years , RC has attracted intense interest . Various advanced neural models have been proposed along with newly released datasets ( Hermann et al . , 2015 ; Rajpurkar et al . , 2016 ; Dunn et al . , 2017 ; Dhingra et al . , 2017b ; He et al . , 2017 ) . In this paper , we formulate the problem of RC as a two-stage process , which first generates candidates with an extraction model , then selects the final answer by combining the information from all the candidates . Furthermore , we treat candidate extraction as a latent variable and jointly train these two stages with RL . Experiments on public open-domain RC datasets Quasar-T and SearchQA show the necessity of introducing the selection model and the effectiveness of fusing candidates information when modeling . Moreover , our joint training strategy leads to significant improvements in performance .

Existing models for reading comprehension do not consider multiple answer candidates which can be problematic when they need to fuse information from multiple passages.

They propose to approach reading comprehension with an extract-then-select procedure, where a model learns two tasks jointly using latent variables and reinforcement learning.

The proposed model can fuse answer candidates from multiple candidates and significantly outperform existing models on two open-domain reading comprehension tasks.


We present two discriminative methods for name transliteration . The methods correspond to local and global modeling approaches in modeling structured output spaces . Both methods do not require alignment of names in different languages -their features are computed directly from the names themselves . We perform an experimental evaluation of the methods for name transliteration from three languages ( Arabic , Korean , and Russian ) into English , and compare the methods experimentally to a state-of-theart joint probabilistic modeling approach . We find that the discriminative methods outperform probabilistic modeling , with the global discriminative modeling approach achieving the best performance in all languages . Name transliteration is an important task of transcribing a name from alphabet to another . For example , an Arabic ,"وﻟﻴﺎم" Korean " 윌리엄 " , and Russian " Вильям " all correspond to English " William " . We address the problem of transliteration in the general setting : it involves trying to recover original English names from their transcription in a foreign language , as well as finding an acceptable spelling of a foreign name in English . We apply name transliteration in the context of cross-lingual information extraction . Name extractors are currently available in multiple languages . Our goal is to make the extracted names understandable to monolingual English speakers by transliterating the names into English . The extraction context of the transliteration application imposes additional complexity constraints on the task . In particular , we aim for the transliteration speed to be comparable to that of extraction speed . Since most current extraction systems are fairly fast ( > 1 Gb of text per hour ) , the complexity requirement reduces the range of techniques applicable to the transliteration . More precisely , we can not use WWW and the web count information to hone in on the right transliteration candidate . Instead , all relevant transliteration information has to be represented within a compact and self-contained transliteration model . We present two methods for creating and applying transliteration models . In contrast to most previous transliteration approaches , our models are discriminative . Using an existing transliteration dictionary D ( a set of name pairs { ( f , e ) } ) , we learn a function that directly maps a name f from one language into a name e in another language . We do not estimate either direct conditional p(e|f ) or reverse conditional p(f|e ) or joint p(e , f ) probability models . Furthermore , we do away with the notion of alignment : our transliteration model does not require and is not defined of in terms of aligned e and f. Instead , all features used by the model are computed directly from the names f and e without any need for their alignment . The two discriminative methods that we present correspond to local and global modeling paradigms for solving complex learning problems with structured output spaces . In the local setting , we learn linear classifiers that predict a letter e i from the previously predicted letters e 1 … e i-1 and the original name f. In the global setting , we learn a function W mapping a pair ( f , e ) into a score W(f , e)∈ R. The function W is linear in features computed from the pair ( f , e ) . We describe the pertinent feature spaces as well as pre-sent both training and decoding algorithms for the local and global settings . We perform an experimental evaluation for three language pairs ( transliteration from Arabic , Korean , and Russian into English ) comparing our methods to a joint probabilistic modeling approach to transliteration , which was shown to deliver superior performance . We show experimentally that both discriminative methods outperform the probabilistic approach , with global discriminative modeling achieving the best performance in all languages . We presented two novel discriminative approaches to name transliteration that do not employ the notion of alignment . We showed experimentally that the approaches lead to superior experimental results in all languages , with the global discriminative modeling approach achieving the best performance . The results are somewhat surprising , for the notion of alignment seems very intuitive and useful for transliteration . We will investigate whether similar alignment-free methodology can be extended to full-text translation . It will also be interesting to study the relationship between our discriminative alignment-free methods and recently proposed discriminative alignment-based methods for transliteration and translation ( Taskar et al . 2005a ; Moore 2005 ) . We also showed that for name transliteration , global discriminative modeling is superior to local classifier-based discriminative modeling . This may have resulted from poor calibration of scores and probabilities produced by individual classifiers . We plan to further investigate the relationship between the local and global approaches to complex learning problems in natural language .

The name transliteration task aims to transcribe extracted names into English, and since current extraction systems are fairly fast, applicable techniques for transliteration are limited.

They present two discriminative methods that learn a map function from one language into another using a dictionary without the notion of alignment.

The proposed methods outperform state-of-the-art probabilistic models on name transliteration from Arabic, Korean, and Russian to English, and the global discriminative modelling performs the best.


Motivated by the success of T5 ( Text-To-Text Transfer Transformer ) in pre-trained natural language processing models , we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech / text representation learning . The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific ( speech / text ) pre / post-nets . After preprocessing the input speech / text through the pre-nets , the shared encoder-decoder network models the sequence-to-sequence transformation , and then the post-nets generate the output in the speech / text modality based on the output of the decoder . Leveraging large-scale unlabeled speech and text data , we pre-train SpeechT5 to learn a unified-modal representation , hoping to improve the modeling capability for both speech and text . To align the textual and speech information into this unified semantic space , we propose a cross-modal vector quantization approach that randomly mixes up speech / text states with latent units as the interface between encoder and decoder . Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks , including automatic speech recognition , speech synthesis , speech translation , voice conversion , speech enhancement , and speaker identification . We release our code and model at https://github.com / microsoft/ SpeechT5 . Starting with ELMo ( Peters et al . , 2018 ) and BERT ( Devlin et al . , 2019 ) , substantial work has shown that pre-trained models can significantly improve in various natural language processing ( NLP ) tasks Figure 1 : An illustration of the SpeechT5 framework , which treats spoken language processing tasks as a speech / text to speech / text format , including automatic speech recognition ( ASR ) , speech translation ( ST ) , speech identification ( SID ) , text to speech ( TTS ) , voice conversion ( VC ) , and speech enhancement ( SE ) . ( Radford et al . , 2019 ; CONNEAU and Lample , 2019 ; Yang et al . , 2019 ; Dong et al . , 2019 ; Lewis et al . , 2020 ) . Following the pre-training techniques in NLP , self-supervised speech representation learning has also been investigated and shown promising results , benefiting from richly learned representations ( Chung and Glass , 2018 ; Chuang et al . , 2020 ; Song et al . , 2019 ; Baevski et al . , 2020 ; Wang et al . , 2021 ; Hsu et al . , 2021 ; Chung et al . , 2021a ) , such as wav2vec 2.0 ( Baevski et al . , 2020 ) and HuBERT ( Hsu et al . , 2021 ) . However , previous speech pre-training work suffers from two problems : ( 1 ) most of them learn the speech representation with only unlabeled speech data but ignore the importance of textual data to spoken language tasks ( e.g. , automatic speech recognition ) which require the modality transformation ; ( 2 ) most of these models solely rely on a pre-trained speech encoder for various downstream tasks , leaving the decoder not pre-trained for the sequence-to-sequence generation tasks . How to design a unified encoder-decoder model that can take advantage of both unlabeled speech and text data to improve various spoken language processing tasks is not well explored . Inspired by the T5 method ( Raffel et al . , 2020 ) , we attempt to formulate each spoken language processing task as a speech / text to speech / text problem via an encoder-decoder framework , which enables us to use the same pre-trained model with bimodal data across diverse tasks , as shown in Figure 1 . To achieve this , we propose a unifiedmodal pre-training framework , SpeechT5 , containing an encoder-decoder backbone network and modal-specific pre / post-nets . With the pre-nets , the input speech / text is embedded in a shared space , and the encoder-decoder backbone network models the sequence-to-sequence conversion , from which the model-specific post-nets generate the speech / text output . Particularly , SpeechT5 is mainly pre-trained with a denoising sequence-tosequence method by leveraging large-scale unlabeled text and speech corpus . To align the textual and acoustic information into a unified semantic space , the proposed SpeechT5 model ( 1 ) maps text and speech representations into a shared vector quantization space , and ( 2 ) randomly mixes up the quantized latent representations and the contextual states , which can better guide the quantizer to learn the cross-modal features . We fine-tune SpeechT5 on a wide variety of downstream spoken language processing tasks , including automatic speech recognition ( ASR ) , textto-speech ( TTS ) , speech translation ( ST ) , voice conversion ( VC ) , speech enhancement ( SE ) , and speaker identification ( SID ) . Massive experiments show that the proposed SpeechT5 model achieves a significant improvement on these spoken language processing tasks compared with the state-of-theart baselines . Specifically , the proposed SpeechT5 outperforms wav2vec 2.0 ( Baevski et al . , 2020 ) and HuBERT ( Hsu et al . , 2021 ) with the BASE model on the ASR task and also performs better than the state-of-the-art voice Transformer network ( Huang et al . , 2021 ) on the VC task . Besides , SpeechT5 is significantly superior to SpeechNet ( Chen et al . , 2021b ) and pre-trained models from SUPERB ( Yang et al . , 2021 ) and achieves the stateof-the-art performance ( i.e. , 96.49 % ) on the SID task . We further provide an empirical comparison of the pre-training tasks and modules , and the ablation study demonstrates the effectiveness of the proposed joint speech-text pre-training method . The contributions of this paper are summarized as follows . • To the best of our knowledge , this is the first work to investigate a unified encoder-decoder framework for various spoken language processing tasks . • We propose a cross-modal vector quantization approach , which learns the implicit alignment between acoustic and textual representation with large-scale unlabeled speech and text data . • Extensive experiments on spoken language processing tasks demonstrate the effectiveness and superiority of the proposed SpeechT5 model . In this paper , we have proposed SpeechT5 as a pretrained encoder-decoder model for various spoken language processing tasks . We convert all spoken language processing tasks into a speech / text to speech / text format and propose a novel joint pretraining method to utilize cross-modal information by leveraging the unlabeled speech and text data . The proposed unified encoder-decoder model can support generation tasks such as speech translation and voice conversion . Massive experiments show that SpeechT5 significantly outperforms all baselines in several spoken language processing tasks . In the future , we are going to pre-train the SpeechT5 with a larger model and more unlabeled data . We are also interested in extending the proposed SpeechT5 framework to address multilingual spoken language processing tasks for future work .

Existing speech pre-training methods ignore the importance of textual data and solely depend on encoders leaving the decoder out of pre-training for generation tasks.

They propose a unified-modal encoder-decoder framework with shared and modal-specific networks for self-supervised speech and text representation learning by using unlabeled text and speech corpus.

The fine-tuned proposed model is evaluated on a variety of spoken processing tasks and outperforms state-of-the-art models on voice conversion speaker identification tasks.


Recent studies on extractive text summarization formulate it as a combinatorial optimization problem such as a Knapsack Problem , a Maximum Coverage Problem or a Budgeted Median Problem . These methods successfully improved summarization quality , but they did not consider the rhetorical relations between the textual units of a source document . Thus , summaries generated by these methods may lack logical coherence . This paper proposes a single document summarization method based on the trimming of a discourse tree . This is a two-fold process . First , we propose rules for transforming a rhetorical structure theorybased discourse tree into a dependency-based discourse tree , which allows us to take a treetrimming approach to summarization . Second , we formulate the problem of trimming a dependency-based discourse tree as a Tree Knapsack Problem , then solve it with integer linear programming ( ILP ) . Evaluation results showed that our method improved ROUGE scores . State-of-the-art extractive text summarization methods regard a document ( or a document set ) as a set of textual units ( e.g. sentences , clauses , phrases ) and formulate summarization as a combinatorial optimization problem , i.e. selecting a subset of the set of textual units that maximizes an objective without violating a length constraint . For example , Mc-Donald ( 2007 ) formulated text summarization as a Knapsack Problem , where he selects a set of textual units that maximize the sum of significance scores of each unit . Filatova et al . ( 2004 ) proposed a summarization method based on a Maximum Coverage Problem , in which they select a set of textual units that maximizes the weighted sum of the conceptual units ( e.g. unigrams ) contained in the set . Although , their greedy solution is only an approximation , Takamura et al . ( 2009a ) extended it to obtain the exact solution . More recently , Takamura et al . ( 2009b ) regarded summarization as a Budgeted Median Problem and obtain exact solutions with integer linear programming . These methods successfully improved ROUGE ( Lin , 2004 ) scores , but they still have one critical shortcoming . Since these methods are based on subset selection , the summaries they generate can not preserve the rhetorical structure of the textual units of a source document . Thus , the resulting summary may lack coherence and may not include significant textual units from a source document . One powerful and potential way to overcome the problem is to include discourse tree constraints in the summarization procedure . Marcu ( 1998 ) regarded a document as a Rhetorical Structure Theory ( RST ) ( William Charles , Mann and Sandra Annear , Thompson , 1988 ) -based discourse tree ( RST-DT ) and selected textual units according to a preference ranking derived from the tree structure to make a summary . Daumé et al . ( 2002 ) proposed a document compression method that directly models the probability of a summary given an RST-DT by using a noisy-channel model . These methods generate well-organized summaries , however , since they do not formulate summarizations as combinatorial op-Figure 1 : Example RST-DT from ( Marcu , 1998 ) . timization problems , the optimality of the generated summaries is not guaranteed . In this paper , we propose a single document summarization method based on the trimming of a discourse tree based on the Tree Knapsack Problem . If a discourse tree explicitly represents parent-child relationships between textual units , we can apply the well-known tree-trimming approach to a discourse tree and reap the benefit of combinatorial optimization methods . In other words , to apply the treetrimming approach , we need a tree whose all nodes represent textual units . Unfortunately , the RST-DT does not allow it , because textual units in the RST-DT are located only on leaf nodes and parent-child relationship between textual units are represented implicitly at higher positions in a tree . Therefore , we first propose rules that transform an RST-DT into a dependency-based discourse tree ( DEP-DT ) that explicitly defines the parent-child relationships . Second , we treat it as a rooted subtree selection , in other words , a Tree Knapsack Problem and formulate the problem as an ILP . 2 From RST-DT to DEP-DT This paper proposed rules for transforming an RST-DT to a DEP-DT to obtain the parent-child relationships between EDUs . We treated a single document summarization method as a Tree Knapsack Problem , i.e. the summarizer selects the best rooted subtree from a DEP-DT . To demonstrate the effectiveness of our method , we conducted an experimental evaluation using 30 documents selected from the RST Discourse Treebank Corpus . The results showed that our method achieved the highest ROUGE-1,2 scores .

Existing optimization-based methods for extractive summarization do not consider the rhetorical relations between textual units leading to generating uncoherent summaries or missing significant textual units.

They propose to first transform a rhetorical discourse tree into a dependency-based tree and then trim it as a Tree Knapsack Problem.

The proposed method achieves the highest ROUGE-1,2 scores on 30 documents selected from the RST Discourse Treebank Corpus.


Pre-training Transformer from large-scale raw texts and fine-tuning on the desired task have achieved state-of-the-art results on diverse NLP tasks . However , it is unclear what the learned attention captures . The attention computed by attention heads seems not to match human intuitions about hierarchical structures . This paper proposes Tree Transformer , which adds an extra constraint to attention heads of the bidirectional Transformer encoder in order to encourage the attention heads to follow tree structures . The tree structures can be automatically induced from raw texts by our proposed " Constituent Attention " module , which is simply implemented by self-attention between two adjacent words . With the same training procedure identical to BERT , the experiments demonstrate the effectiveness of Tree Transformer in terms of inducing tree structures , better language modeling , and further learning more explainable attention scores 1 . Human languages exhibit a rich hierarchical structure which is currently not exploited nor mirrored by the self-attention mechanism that is the core of the now popular Transformer architecture . Prior work that integrated hierarchical structure into neural networks either used recursive neural networks ( Tree-RNNs ) ( C. Goller and A.Kuchler , 1996 ; Socher et al . , 2011 ; Tai et al . , 2015 ) or simultaneously generated a syntax tree and language in RNN ( Dyer et al . , 2016 ) , which have shown beneficial for many downstream tasks ( Aharoni and Goldberg , 2017 ; Eriguchi et al . , 2017 ; Strubell et al . , 2018 ; Zaremoodi and Haffari , 2018 ) . Considering the requirement of the annotated parse trees and the 1 The source code is publicly available at https:// github.com / yaushian / Tree-Transformer . costly annotation effort , most prior work relied on the supervised syntactic parser . However , a supervised parser may be unavailable when the language is low-resourced or the target data has different distribution from the source domain . Therefore , the task of learning latent tree structures without human-annotated data , called grammar induction ( Carroll and Charniak , 1992 ; Klein and Manning , 2002 ; Smith and Eisner , 2005 ) , has become an important problem and attractd more attention from researchers recently . Prior work mainly focused on inducing tree structures from recurrent neural networks ( Shen et al . , 2018a , b ) or recursive neural networks ( Yogatama et al . , 2017 ; Drozdov et al . , 2019 ) , while integrating tree structures into Transformer remains an unexplored direction . Pre-training Transformer from large-scale raw texts successfully learns high-quality language representations . By further fine-tuning pre-trained Transformer on desired tasks , wide range of NLP tasks obtain the state-of-the-art results ( Radford et al . , 2019 ; Devlin et al . , 2018 ; Dong et al . , 2019 ) . However , what pre-trained Transformer self-attention heads capture remains unknown . Although an attention can be easily explained by observing how words attend to each other , only some distinct patterns such as attending previous words or named entities can be found informative ( Vig , 2019 ) . The attention matrices do not match our intuitions about hierarchical structures . In order to make the attention learned by Transformer more interpretable and allow Transformer to comprehend language hierarchically , we propose Tree Transformer , which integrates tree structures into bidirectional Transformer encoder . At each layer , words are constrained to attend to other words in the same constituents . This constraint has been proven to be effective in prior work ( Wu et al . , 2018 ) . Different from the prior work that required a supervised parser , in Tree Transformer , the constituency tree structures is automatically induced from raw texts by our proposed " Constituent Attention " module , which is simply implemented by self-attention . Motivated by Tree-RNNs , which compose each phrase and the sentence representation from its constituent sub-phrases , Tree Transformer gradually attaches several smaller constituents into larger ones from lower layers to higher layers . The contributions of this paper are 3-fold : • Our proposed Tree Transformer is easy to implement , which simply inserts an additional " Constituent Attention " module implemented by self-attention to the original Transformer encoder , and achieves good performance on the unsupervised parsing task . • As the induced tree structures guide words to compose the meaning of longer phrases hierarchically , Tree Transformer improves the perplexity on masked language modeling compared to the original Transformer . • The behavior of attention heads learned by Tree Transformer expresses better interpretability , because they are constrained to follow the induced tree structures . By visualizing the self-attention matrices , our model provides the information that better matchs the human intuition about hierarchical structures than the original Transformer . This paper proposes Tree Transformer , a first attempt of integrating tree structures into Transformer by constraining the attention heads to attend within constituents . The tree structures are automatically induced from the raw texts by our proposed Constituent Attention module , which attaches the constituents to each other by selfattention . The performance on unsupervised parsing demonstrates the effectiveness of our model in terms of inducing tree structures coherent to human expert annotations . We believe that incorporating tree structures into Transformer is an important and worth exploring direction , because it allows Transformer to learn more interpretable attention heads and achieve better language modeling . The interpretable attention can better explain how the model processes the natural language and guide the future improvement .

This is unclear what attention heads in pre-training transformers models capture and it seems not to match human intuitions about hierarchical structures.

They propose to add an extra constraint to attention heads of the bidirectional Transformer encoder and a module induces tree structures from raw texts.

The proposed model achieves better unsupervised tree structure induction, language modelling, and more explainable attention scores which are coherent to human expert annotations.


This paper employs morphological structures and relations between sentence segments for opinion analysis on words and sentences . Chinese words are classified into eight morphological types by two proposed classifiers , CRF classifier and SVM classifier . Experiments show that the injection of morphological information improves the performance of the word polarity detection . To utilize syntactic structures , we annotate structural trios to represent relations between sentence segments . Experiments show that considering structural trios is useful for sentence opinion analysis . The best f-score achieves 0.77 for opinion word extraction , 0.62 for opinion word polarity detection , 0.80 for opinion sentence extraction , and 0.54 for opinion sentence polarity detection . Sentiment analysis has attracted much attention in recent years because a large scale of subjective information is disseminated through various platforms on the web . Sentiment information can be applied to a wide variety of fields , including product recommendation , review summarization , public polling , and so on . Opinion dictionaries are important resources for identifying subjective information . Several approaches were proposed to collect such resources . Wiebe ( 2000 ) learned subjective adjectives from corpora . Takamura et al . ( 2005 ) extracted semantic orientations of words . Ku et al . ( 2007 ) measured sentiment degrees of Chinese words by averaging the sentiment scores of the composing characters . When the opinion words are available , the polarities of sentences and documents can be determined by them . Riloff and Wiebe ( 2003 ) learned the extraction patterns for subjective expressions . Kim and Hovy ( 2004 ) found the polarity of subjective expressions . Pang et al . ( 2002 ) and Dave et al . ( 2003 ) explored various techniques at document level . Morphological information has been widely used in classifying words , telling the meanings , and doing other in-depth analysis ( Tzeng and Chen , 2002 ) . However , morphological information was seldom applied either in Chinese opinion extraction , or in solving the coverage problem of opinion dictionary . Instead of bag-ofcharacters approach ( Ku et al . , 2007 ) , this paper employs morphological structures of words to extract opinion words . Relations between sentence segments are also defined by linguistics in the Chinese language . These are similar to morphological structures between Chinese characters . Based on parsing trees of sentences , we identify these relations and utilize them for opinion analysis on sentences . As the experimental corpus , some researchers managed to generate annotated materials and gold standards under many constraints . Ku set a standard for generating final answers from annotations of multiple annotators ( Ku et al . , 2007 ) , and Somasundaran annotated discourse information from meeting dialogs to train a sentiment model ( Somasundaran et al . , 2007 ) . For multilingual issues , researchers concerned mainly about the applicability of corpus and algorithms from the native language to foreign languages ( Banea et al . , 2008 ; Bautin et al . , 2008 ) . Several opinion analysis systems have been developed so far . OASYS ( Cesarano et al . , 2007 ) and CopeOpi ( Ku et al . , 2007 ) allow users input their queries and select preferred data sources , and then track opinions in a time zone . For both systems , extracting opinions is the main focus , while holders and targets are identified implicitly when retrieving relevant documents . Carenini 's team proposed a graphical user interface for evaluative texts ( 2006 ) , in which color blocks were used to present the evaluations for components of products . Fair News Reader , a Japanese news Web system , incorporates sentiment information insensibly in an interesting way ( Kawai et al . , 2007 ) . It provides readers " balanced " reports by analyzing the sentiment in news articles which readers have read , and suggests them new articles according to the analysis results . It leads the application of opinion analysis to the direction of personalization . This paper considers morphological and syntactic structures in analyzing Chinese opinion words and sentences . For morphological structures , eight Chinese morphological types are defined . CRF classifier and SVM classifier for morphological type classification are proposed . Experiments show that CRF classifier achieves the best accuracy 0.70 in type classification , which is 8 % better than SVM classifier . We further show that word morphological structures benefit the opinion word extraction significantly . With the help of the sentiment dictionary NTUSD , the f-score of opinion word extraction achieves 0.77 and the f-score of the word polarity detection achieves 0.62 when the word morphological types are provided by the SVM classifier . They are comparably better than bag-of-character approach and the dictionary based approach . We defined structural trios to represent the relations between sentence segments and also extract these relations using CRF algorithm . Results show that considering structural trios benefits the opinion analysis on sentences . An fscore 0.80 for opinion extraction and an f-score 0.54 for polarity detection are achieved , which is a great improvement . The opinion scoring functions for morphological types and structural trios are critical for polarity detection , and scoring functions for words determine the scoring functions for sentences . Now we define these functions intuitively based on linguistic rules , but learning methods like regression will be investigated in the future . Examining the interaction of cues from word and sentence levels on the opinion sentence extraction and the opinion polarity detection is our next goal .

There is not much work on applying morphological information in opinion extraction in Chinese.

They propose to utilize morphological and syntactic features for Chinese opinion analysis on word and sentence levels.

They show that using morphological structures helps opinion analysis in Chinese, outperforming the existing bat-of-character approach and the dictionary-based approach.


In this paper , we investigate the importance of social network information compared to content information in the prediction of a Twitter user 's occupational class . We show that the content information of a user 's tweets , the profile descriptions of a user 's follower / following community , and the user 's social network provide useful information for classifying a user 's occupational group . In our study , we extend an existing dataset for this problem , and we achieve significantly better performance by using social network homophily that has not been fully exploited in previous work . In our analysis , we found that by using the graph convolutional network to exploit social homophily , we can achieve competitive performance on this dataset with just a small fraction of the training data . * Equal Contribution ; work performed while both authors were visiting Singapore University of Technology and Design ( SUTD ) . T1 : Day at the races . T2 : The new pitch era starts here . Groundsman . Wolves fan . Horse racing enthusiast . A Commercial and Domestic Grounds Maintenance company T1 : Pitch at wembley looks great . T2 : Spurs will have best stadium in uk . Twitter ( http://twitter.com ) is a microblogging service launched in 2006 , where , a user can publish messages with up to 280 characters , called " tweets " . Unlike many other social networking platforms , such as Facebook and LinkedIn , Twitter does not provide structured fields for users to fill in personal information . However , a user can write a 160-character-long small public summary about itself called a " Bio " . Besides linguistic information from tweets and Bios , online social media is a rich source of network information . People 's personal networks are homogeneous , i.e. , friends share more attributes such as race , ethnicity , religion , and occupation-known as the homophily principle ( McPherson et al . , 2001 ) . Such network information has been utilized in friend recommendation ( Guy et al . , 2010 ) , community detection Follow Figure 1 : User and Network information on Twitter Microblog . ( Yang and Leskovec , 2013 ) , etc . Figure 1 shows two users connected on Twitter . By looking at their Bio and tweets , it can be inferred that these users share the same occupational interest . Profiling users can enhance service quality and improve product recommendation , and hence is a widely studied problem . User occupational class prediction is an important component of user profiling and a sub-task of user demographic feature prediction . Existing approaches to predicting Twitter users ' demographic attributes explore , select , and combine various features generated from text and network to achieve the best predictive performances in respective classification tasks ( Han et al . , 2013 ; Miller et al . , 2012 ; Preot ¸iuc-Pietro et al . , 2015 ; Huang et al . , 2015 ; Aletras and Chamberlain , 2018 ) . The three categories of features are : account level features , tweet text features , and network based features . Past research have shown the distinctive usage of language across gender , age , location , etc . in tweets ( Sloan et al . , 2015 ; Cheng et al . , 2010 ; Burger et al . , 2011 ; Rao et al . , 2010 ) , which makes content based prediction effective . As for user occupational class prediction , Preot ¸iuc-Pietro et al . ( 2015 ) built a dataset where users are assigned to hierarchical job categories . They used word cluster distribution features of content information to predict a user 's occupational group . Aletras and Chamberlain ( 2018 ) constructed a user 's followings connections to learn the user embedding as a feature input to the classification models . Considering the regional disparities of economic development stages , the major job categories may vary significantly across regions . Sloan et al . ( 2015 ) summarized occupation distribution of Twitter users in the UK by looking into their profiles . In this paper , we analyze the usefulness of a user 's network information over the user 's tweets for predicting its occupational group . We extend the existing dataset for occupation classification ( Preot ¸iuc-Pietro et al . ( 2015 ) ) by introducing the network information about a user , i.e. follower / following IDs together with their Bio descriptions , and we construct a user-centric network to extract useful community and text based features . The acquired features from the network are then exploited using a graph neural network . The obtained results show the importance of a network information over tweet information from a user for such a task . Previous works have used tweets or a fraction of the network information to extract features for occupation classification . To analyze the importance of network information , we extended an existing Twitter dataset for a user 's social media connections ( follow information ) . We showed that by using only follow information as an input to graph convolutional networks , one can achieve a significantly higher accuracy on the prediction task as compared to the existing approaches utilizing tweet-only information or partial network structure . Directions of future research include adaptation of our methods to a large scale , sparsely connected social network . One might also want to investigate the inductive settings of GCN ( Hamilton et al . , 2017 ) to predict demographic information of a user from outside the black network .

Existing systems only use limited information from the tweets network to perform occupation classification.

They extend existing Twitter occupation classification graph-based models to exploit content information by adding textual data to existing datasets.

They show that textual feature enables graph neural networks to predict Twitter user occupation well even with a limited amount of training data.


Short vowels and other diacritics are not part of written Arabic scripts . Exceptions are made for important political and religious texts and in scripts for beginning students of Arabic . Script without diacritics have considerable ambiguity because many words with different diacritic patterns appear identical in a diacritic-less setting . We propose in this paper a maximum entropy approach for restoring diacritics in a document . The approach can easily integrate and make effective use of diverse types of information ; the model we propose integrates a wide array of lexical , segmentbased and part-of-speech tag features . The combination of these feature types leads to a state-of-the-art diacritization model . Using a publicly available corpus ( LDC 's Arabic Treebank Part 3 ) , we achieve a diacritic error rate of 5.1 % , a segment error rate 8.5 % , and a word error rate of 17.3 % . In case-ending-less setting , we obtain a diacritic error rate of 2.2 % , a segment error rate 4.0 % , and a word error rate of 7.2 % . Modern Arabic written texts are composed of scripts without short vowels and other diacritic marks . This often leads to considerable ambiguity since several words that have different diacritic patterns may appear identical in a diacritic-less setting . Educated modern Arabic speakers are able to accurately restore diacritics in a document . This is based on the context and their knowledge of the grammar and the lexicon of Arabic . However , a text without diacritics becomes a source of confusion for beginning readers and people with learning disabilities . A text without diacritics is also problematic for applications such as text-to-speech or speech-to-text , where the lack of diacritics adds another layer of ambiguity when processing the data . As an example , full vocalization of text is required for text-to-speech applications , where the mapping from graphemes to phonemes is simple compared to languages such as English and French ; where there is , in most cases , one-to-one relationship . Also , using data with diacritics shows an improvement in the accuracy of speech-recognition applications ( Afify et al . , 2004 ) . Currently , text-tospeech , speech-to-text , and other applications use data where diacritics are placed manually , which is a tedious and time consuming excercise . A diacritization system that restores the diacritics of scripts , i.e. supply the full diacritical markings , would be of interest to these applications . It also would greatly benefit nonnative speakers , sufferers of dyslexia and could assist in restoring diacritics of children 's and poetry books , a task that is currently done manually . We propose in this paper a statistical approach that restores diacritics in a text document . The proposed approach is based on the maximum entropy framework where several diverse sources of information are employed . The model implicitly learns the correlation between these types of information and the output diacritics . In the next section , we present the set of diacritics to be restored and the ambiguity we face when processing a non-diacritized text . Section 3 gives a brief summary of previous related works . Section 4 presents our diacritization model ; we explain the training and decoding process as well as the different feature categories employed to restore the diacritics . Section 5 describes a clearly defined and replicable split of the LDC 's Arabic Treebank Part 3 corpus , used to built and evaluate the system , so that the reproduction of the results and future comparison can accurately be established . Section 6 presents the experimental results . Section 7 reports a comparison of our approach to the finite state machine modeling technique that showed promissing results in ( Nelken and Shieber , 2005 ) . Finally , section 8 concludes the paper and discusses future directions . We presented in this paper a statistical model for Arabic diacritic restoration . The approach we propose is based on the Maximum entropy framework , which gives the system the ability to integrate different sources of knowledge . Our model has the advantage of successfully combining diverse sources of information ranging from lexical , segment-based and POS features . Both POS and segment-based features are generated by separate statistical systems -not extracted manually -in order to simulate real world applications . The segment-based features are extracted from a statistical morphological analysis system using WFST approach and the POS features are generated by a parsing model Table 5 : Error Rate in % for n-gram diacritic restoration and segmentation using FST and Kneser-Ney LM . Columns marked with " True shadda " represent results on documents containing the original consonant doubling " shadda " while columns marked with " Predicted shadda " represent results where the system restored all diacritics including shadda . that also uses Maximum entropy framework . Evaluation results show that combining these sources of information lead to state-of-the-art performance . As future work , we plan to incorporate Buckwalter morphological analyzer information to extract new features that reduce the search space . One idea will be to reduce the search to the number of hypotheses , if any , proposed by the morphological analyzer . We also plan to investigate additional conjunction features to improve the accuracy of the model .

Short vowels and other diacritics are not expressed in written Arabic, making it difficult to read for beginner readers or system developments.

They propose an approach that uses maximum entropy to restore diacritics in Arabic documents by learning relations between a wide varieties of features.

They show that by taking various kinds of features their system outperforms the existing state-of-the-art decritization model.


Current state-of-the-art systems for joint entity relation extraction ( Luan et al . , 2019 ; Wadden et al . , 2019 ) usually adopt the multi-task learning framework . However , annotations for these additional tasks such as coreference resolution and event extraction are always equally hard ( or even harder ) to obtain . In this work , we propose a pre-training method ENPAR to improve the joint extraction performance . EN-PAR requires only the additional entity annotations that are much easier to collect . Unlike most existing works that only consider incorporating entity information into the sentence encoder , we further utilize the entity pair information . Specifically , we devise four novel objectives , i.e. , masked entity typing , masked entity prediction , adversarial context discrimination , and permutation prediction , to pretrain an entity encoder and an entity pair encoder . Comprehensive experiments show that the proposed pre-training method achieves significant improvement over BERT on ACE05 , SciERC , and NYT , and outperforms current state-of-the-art on ACE05 . Joint extraction of entities and relations is a fundamental task in information extraction , it aims to extract entities and relations with a unified model . Current approaches ( Luan et al . , 2019 ; Wadden et al . , 2019 ) usually adopt the multi-task learning framework that optimizes many objectives simultaneously , including entity recognition , relation extraction , coreference resolution , and event extraction . However , as large-scale manually labeled data required by these methods is unavailable in many domains , their applicability is severely restricted . Therefore , we expect to catch or even surpass the multi-task based joint models with less annotation cost . Compared with the annotations of coreference resolution and event extraction , entity annotations can be easily obtained through automatic NER annotation tools ( e.g. , spaCy1 ) . In this paper , we focus on improving the model 's performance with just extra entity annotations . Although pre-trained models , like BERT , have shown impressive performance in many downstream tasks , they have mainly two limitations when applied in the joint entity relation extraction task . One is that currently pre-trained objectives are insufficient for this task . Specifically , these commonly used universal pre-trained model ( e.g. , BERT ) do not consider the entity-related knowledge that is crucial for better extracting entities and relations . The other is that these models only provide pre-trained representations for tokens and sentences , but not entities and entity pairs . To obtain the representations for entities and entity pairs , additional parameters that are not pre-trained are introduced in the fine-tuning stage , which may futher impair the joint extraction performance . To address the first limitation , recent several works try to incorporate entity-related information into pre-training objectives . Zhang et al . ( 2019 ) fuses heterogeneous information from both texts and knowledge graphs and proposes a denoising entity auto-encoder objective based on BERT . Sun et al . ( 2019c ) presents two knowledge masking strategies in the pre-training stage ( entity-level masking and phrase-level masking ) . Both of them utilize extra entity annotations ( i.e. , entities in knowledge graphs and automatic entity annotations , respectively ) . In this paper , we follow this line of works and build a large-scale entity annotated corpus using the spaCy NER tool . For the second limitation , we propose ENPAR , a pre-training method customized for entity relation extraction . ENPAR consists of an underlying sentence encoder , an entity encoder , and an entity pair encoder . Compared with BERT ( Figure 1 ( a ) ) , the proposed entity encoder and entity pair encoder directly provide representations of entities and entity pairs . To train the three encoders , we devise four novel pre-training tasks : masked entity typing , masked entity prediction , adversarial context discrimination and permutation prediction ( Figure 1(b ) ) . In the first two tasks , we randomly mask some entity words and then predict the masked tokens and the entity type . These two tasks are natural extensions of the masked language model . To learn a better entity pair encoder , we draw inspirations from the denoising auto-encoder ( Zhang et al . , 2019 ) and propose the last two tasks . Specifically , when the entity pair or its context in a sentence are perturbed , we hope that the entity pair encoder is capable of tracking such changes . We employ the parameter-sharing method for these four tasks and train these objectives jointly . To sum up , our main contributions are as follows2 : • We introduce an entity encoder and an entity pair encoder to incorporate not only the entity information but also the entity pair information , which were ignored in current universal pre-trained models . • We propose four novel pre-training tasks that help to learn the proposed encoders . These tasks only require additional entity annotations ( with commonly used entity types ) , which can be automatically generated by public annotation tools , such as spaCy NER . • We conduct comprehensive experiments and demonstrate that the proposed method achieves significant improvement on ACE05 and NYT dataset and is comparable with the state-of-the-art on the SciERC dataset . We propose ENPAR , a pre-training method customized for entity relation extraction only with additional entity annotations . Instead of only pretraining sentence encoder in universal pre-trained models , we also pre-train an entity encoder and an entity pair encoder . Then the proposed four objec-tives can incorporate entity and entity-pair knowledge into the pre-trained encoders to enhance the encoders ' representations . Experiments on three datasets demonstrate that ENPAR achieves comparable or even superior performances compared with multi-task based joint models .

Current joint entity relation extraction models follow a multitask learning setup however datasets with multiple types of annotation are not available for many domains.

They propose to pre-train a language model entity relation extraction with four newly introduced objective functions which utilize automatically obtained annotations by NER models.

The models pre-trained by the proposed method significantly outperform BERT and current state-of-the-art models on three entity relation extraction datasets.


We present the Siamese Continuous Bag of Words ( Siamese CBOW ) model , a neural network for efficient estimation of highquality sentence embeddings . Averaging the embeddings of words in a sentence has proven to be a surprisingly successful and efficient way of obtaining sentence embeddings . However , word embeddings trained with the methods currently available are not optimized for the task of sentence representation , and , thus , likely to be suboptimal . Siamese CBOW handles this problem by training word embeddings directly for the purpose of being averaged . The underlying neural network learns word embeddings by predicting , from a sentence representation , its surrounding sentences . We show the robustness of the Siamese CBOW model by evaluating it on 20 datasets stemming from a wide variety of sources . Word embeddings have proven to be beneficial in a variety of tasks in NLP such as machine translation ( Zou et al . , 2013 ) , parsing ( Chen and Manning , 2014 ) , semantic search ( Reinanda et al . , 2015 ; Voskarides et al . , 2015 ) , and tracking the meaning of words and concepts over time ( Kim et al . , 2014 ; Kenter et al . , 2015 ) . It is not evident , however , how word embeddings should be combined to represent larger pieces of text , like sentences , paragraphs or documents . Surprisingly , simply averaging word embeddings of all words in a text has proven to be a strong baseline or feature across a multitude of tasks ( Faruqui et al . , 2014 ; Yu et al . , 2014 ; Gershman and Tenenbaum , 2015 ; Kenter and de Rijke , 2015 ) . Word embeddings , however , are not optimized specifically for representing sentences . In this paper we present a model for obtaining word embeddings that are tailored specifically for the task of averaging them . We do this by directly including a comparison of sentence embeddings-the averaged embeddings of the words they contain-in the cost function of our network . Word embeddings are typically trained in a fast and scalable way from unlabeled training data . As the training data is unlabeled , word embeddings are usually not task-specific . Rather , word embeddings trained on a large training corpus , like the ones from ( Collobert and Weston , 2008 ; Mikolov et al . , 2013b ) are employed across different tasks ( Socher et al . , 2012 ; Kenter and de Rijke , 2015 ; Hu et al . , 2014 ) . These two qualities-(i ) being trainable from large quantities of unlabeled data in a reasonable amount of time , and ( ii ) robust performance across different tasks-are highly desirable and allow word embeddings to be used in many large-scale applications . In this work we aim to optimize word embeddings for sentence representations in the same manner . We want to produce general purpose sentence embeddings that should score robustly across multiple test sets , and we want to leverage large amounts of unlabeled training material . In the word2vec algorithm , Mikolov et al . ( 2013a ) construe a supervised training criterion for obtaining word embeddings from unsupervised data , by predicting , for every word , its surrounding words . We apply this strategy at the sentence level , where we aim to predict a sentence from its adjacent sentences ( Kiros et al . , 2015 ; Hill et al . , 2016 ) . This allows us to use unlabeled training data , which is easy to obtain ; the only restriction is that documents need to be split into sentences and that the order between sentences is preserved . The main research question we address is whether directly optimizing word embeddings for the task of being averaged to produce sentence embeddings leads to word embeddings that are better suited for this task than word2vec does . Therefore , we test the embeddings in an unsupervised learning scenario . We use 20 evaluation sets that stem from a wide variety of sources ( newswire , video descriptions , dictionary descriptions , microblog posts ) . Furthermore , we analyze the time complexity of our method and compare it to our baselines methods . Summarizing , our main contributions are : We have presented Siamese CBOW , a neural network architecture that efficiently learns word embeddings optimized for producing sentence representations . The model is trained using only unla-beled text data . It predicts , from an input sentence representation , the preceding and following sentence . We evaluated the model on 20 test sets and show that in a majority of cases , 14 out of 20 , Siamese CBOW outperforms a word2vec baseline and a baseline based on the recently proposed skip-thought architecture . As further analysis on various choices of parameters show that the method is stable across settings , we conclude that Siamese CBOW provides a robust way of generating high-quality sentence representations . Word and sentence embeddings are ubiquitous and many different ways of using them in supervised tasks have been proposed . It is beyond the scope of this paper to provide a comprehensive analysis of all supervised methods using word or sentence embeddings and the effect Siamese CBOW would have on them . However , it would be interesting to see how Siamese CBOW embeddings would affect results in supervised tasks . Lastly , although we evaluated Siamese CBOW on sentence pairs , there is no theoretical limitation restricting it to sentences . It would be interesting to see how embeddings for larger pieces of texts , such as documents , would perform in document clustering or filtering tasks .

While an average of word embeddings has proven to be successful as sentence-level representations, it is suboptimal because they are not optimized to represent sentences.

They propose to train word embeddings directly for the purpose of being averaged by predicting sounding sentences from a sentence representation using unlabeled data.

Evaluations show that their word embeddings outperform existing methods in 14 out of 20 datasets and they are stable in choice of parameters.


We propose a new framework for abstractive text summarization based on a sequence-to-sequence oriented encoderdecoder model equipped with a deep recurrent generative decoder ( DRGN ) . Latent structure information implied in the target summaries is learned based on a recurrent latent random model for improving the summarization quality . Neural variational inference is employed to address the intractable posterior inference for the recurrent latent variables . Abstractive summaries are generated based on both the generative latent variables and the discriminative deterministic states . Extensive experiments on some benchmark datasets in different languages show that DRGN achieves improvements over the state-ofthe-art methods . Automatic summarization is the process of automatically generating a summary that retains the most important content of the original text document ( Edmundson , 1969 ; Luhn , 1958 ; Nenkova and McKeown , 2012 ) . Different from the common extraction-based and compression-based methods , abstraction-based methods aim at constructing new sentences as summaries , thus they require a deeper understanding of the text and the capability of generating new sentences , which provide an obvious advantage in improving the focus of a summary , reducing the redundancy , and keeping a good compression rate ( Bing et al . , 2015 ; Rush et al . , 2015 ; Nallapati et al . , 2016 ) . * The work described in this paper is supported by a grant from the Grant Council of the Hong Kong Special Administrative Region , China ( Project Code : 14203414 ) . Some previous research works show that human-written summaries are more abstractive ( Jing and McKeown , 2000 ) . Moreover , our investigation reveals that people may naturally follow some inherent structures when they write the abstractive summaries . To illustrate this observation , we show some examples in Figure 1 , which are some top story summaries or headlines from the channel " Technology " of CNN . After analyzing the summaries carefully , we can find some common structures from them , such as " What " , " What-Happened " , " Who Action What " , etc . For example , the summary " Apple sues Qualcomm for nearly $ 1 billion " can be structuralized as " Who ( Apple ) Action ( sues ) What ( Qualcomm ) " . Similarly , the summaries " [ Twitter ] [ fixes ] [ botched @POTUS account transfer ] " , " [ Uber ] [ to pay ] [ $ 20 million ] for misleading drivers " , and " [ Bipartisan bill ] aims to [ reform ] [ H-1B visa system ] " also follow the structure of " Who Action What " . The summary " The emergence of the ' cyber cold war " ' matches with the structure of " What " , and the summary " St. Louis ' public library computers hacked " follows the structure of " What-Happened " . Intuitively , if we can incorporate the latent structure information of summaries into the abstractive summarization model , it will improve the quality of the generated summaries . However , very few existing works specifically consider the latent structure information of summaries in their summarization models . Although a very popular neural network based sequence-to-sequence ( seq2seq ) framework has been proposed to tackle the abstractive summarization problem ( Lopyrev , 2015 ; Rush et al . , 2015 ; Nallapati et al . , 2016 ) , the calculation of the internal decoding states is entirely deterministic . The deterministic transformations in these discriminative models lead to limitations on the representation ability of the latent structure information . Miao and Blunsom ( 2016 ) extended the seq2seq framework and proposed a generative model to capture the latent summary information , but they did not consider the recurrent dependencies in their generative model leading to limited representation ability . To tackle the above mentioned problems , we design a new framework based on sequenceto-sequence oriented encoder-decoder model equipped with a latent structure modeling component . We employ Variational Auto-Encoders ( VAEs ) ( Kingma and Welling , 2013 ; Rezende et al . , 2014 ) as the base model for our generative framework which can handle the inference problem associated with complex generative modeling . However , the standard framework of VAEs is not designed for sequence modeling related tasks . Inspired by ( Chung et al . , 2015 ) , we add historical dependencies on the latent variables of VAEs and propose a deep recurrent generative decoder ( DRGD ) for latent structure modeling . Then the standard discriminative deterministic decoder and the recurrent generative decoder are integrated into a unified decoding framework . The target summaries will be decoded based on both the discriminative deterministic variables and the generative latent structural information . All the neural parameters are learned by back-propagation in an end-to-end training paradigm . The main contributions of our framework are summarized as follows : ( 1 ) We propose a sequence-to-sequence oriented encoder-decoder model equipped with a deep recurrent generative decoder ( DRGD ) to model and learn the latent structure information implied in the target summaries of the training data . Neural variational inference is employed to address the intractable posterior inference for the recurrent latent variables . ( 2 ) Both the generative latent structural information and the discriminative deterministic variables are jointly considered in the generation process of the abstractive summaries . ( 3 ) Experimental results on some benchmark datasets in different languages show that our framework achieves better performance than the state-of-the-art models . We propose a deep recurrent generative decoder ( DRGD ) to improve the abstractive summarization performance . The model is a sequenceto-sequence oriented encoder-decoder framework equipped with a latent structure modeling component . Abstractive summaries are generated based on both the latent variables and the deterministic states . Extensive experiments on benchmark

Although humans follow inherent structures in summary writing, currently there are no abstractive summarization models which take latent structure information and recurrent dependencies into account.

They propose a Variational Auto-Encoder-based sequence-to-sequence oriented encoder-decoder model with a deep recurrent generative decoder which learns latent structure information implied in the target summaries.

The proposed model outperforms the state-of-the-art models on some datasets in different languages.


Multi-layer multi-head self-attention mechanism is widely applied in modern neural language models . Attention redundancy has been observed among attention heads but has not been deeply studied in the literature . Using BERT-base model as an example , this paper provides a comprehensive study on attention redundancy which is helpful for model interpretation and model compression . We analyze the attention redundancy with Five-Ws and How . ( What ) We define and focus the study on redundancy matrices generated from pre-trained and fine-tuned BERT-base model for GLUE datasets . ( How ) We use both token-based and sentence-based distance functions to measure the redundancy . ( Where ) Clear and similar redundancy patterns ( cluster structure ) are observed among attention heads . ( When ) Redundancy patterns are similar in both pre-training and fine-tuning phases . ( Who ) We discover that redundancy patterns are task-agnostic . Similar redundancy patterns even exist for randomly generated token sequences . ( " Why " ) We also evaluate influences of the pre-training dropout ratios on attention redundancy . Based on the phaseindependent and task-agnostic attention redundancy patterns , we propose a simple zero-shot pruning method as a case study . Experiments on fine-tuning GLUE tasks verify its effectiveness . The comprehensive analyses on attention redundancy make model understanding and zero-shot model pruning promising . Multi-layer multi-head self-attention architectures ( Transformer ( Vaswani et al . , 2017 ) ) are widely applied in modern language models , such as BERT ( Devlin et al . , 2019 ) , RoBERTa ( Liu et al . , 2019 ) , OpenAI GPT ( Radford et al . , 2018 ) , GPT-2 ( Radford et al . , 2019 ) and ERNIE2.0 ( Sun et al . , 2019 ) , to name a few . Redundancy phenomenon is discovered among attention heads . It demonstrates that many attention heads generate very similar attention matrices ( Clark et al . , 2019 ; Kovaleva et al . , 2019 ) . We take the pre-trained BERT-base model as an example . It learns 12-layer-12-head self-attention matrices describing dependencies between each pair of tokens in a sentence . Then for each token , there are 144 attention vectors . We use Jensen-Shannon distance to measure the relationship between each pair of vectors . Then for one sentence ( consisting of a sequence of tokens ) , the token-averaged distance is utilized to imply the redundancy between each pair of attention matrices . Smaller distance values reflect more redundancy . Figure 1 shows the redundancy ( distance ) among 144 × 144 pairs of attention matrices averaged over 1000 randomly sampled sentences . We can see clear redundancy patterns ( clusters with smaller distance areas ) in consecutive attention layers . Analyzing the attention redundancy helps to interpret the multi-layer multi-head self-attention architecture . Various studies have attempted to re-veal the relationship among attention heads . Examples are attention visualization ( Vig and Belinkov , 2019 ) , common attention patterns ( Kovaleva et al . , 2019 ) , attention head pruning ( Voita et al . , 2019 ) , and probing test ( Clark et al . , 2019 ) . Existing works either focus on the 12 × 12 attention matrices and their effects on ( pre-training or / and finetuning ) performances or focus on linguistic features extracted by latent token vectors and attention matrices . Though the redundancy phenomenon was discovered , no existing work studies the attention redundancy pattern itself ( i.e. , the 144 × 144 distance matrix in Figure 1 ) deeply . This motivates us to conduct a comprehensive and complementary study on the attention redundancy phenomenon . In this paper , we take the BERT-base model as a representative model to analyze the attention redundancy with Five Ws and How . As far as we know , many of the following discoveries are new to the research community . What is attention redundancy ? Given a distance function , we define the pairwise distance matrix ( ∈ R 144×144 ) of the 12 × 12 attention matrices of BERT-base model as attention redundancy matrix . In this paper , we obtain redundancy matrices from both pre-trained and finetuned BERT-base model for GLUE tasks as the research objects . How to measure attention redundancy ? Except for the two token-based measures , Jensen-Shannon distance ( Clark et al . , 2019 ) and cosine similarity ( Kovaleva et al . , 2019 ) used in literature , we employ two more token-based distance function and three sentence-based ones to measure attention redundancy and analyze their similar redundancy patterns ( please refer to Section 4.1 for more details ) . The purpose is to alleviate the measuring bias of just using one distance function . Sentence-based distances directly measure the relationship between two attention matrices without averaging over tokens . We visualize the redundancy patterns using various distance functions . Where does attention redundancy exist ? We find common hierarchical cluster structures in the set of token-based redundancy matrices and the set of sentence-based redundancy matrices , respectively . Attention heads of earlier , middle , and deeper attention layers are clearly clustered in the redundancy matrices . We also demonstrate that highly correlated similar redundancy patterns exist in redundancy matrices generated based on different type of distances . When does attention redundancy occur ? The redundancy is phase-independent . Common redundancy patterns are discovered in both the pre-trained phase and fine-tuned phases . For any downstream task with any distance function , we notice highly correlated attention redundancy patterns between two phases . Who ( which task ) has attention redundancy ? We surprisingly realize that the redundancy is task-agnostic . The redundancy patterns are highly correlated across different tasks . We even randomly generate token sequences as input in the pre-trained BERT-base model . Very similar attention redundancy patterns occur as well . Based on this astonishing discovery , as a case study application , we propose a simple zero-shot head-pruning strategy based on clustering results using redundancy matrices . Compared to other complex pruning strategies , e.g. , ( Tang et al . , 2019 ; Jiao et al . , 2019 ; Fan et al . , 2019 ; Wang et al . , 2019 ; McCarley , 2019 ) , the most important is that without knowing any data of fine-tuning tasks , this pruning can be effectively and efficiently conducted just based on some randomly generated token sequences with the pre-trained BERT-base model . The only effort is to compute one or several redundancy matrices . Results reflect that for most GLUE tasks , the proposed pruning strategy based on redundancy matrices can prune up to 75 % to 85 % of attention heads while keeping comparable fine-tuning performances . " Why " does the phase-independent and taskagnostic attention redundancy happen ? It 's hard to tell the reason of the redundancy patterns ( that 's why we use the quoted " Why " ) . However , we conduct experiments to evaluate the effects on attention redundancy of dropout ratios in the pre-training phase which are suspected as one of reasons ( Clark et al . , 2019 ) . When we use sentence-based distance , a monotonic trend is found . Attention heads tend to be more redundant when increasing dropout ratios . When we use token-based distances , a complex " N"-shape effect exists . We also notice that the redundancy is more sensitive to dropouts in hidden linear transformations than to dropouts in the self-attention mechanism . We believe that above-mentioned new findings in this paper make the redundancy analyses a promis-ing research direction in model interpretation and model compression , and so on . Using BERT-base model as an example , we comprehensively investigated the attention redundancy in multi-layer multi-head self-attention based language models . The redundancy was measured by distance functions at token level and sentence level . At both levels , we found that many heads are not distinct from each other , and clear clustering effects were observed . We discovered that the attention redundancy is phase-independent and taskagnostic . Specifically , compared to a pre-trained model , the redundancy patterns do not change much after fine-tuning on multiple downstream tasks . We also shown complex influences on redundancy of dropout ratios in hidden transformations and self-attention . Based on these discoveries , we design a zero-shot strategy to prune attention heads . Compared to existing methods , the zero-shot pruning is simple and robust ( task-agnostic ) . In the near future , we are interested in experimenting this method over more self-attention based pre-trained language models and more downstream tasks . This is the appendix for NAACL-HLT 2021 paper : Yuchen Bian , Jiaji Huang , Xingyu Cai , Jiahong Yuan , and Kenneth Church . On Attention Redundancy : A Comprehensive Study .

While there are works that report a redundancy among attention heads in modern language models, no works investigate its pattern deeply.

They perform token and sentence level analysis on redundancy matrices from pre-trained and fine-tuned BERT-base models and further propose a pruning method based on findings.

They find that many heads are redundant regardless of phase and task, and show the proposed pruning method can perform robustly.


Comprehending lyrics , as found in songs and poems , can pose a challenge to human and machine readers alike . This motivates the need for systems that can understand the ambiguity and jargon found in such creative texts , and provide commentary to aid readers in reaching the correct interpretation . We introduce the task of automated lyric annotation ( ALA ) . Like text simplification , a goal of ALA is to rephrase the original text in a more easily understandable manner . However , in ALA the system must often include additional information to clarify niche terminology and abstract concepts . To stimulate research on this task , we release a large collection of crowdsourced annotations for song lyrics . We analyze the performance of translation and retrieval models on this task , measuring performance with both automated and human evaluation . We find that each model captures a unique type of information important to the task . Song lyrics and poetry often make use of ambiguity , symbolism , irony , and other stylistic elements to evoke emotive responses . These characteristics sometimes make it challenging to interpret obscure lyrics , especially for readers or listeners who are unfamiliar with the genre . To address this problem , several online lyric databases have been created where users can explain , contextualize , or discuss lyrics . Examples include MetroLyrics1 and Genius.com2 . We refer to such How does it feel ? To be without a home Like a complete unknown , We presented and released the Genius dataset to study the task of Automated Lyric Annotation . As a first investigation , we studied automatic generation of context independent annotations as machine translation and information retrieval . Our baseline system tests indicate that our corpus is suitable to train machine translation systems . Standard SMT models are capable of rephrasing and simplifying song lyrics but tend to keep close to the structure of the song lyric . Seq2Seq models demonstrated potential to generate more fluent and informative text , dissimilar to the lyric . A large fraction of the annotations is heavily based on context and background knowledge ( CS ) , one of their most appealing aspects . As future work we suggest injection of structured and unstructured external knowledge ( Ahn et al . , 2016 ) and explicit modeling of references ( Yang et al . , 2016 ) .

Because of its creative nature, understanding lyrics can be challenging both for humans and machines.

They propose a task of automated lyric annotation with a dataset collected from an online platform which explains lyrics to readers.

They evaluate translation and retrieval models with automatic and human evaluation and show that different models capture different aspects well.


Most statistical translation systems are based on phrase translation pairs , or " blocks " , which are obtained mainly from word alignment . We use blocks to infer better word alignment and improved word alignment which , in turn , leads to better inference of blocks . We propose two new probabilistic models based on the innerouter segmentations and use EM algorithms for estimating the models ' parameters . The first model recovers IBM Model-1 as a special case . Both models outperform bidirectional IBM Model-4 in terms of word alignment accuracy by 10 % absolute on the F-measure . Using blocks obtained from the models in actual translation systems yields statistically significant improvements in Chinese-English SMT evaluation . Today 's statistical machine translation systems rely on high quality phrase translation pairs to acquire state-of-the-art performance , see ( Koehn et al . , 2003 ; Zens and Ney , 2004 ; Och and Ney , 2003 ) . Here , phrase pairs , or " blocks " are obtained automatically from parallel sentence pairs via the underlying word alignments . Word alignments traditionally are based on IBM Models 1 - 5 ( Brown et al . , 1993 ) or on HMMs ( Vogel et al . , 1996 ) . Automatic word alignment is challenging in that its accuracy is not yet close to inter-annotator agreement in some language pairs : for Chinese-English , inter-annotator agreement exceeds 90 on F-measure whereas IBM Model-4 or HMM accuracy is typically below 80s . HMMs assume that words " close-in-source " are aligned to words " close-in-target " . While this locality assumption is generally sound , HMMs do have limitations : the self-transition probability of a state ( word ) is the only control on the duration in the state , the length of the phrase aligned to the word . Also there is no natural way to control repeated non-contiguous visits to a state . Despite these problems , HMMs remain attractive for their speed and reasonable accuracy . We propose a new method for localizing word alignments . We use blocks to achieve locality in the following manner : a block in a sentence pair is a source phrase aligned to a target phrase . We assume that words inside the source phrase can not align to words outside the target phrase and that words outside the source phrase can not align to words inside the target phrase . Furthermore , a block divides the sentence pair into two smaller regions : the inner part of the block , which corresponds to the source and target phrase in the block , and the outer part of the block , which corresponds to the remaining source and target words in the parallel sentence pair . The two regions are non-overlapping ; and each of them is shorter than the original parallel sentence pair . The regions are thus easier to align than the original sentence pairs ( e.g. , using IBM Model-1 ) . While the model uses a single block to split the sentence pair into two independent regions , it is not clear which block we should select for this purpose . Therefore , we treat the splitting block as a hidden variable . This proposed approach is far simpler than treating the entire sentence as a sequence of nonoverlapping phrases ( or chunks ) and considering such sequential segmentation either explicitly or implicitly . For example , ( Marcu and Wong , 2002 ) for a joint phrase based model , ( Huang et al . , 2003 ) for a translation memory system ; and ( Watanabe et al . , 2003 ) for a complex model of insertion , deletion and head-word driven chunk reordering . Other approaches including ( Watanabe et al . , 2002 ) treat extracted phrase-pairs as new parallel data with limited success . Typically , they share a similar architecture of phrase level segmentation , reordering , translation as in ( Och and Ney , 2002 ; Koehn and Knight , 2002 ; Yamada and Knight , 2001 ) . The phrase level interaction has to be taken care of for the non-overlapping sequential segmentation in a complicated way . Our models model such interactions in a soft way . The hidden blocks are allowed to overlap with each other , while each block induced two non-overlapping regions , i.e. the model brackets the sentence pair into two independent parts which are generated synchronously . In this respect , it resembles bilingual bracketing ( Wu , 1997 ) , but our model has more lexical items in the blocks with many-to-many word alignment freedom in both inner and outer parts . We present our localization constraints using blocks for word alignment in Section 2 ; we detail our two new probabilistic models and their EM training algorithms in Section 3 ; our baseline system , a maximum-posterior inference for word alignment , is explained in Section 4 ; experimental results of alignments and translations are in Section 5 ; and Section 6 contains discussion and conclusions . Our main contributions are two novel Inner-Outer Bracket models based on segmentations induced by hidden blocks . Modeling the Inner-Outer hidden segmentations , we get significantly improved word alignments for both the small training set and the large training set over the widely-practiced bidirectional IBM Model-4 alignment . We also show significant improvements in translation quality using our proposed bracket models . Robustness to noisy blocks merits further investigation .

Automatic word alignment used in statistical machine translations, does not achieve satisfactory performance in some language pairs such as because of the limitations of HMMs.

They propose to use phrase translation pairs to get better word alignments using two new probabilistic models based on EM-algorithm that localizes the alignments.

The proposed models outperform IBM Model-4 by 10%, both on small and large training setups, and the translation models based on this result improve qualities.


We propose an approach for extracting relations between entities from biomedical literature based solely on shallow linguistic information . We use a combination of kernel functions to integrate two different information sources : ( i ) the whole sentence where the relation appears , and ( ii ) the local contexts around the interacting entities . We performed experiments on extracting gene and protein interactions from two different data sets . The results show that our approach outperforms most of the previous methods based on syntactic and semantic information . Information Extraction ( IE ) is the process of finding relevant entities and their relationships within textual documents . Applications of IE range from Semantic Web to Bioinformatics . For example , there is an increasing interest in automatically extracting relevant information from biomedical literature . Recent evaluation campaigns on bio-entity recognition , such as BioCreAtIvE and JNLPBA 2004 shared task , have shown that several systems are able to achieve good performance ( even if it is a bit worse than that reported on news articles ) . However , relation identification is more useful from an applicative perspective but it is still a considerable challenge for automatic tools . In this work , we propose a supervised machine learning approach to relation extraction which is applicable even when ( deep ) linguistic processing is not available or reliable . In particular , we explore a kernel-based approach based solely on shallow linguistic processing , such as tokeniza-tion , sentence splitting , Part-of-Speech ( PoS ) tagging and lemmatization . Kernel methods ( Shawe-Taylor and Cristianini , 2004 ) show their full potential when an explicit computation of the feature map becomes computationally infeasible , due to the high or even infinite dimension of the feature space . For this reason , kernels have been recently used to develop innovative approaches to relation extraction based on syntactic information , in which the examples preserve their original representations ( i.e. parse trees ) and are compared by the kernel function ( Zelenko et al . , 2003 ; Culotta and Sorensen , 2004 ; Zhao and Grishman , 2005 ) . Despite the positive results obtained exploiting syntactic information , we claim that there is still room for improvement relying exclusively on shallow linguistic information for two main reasons . First of all , previous comparative evaluations put more stress on the deep linguistic approaches and did not put as much effort on developing effective methods based on shallow linguistic information . A second reason concerns the fact that syntactic parsing is not always robust enough to deal with real-world sentences . This may prevent approaches based on syntactic features from producing any result . Another related issue concerns the fact that parsers are available only for few languages and may not produce reliable results when used on domain specific texts ( as is the case of the biomedical literature ) . For example , most of the participants at the Learning Language in Logic ( LLL ) challenge on Genic Interaction Extraction ( see Section 4.2 ) were unable to successfully exploit linguistic information provided by parsers . It is still an open issue whether the use of domainspecific treebanks ( such as the Genia treebank 1 ) can be successfully exploited to overcome this problem . Therefore it is essential to better investigate the potential of approaches based exclusively on simple linguistic features . In our approach we use a combination of kernel functions to represent two distinct information sources : the global context where entities appear and their local contexts . The whole sentence where the entities appear ( global context ) is used to discover the presence of a relation between two entities , similarly to what was done by Bunescu and Mooney ( 2005b ) . Windows of limited size around the entities ( local contexts ) provide useful clues to identify the roles of the entities within a relation . The approach has some resemblance with what was proposed by Roth and Yih ( 2002 ) . The main difference is that we perform the extraction task in a single step via a combined kernel , while they used two separate classifiers to identify entities and relations and their output is later combined with a probabilistic global inference . We evaluated our relation extraction algorithm on two biomedical data sets ( i.e. the AImed corpus and the LLL challenge data set ; see Section 4 ) . The motivations for using these benchmarks derive from the increasing applicative interest in tools able to extract relations between relevant entities in biomedical texts and , consequently , from the growing availability of annotated data sets . The experiments show clearly that our approach consistently improves previous results . Surprisingly , it outperforms most of the systems based on syntactic or semantic information , even when this information is manually annotated ( i.e. the LLL challenge ) . The good results obtained using only shallow linguistic features provide a higher baseline against which it is possible to measure improvements obtained using methods based on deep linguistic processing . In the near future , we plan to extend our work in several ways . First , we would like to evaluate the contribution of syntactic information to relation extraction from biomedical literature . With this aim , we will integrate the output of a parser ( possibly trained on a domain-specific resource such the Genia Treebank ) . Second , we plan to test the portability of our model on ACE and MUC data sets . Third , we would like to use a named entity recognizer instead of assuming that entities are already extracted or given by a dictionary . Our long term goal is to populate databases and ontologies by extracting information from large text collections such as Medline .

Deep linguistic features obtained by parsers are not always robust and available for limited languages and domains, however; applications of shallow features are under investigated.

They propose an approach for entity relation extraction using shallow linguistic information such as tokenization, sentence splitting, Part-of-Speech tagging and lemmatization coupled with kernel functions.

Evaluations of two biomedical datasets show that the proposed method outperforms existing systems which depend on syntactic or manually annotated semantic information.