Pre-training Transformer from large-scale raw texts and fine-tuning on the desired task have achieved state-of-the-art results on diverse NLP tasks .

However , it is unclear what the learned attention captures .

The attention computed by attention heads seems not to match human intuitions about hierarchical structures .

This paper proposes Tree Transformer , which adds an extra constraint to attention heads of the bidirectional Transformer encoder in order to encourage the attention heads to follow tree structures .

The tree structures can be automatically induced from raw texts by our proposed " Constituent Attention " module , which is simply implemented by self-attention between two adjacent words .

With the same training procedure identical to BERT , the experiments demonstrate the effectiveness of Tree Transformer in terms of inducing tree structures , better language modeling , and further learning more explainable attention scores 1 .

Human languages exhibit a rich hierarchical structure which is currently not exploited nor mirrored by the self-attention mechanism that is the core of the now popular Transformer architecture .

Prior work that integrated hierarchical structure into neural networks either used recursive neural networks ( Tree-RNNs ) ( C. Goller and A.Kuchler , 1996 ; Socher et al . , 2011 ; Tai et al . , 2015 ) or simultaneously generated a syntax tree and language in RNN ( Dyer et al . , 2016 ) , which have shown beneficial for many downstream tasks ( Aharoni and Goldberg , 2017 ; Eriguchi et al . , 2017 ; Strubell et al . , 2018 ; Zaremoodi and Haffari , 2018 ) .

Considering the requirement of the annotated parse trees and the 1 The source code is publicly available at https:// github.com / yaushian / Tree-Transformer .

costly annotation effort , most prior work relied on the supervised syntactic parser .

However , a supervised parser may be unavailable when the language is low-resourced or the target data has different distribution from the source domain .

Therefore , the task of learning latent tree structures without human-annotated data , called grammar induction ( Carroll and Charniak , 1992 ; Klein and Manning , 2002 ; Smith and Eisner , 2005 ) , has become an important problem and attractd more attention from researchers recently .

Prior work mainly focused on inducing tree structures from recurrent neural networks ( Shen et al . , 2018a , b ) or recursive neural networks ( Yogatama et al . , 2017 ; Drozdov et al . , 2019 ) , while integrating tree structures into Transformer remains an unexplored direction .

Pre-training Transformer from large-scale raw texts successfully learns high-quality language representations .

By further fine-tuning pre-trained Transformer on desired tasks , wide range of NLP tasks obtain the state-of-the-art results ( Radford et al . , 2019 ; Devlin et al . , 2018 ; Dong et al . , 2019 ) .

However , what pre-trained Transformer self-attention heads capture remains unknown .

Although an attention can be easily explained by observing how words attend to each other , only some distinct patterns such as attending previous words or named entities can be found informative ( Vig , 2019 ) .

The attention matrices do not match our intuitions about hierarchical structures .

In order to make the attention learned by Transformer more interpretable and allow Transformer to comprehend language hierarchically , we propose Tree Transformer , which integrates tree structures into bidirectional Transformer encoder .

At each layer , words are constrained to attend to other words in the same constituents .

This constraint has been proven to be effective in prior work ( Wu et al . , 2018 ) .

Different from the prior work that required a supervised parser , in Tree Transformer , the constituency tree structures is automatically induced from raw texts by our proposed " Constituent Attention " module , which is simply implemented by self-attention .

Motivated by Tree-RNNs , which compose each phrase and the sentence representation from its constituent sub-phrases , Tree Transformer gradually attaches several smaller constituents into larger ones from lower layers to higher layers .

The contributions of this paper are 3-fold :

• Our proposed Tree Transformer is easy to implement , which simply inserts an additional " Constituent Attention " module implemented by self-attention to the original Transformer encoder , and achieves good performance on the unsupervised parsing task .

• As the induced tree structures guide words to compose the meaning of longer phrases hierarchically , Tree Transformer improves the perplexity on masked language modeling compared to the original Transformer .

• The behavior of attention heads learned by Tree Transformer expresses better interpretability , because they are constrained to follow the induced tree structures .

By visualizing the self-attention matrices , our model provides the information that better matchs the human intuition about hierarchical structures than the original Transformer .

This paper proposes Tree Transformer , a first attempt of integrating tree structures into Transformer by constraining the attention heads to attend within constituents .

The tree structures are automatically induced from the raw texts by our proposed Constituent Attention module , which attaches the constituents to each other by selfattention .

The performance on unsupervised parsing demonstrates the effectiveness of our model in terms of inducing tree structures coherent to human expert annotations .

We believe that incorporating tree structures into Transformer is an important and worth exploring direction , because it allows Transformer to learn more interpretable attention heads and achieve better language modeling .

The interpretable attention can better explain how the model processes the natural language and guide the future improvement .


This paper employs morphological structures and relations between sentence segments for opinion analysis on words and sentences .

Chinese words are classified into eight morphological types by two proposed classifiers , CRF classifier and SVM classifier .

Experiments show that the injection of morphological information improves the performance of the word polarity detection .

To utilize syntactic structures , we annotate structural trios to represent relations between sentence segments .

Experiments show that considering structural trios is useful for sentence opinion analysis .

The best f-score achieves 0.77 for opinion word extraction , 0.62 for opinion word polarity detection , 0.80 for opinion sentence extraction , and 0.54 for opinion sentence polarity detection .

Sentiment analysis has attracted much attention in recent years because a large scale of subjective information is disseminated through various platforms on the web .

Sentiment information can be applied to a wide variety of fields , including product recommendation , review summarization , public polling , and so on .

Opinion dictionaries are important resources for identifying subjective information .

Several approaches were proposed to collect such resources .

Wiebe ( 2000 ) learned subjective adjectives from corpora .

Takamura et al . ( 2005 ) extracted semantic orientations of words .

Ku et al . ( 2007 ) measured sentiment degrees of Chinese words by averaging the sentiment scores of the composing characters .

When the opinion words are available , the polarities of sentences and documents can be determined by them .

Riloff and Wiebe ( 2003 ) learned the extraction patterns for subjective expressions .

Kim and Hovy ( 2004 ) found the polarity of subjective expressions .

Pang et al . ( 2002 ) and Dave et al . ( 2003 ) explored various techniques at document level .

Morphological information has been widely used in classifying words , telling the meanings , and doing other in-depth analysis ( Tzeng and Chen , 2002 ) .

However , morphological information was seldom applied either in Chinese opinion extraction , or in solving the coverage problem of opinion dictionary .

Instead of bag-ofcharacters approach ( Ku et al . , 2007 ) , this paper employs morphological structures of words to extract opinion words .

Relations between sentence segments are also defined by linguistics in the Chinese language .

These are similar to morphological structures between Chinese characters .

Based on parsing trees of sentences , we identify these relations and utilize them for opinion analysis on sentences .

As the experimental corpus , some researchers managed to generate annotated materials and gold standards under many constraints .

Ku set a standard for generating final answers from annotations of multiple annotators ( Ku et al . , 2007 ) , and Somasundaran annotated discourse information from meeting dialogs to train a sentiment model ( Somasundaran et al . , 2007 ) .

For multilingual issues , researchers concerned mainly about the applicability of corpus and algorithms from the native language to foreign languages ( Banea et al . , 2008 ; Bautin et al . , 2008 ) .

Several opinion analysis systems have been developed so far .

OASYS ( Cesarano et al . , 2007 ) and CopeOpi ( Ku et al . , 2007 ) allow users input their queries and select preferred data sources , and then track opinions in a time zone .

For both systems , extracting opinions is the main focus , while holders and targets are identified implicitly when retrieving relevant documents .

Carenini 's team proposed a graphical user interface for evaluative texts ( 2006 ) , in which color blocks were used to present the evaluations for components of products .

Fair News Reader , a Japanese news Web system , incorporates sentiment information insensibly in an interesting way ( Kawai et al . , 2007 ) .

It provides readers " balanced " reports by analyzing the sentiment in news articles which readers have read , and suggests them new articles according to the analysis results .

It leads the application of opinion analysis to the direction of personalization .

This paper considers morphological and syntactic structures in analyzing Chinese opinion words and sentences .

For morphological structures , eight Chinese morphological types are defined .

CRF classifier and SVM classifier for morphological type classification are proposed .

Experiments show that CRF classifier achieves the best accuracy 0.70 in type classification , which is 8 % better than SVM classifier .

We further show that word morphological structures benefit the opinion word extraction significantly .

With the help of the sentiment dictionary NTUSD , the f-score of opinion word extraction achieves 0.77 and the f-score of the word polarity detection achieves 0.62 when the word morphological types are provided by the SVM classifier .

They are comparably better than bag-of-character approach and the dictionary based approach .

We defined structural trios to represent the relations between sentence segments and also extract these relations using CRF algorithm .

Results show that considering structural trios benefits the opinion analysis on sentences .

An fscore 0.80 for opinion extraction and an f-score 0.54 for polarity detection are achieved , which is a great improvement .

The opinion scoring functions for morphological types and structural trios are critical for polarity detection , and scoring functions for words determine the scoring functions for sentences .

Now we define these functions intuitively based on linguistic rules , but learning methods like regression will be investigated in the future .

Examining the interaction of cues from word and sentence levels on the opinion sentence extraction and the opinion polarity detection is our next goal .


In this paper , we investigate the importance of social network information compared to content information in the prediction of a Twitter user 's occupational class .

We show that the content information of a user 's tweets , the profile descriptions of a user 's follower / following community , and the user 's social network provide useful information for classifying a user 's occupational group .

In our study , we extend an existing dataset for this problem , and we achieve significantly better performance by using social network homophily that has not been fully exploited in previous work .

In our analysis , we found that by using the graph convolutional network to exploit social homophily , we can achieve competitive performance on this dataset with just a small fraction of the training data .

Equal Contribution ; work performed while both authors were visiting Singapore University of Technology and Design ( SUTD ) .

A Commercial and Domestic Grounds Maintenance company T1 : Pitch at wembley looks great .

T2 : Spurs will have best stadium in uk .

Twitter ( http://twitter.com ) is a microblogging service launched in 2006 , where , a user can publish messages with up to 280 characters , called " tweets " .

Unlike many other social networking platforms , such as Facebook and LinkedIn , Twitter does not provide structured fields for users to fill in personal information .

However , a user can write a 160-character-long small public summary about itself called a " Bio " .

Besides linguistic information from tweets and Bios , online social media is a rich source of network information .

People 's personal networks are homogeneous , i.e. , friends share more attributes such as race , ethnicity , religion , and occupation-known as the homophily principle ( McPherson et al . , 2001 ) .

Such network information has been utilized in friend recommendation ( Guy et al . , 2010 ) , community detection Follow Figure 1 : User and Network information on Twitter Microblog .

( Yang and Leskovec , 2013 ) , etc . Figure 1 shows two users connected on Twitter .

By looking at their Bio and tweets , it can be inferred that these users share the same occupational interest .

Profiling users can enhance service quality and improve product recommendation , and hence is a widely studied problem .

User occupational class prediction is an important component of user profiling and a sub-task of user demographic feature prediction .

Existing approaches to predicting Twitter users ' demographic attributes explore , select , and combine various features generated from text and network to achieve the best predictive performances in respective classification tasks ( Han et al . , 2013 ; Miller et al . , 2012 ; Preot ¸iuc-Pietro et al . , 2015 ; Huang et al . , 2015 ; Aletras and Chamberlain , 2018 ) .

The three categories of features are : account level features , tweet text features , and network based features .

Past research have shown the distinctive usage of language across gender , age , location , etc . in tweets ( Sloan et al . , 2015 ; Cheng et al . , 2010 ; Burger et al . , 2011 ; Rao et al . , 2010 ) , which makes content based prediction effective .

As for user occupational class prediction , Preot ¸iuc-Pietro et al . ( 2015 ) built a dataset where users are assigned to hierarchical job categories .

They used word cluster distribution features of content information to predict a user 's occupational group .

Aletras and Chamberlain ( 2018 ) constructed a user 's followings connections to learn the user embedding as a feature input to the classification models .

Considering the regional disparities of economic development stages , the major job categories may vary significantly across regions .

Sloan et al . ( 2015 ) summarized occupation distribution of Twitter users in the UK by looking into their profiles .

In this paper , we analyze the usefulness of a user 's network information over the user 's tweets for predicting its occupational group .

We extend the existing dataset for occupation classification ( Preot ¸iuc-Pietro et al . ( 2015 ) ) by introducing the network information about a user , i.e. follower / following IDs together with their Bio descriptions , and we construct a user-centric network to extract useful community and text based features .

The acquired features from the network are then exploited using a graph neural network .

The obtained results show the importance of a network information over tweet information from a user for such a task .

Previous works have used tweets or a fraction of the network information to extract features for occupation classification .

To analyze the importance of network information , we extended an existing Twitter dataset for a user 's social media connections ( follow information ) .

We showed that by using only follow information as an input to graph convolutional networks , one can achieve a significantly higher accuracy on the prediction task as compared to the existing approaches utilizing tweet-only information or partial network structure .

Directions of future research include adaptation of our methods to a large scale , sparsely connected social network .

One might also want to investigate the inductive settings of GCN ( Hamilton et al . , 2017 ) to predict demographic information of a user from outside the black network .


Short vowels and other diacritics are not part of written Arabic scripts .

Exceptions are made for important political and religious texts and in scripts for beginning students of Arabic .

Script without diacritics have considerable ambiguity because many words with different diacritic patterns appear identical in a diacritic-less setting .

We propose in this paper a maximum entropy approach for restoring diacritics in a document .

The approach can easily integrate and make effective use of diverse types of information ; the model we propose integrates a wide array of lexical , segmentbased and part-of-speech tag features .

The combination of these feature types leads to a state-of-the-art diacritization model .

Using a publicly available corpus ( LDC 's Arabic Treebank Part 3 ) , we achieve a diacritic error rate of 5.1 % , a segment error rate 8.5 % , and a word error rate of 17.3 % .

In case-ending-less setting , we obtain a diacritic error rate of 2.2 % , a segment error rate 4.0 % , and a word error rate of 7.2 % .

Modern Arabic written texts are composed of scripts without short vowels and other diacritic marks .

This often leads to considerable ambiguity since several words that have different diacritic patterns may appear identical in a diacritic-less setting .

Educated modern Arabic speakers are able to accurately restore diacritics in a document .

This is based on the context and their knowledge of the grammar and the lexicon of Arabic .

However , a text without diacritics becomes a source of confusion for beginning readers and people with learning disabilities .

A text without diacritics is also problematic for applications such as text-to-speech or speech-to-text , where the lack of diacritics adds another layer of ambiguity when processing the data .

As an example , full vocalization of text is required for text-to-speech applications , where the mapping from graphemes to phonemes is simple compared to languages such as English and French ; where there is , in most cases , one-to-one relationship .

Also , using data with diacritics shows an improvement in the accuracy of speech-recognition applications ( Afify et al . , 2004 ) .

Currently , text-tospeech , speech-to-text , and other applications use data where diacritics are placed manually , which is a tedious and time consuming excercise .

A diacritization system that restores the diacritics of scripts , i.e. supply the full diacritical markings , would be of interest to these applications .

It also would greatly benefit nonnative speakers , sufferers of dyslexia and could assist in restoring diacritics of children 's and poetry books , a task that is currently done manually .

We propose in this paper a statistical approach that restores diacritics in a text document .

The proposed approach is based on the maximum entropy framework where several diverse sources of information are employed .

The model implicitly learns the correlation between these types of information and the output diacritics .

In the next section , we present the set of diacritics to be restored and the ambiguity we face when processing a non-diacritized text .

Section 3 gives a brief summary of previous related works .

Section 4 presents our diacritization model ; we explain the training and decoding process as well as the different feature categories employed to restore the diacritics .

Section 5 describes a clearly defined and replicable split of the LDC 's Arabic Treebank Part 3 corpus , used to built and evaluate the system , so that the reproduction of the results and future comparison can accurately be established .

Section 6 presents the experimental results .

Section 7 reports a comparison of our approach to the finite state machine modeling technique that showed promissing results in ( Nelken and Shieber , 2005 ) .

Finally , section 8 concludes the paper and discusses future directions .

We presented in this paper a statistical model for Arabic diacritic restoration .

The approach we propose is based on the Maximum entropy framework , which gives the system the ability to integrate different sources of knowledge .

Our model has the advantage of successfully combining diverse sources of information ranging from lexical , segment-based and POS features .

Both POS and segment-based features are generated by separate statistical systems -not extracted manually -in order to simulate real world applications .

The segment-based features are extracted from a statistical morphological analysis system using WFST approach and the POS features are generated by a parsing model Table 5 : Error Rate in % for n-gram diacritic restoration and segmentation using FST and Kneser-Ney LM .

Columns marked with " True shadda " represent results on documents containing the original consonant doubling " shadda " while columns marked with " Predicted shadda " represent results where the system restored all diacritics including shadda .

that also uses Maximum entropy framework .

Evaluation results show that combining these sources of information lead to state-of-the-art performance .

As future work , we plan to incorporate Buckwalter morphological analyzer information to extract new features that reduce the search space .

One idea will be to reduce the search to the number of hypotheses , if any , proposed by the morphological analyzer .

We also plan to investigate additional conjunction features to improve the accuracy of the model .


Current state-of-the-art systems for joint entity relation extraction ( Luan et al . , 2019 ; Wadden et al . , 2019 ) usually adopt the multi-task learning framework .

However , annotations for these additional tasks such as coreference resolution and event extraction are always equally hard ( or even harder ) to obtain .

In this work , we propose a pre-training method ENPAR to improve the joint extraction performance .

EN-PAR requires only the additional entity annotations that are much easier to collect .

Unlike most existing works that only consider incorporating entity information into the sentence encoder , we further utilize the entity pair information .

Specifically , we devise four novel objectives , i.e. , masked entity typing , masked entity prediction , adversarial context discrimination , and permutation prediction , to pretrain an entity encoder and an entity pair encoder .

Comprehensive experiments show that the proposed pre-training method achieves significant improvement over BERT on ACE05 , SciERC , and NYT , and outperforms current state-of-the-art on ACE05 .

Joint extraction of entities and relations is a fundamental task in information extraction , it aims to extract entities and relations with a unified model .

Current approaches ( Luan et al . , 2019 ; Wadden et al . , 2019 ) usually adopt the multi-task learning framework that optimizes many objectives simultaneously , including entity recognition , relation extraction , coreference resolution , and event extraction .

However , as large-scale manually labeled data required by these methods is unavailable in many domains , their applicability is severely restricted .

Therefore , we expect to catch or even surpass the multi-task based joint models with less annotation cost .

Compared with the annotations of coreference resolution and event extraction , entity annotations can be easily obtained through automatic NER annotation tools ( e.g. , spaCy1 ) .

In this paper , we focus on improving the model 's performance with just extra entity annotations .

Although pre-trained models , like BERT , have shown impressive performance in many downstream tasks , they have mainly two limitations when applied in the joint entity relation extraction task .

One is that currently pre-trained objectives are insufficient for this task .

Specifically , these commonly used universal pre-trained model ( e.g. , BERT ) do not consider the entity-related knowledge that is crucial for better extracting entities and relations .

The other is that these models only provide pre-trained representations for tokens and sentences , but not entities and entity pairs .

To obtain the representations for entities and entity pairs , additional parameters that are not pre-trained are introduced in the fine-tuning stage , which may futher impair the joint extraction performance .

To address the first limitation , recent several works try to incorporate entity-related information into pre-training objectives .

Zhang et al . ( 2019 ) fuses heterogeneous information from both texts and knowledge graphs and proposes a denoising entity auto-encoder objective based on BERT .

Sun et al . ( 2019c ) presents two knowledge masking strategies in the pre-training stage ( entity-level masking and phrase-level masking ) .

Both of them utilize extra entity annotations ( i.e. , entities in knowledge graphs and automatic entity annotations , respectively ) .

In this paper , we follow this line of works and build a large-scale entity annotated corpus using the spaCy NER tool .

For the second limitation , we propose ENPAR , a pre-training method customized for entity relation extraction .

ENPAR consists of an underlying sentence encoder , an entity encoder , and an entity pair encoder .

Compared with BERT ( Figure 1 ( a ) ) , the proposed entity encoder and entity pair encoder directly provide representations of entities and entity pairs .

To train the three encoders , we devise four novel pre-training tasks : masked entity typing , masked entity prediction , adversarial context discrimination and permutation prediction ( Figure 1(b ) ) .

In the first two tasks , we randomly mask some entity words and then predict the masked tokens and the entity type .

These two tasks are natural extensions of the masked language model .

To learn a better entity pair encoder , we draw inspirations from the denoising auto-encoder ( Zhang et al . , 2019 ) and propose the last two tasks .

Specifically , when the entity pair or its context in a sentence are perturbed , we hope that the entity pair encoder is capable of tracking such changes .

We employ the parameter-sharing method for these four tasks and train these objectives jointly .

To sum up , our main contributions are as follows2 :

• We introduce an entity encoder and an entity pair encoder to incorporate not only the entity information but also the entity pair information , which were ignored in current universal pre-trained models .

• We propose four novel pre-training tasks that help to learn the proposed encoders .

These tasks only require additional entity annotations ( with commonly used entity types ) , which can be automatically generated by public annotation tools , such as spaCy NER .

• We conduct comprehensive experiments and demonstrate that the proposed method achieves significant improvement on ACE05 and NYT dataset and is comparable with the state-of-the-art on the SciERC dataset .

We propose ENPAR , a pre-training method customized for entity relation extraction only with additional entity annotations .

Instead of only pretraining sentence encoder in universal pre-trained models , we also pre-train an entity encoder and an entity pair encoder .

Then the proposed four objec-tives can incorporate entity and entity-pair knowledge into the pre-trained encoders to enhance the encoders ' representations .

Experiments on three datasets demonstrate that ENPAR achieves comparable or even superior performances compared with multi-task based joint models .


We present the Siamese Continuous Bag of Words ( Siamese CBOW ) model , a neural network for efficient estimation of highquality sentence embeddings .

Averaging the embeddings of words in a sentence has proven to be a surprisingly successful and efficient way of obtaining sentence embeddings .

However , word embeddings trained with the methods currently available are not optimized for the task of sentence representation , and , thus , likely to be suboptimal .

Siamese CBOW handles this problem by training word embeddings directly for the purpose of being averaged .

The underlying neural network learns word embeddings by predicting , from a sentence representation , its surrounding sentences .

We show the robustness of the Siamese CBOW model by evaluating it on 20 datasets stemming from a wide variety of sources .

Word embeddings have proven to be beneficial in a variety of tasks in NLP such as machine translation ( Zou et al . , 2013 ) , parsing ( Chen and Manning , 2014 ) , semantic search ( Reinanda et al . , 2015 ; Voskarides et al . , 2015 ) , and tracking the meaning of words and concepts over time ( Kim et al . , 2014 ; Kenter et al . , 2015 ) .

It is not evident , however , how word embeddings should be combined to represent larger pieces of text , like sentences , paragraphs or documents .

Surprisingly , simply averaging word embeddings of all words in a text has proven to be a strong baseline or feature across a multitude of tasks ( Faruqui et al . , 2014 ; Yu et al . , 2014 ; Gershman and Tenenbaum , 2015 ; Kenter and de Rijke , 2015 ) .

Word embeddings , however , are not optimized specifically for representing sentences .

In this paper we present a model for obtaining word embeddings that are tailored specifically for the task of averaging them .

We do this by directly including a comparison of sentence embeddings-the averaged embeddings of the words they contain-in the cost function of our network .

Word embeddings are typically trained in a fast and scalable way from unlabeled training data .

As the training data is unlabeled , word embeddings are usually not task-specific .

Rather , word embeddings trained on a large training corpus , like the ones from ( Collobert and Weston , 2008 ; Mikolov et al . , 2013b ) are employed across different tasks ( Socher et al . , 2012 ; Kenter and de Rijke , 2015 ; Hu et al . , 2014 ) .

These two qualities-(i ) being trainable from large quantities of unlabeled data in a reasonable amount of time , and ( ii ) robust performance across different tasks-are highly desirable and allow word embeddings to be used in many large-scale applications .

In this work we aim to optimize word embeddings for sentence representations in the same manner .

We want to produce general purpose sentence embeddings that should score robustly across multiple test sets , and we want to leverage large amounts of unlabeled training material .

In the word2vec algorithm , Mikolov et al . ( 2013a ) construe a supervised training criterion for obtaining word embeddings from unsupervised data , by predicting , for every word , its surrounding words .

We apply this strategy at the sentence level , where we aim to predict a sentence from its adjacent sentences ( Kiros et al . , 2015 ; Hill et al . , 2016 ) .

This allows us to use unlabeled training data , which is easy to obtain ; the only restriction is that documents need to be split into sentences and that the order between sentences is preserved .

The main research question we address is whether directly optimizing word embeddings for the task of being averaged to produce sentence embeddings leads to word embeddings that are better suited for this task than word2vec does .

Therefore , we test the embeddings in an unsupervised learning scenario .

We use 20 evaluation sets that stem from a wide variety of sources ( newswire , video descriptions , dictionary descriptions , microblog posts ) .

Furthermore , we analyze the time complexity of our method and compare it to our baselines methods .

Summarizing , our main contributions are :

We have presented Siamese CBOW , a neural network architecture that efficiently learns word embeddings optimized for producing sentence representations .

The model is trained using only unla-beled text data .

It predicts , from an input sentence representation , the preceding and following sentence .

We evaluated the model on 20 test sets and show that in a majority of cases , 14 out of 20 , Siamese CBOW outperforms a word2vec baseline and a baseline based on the recently proposed skip-thought architecture .

As further analysis on various choices of parameters show that the method is stable across settings , we conclude that Siamese CBOW provides a robust way of generating high-quality sentence representations .

Word and sentence embeddings are ubiquitous and many different ways of using them in supervised tasks have been proposed .

It is beyond the scope of this paper to provide a comprehensive analysis of all supervised methods using word or sentence embeddings and the effect Siamese CBOW would have on them .

However , it would be interesting to see how Siamese CBOW embeddings would affect results in supervised tasks .

Lastly , although we evaluated Siamese CBOW on sentence pairs , there is no theoretical limitation restricting it to sentences .

It would be interesting to see how embeddings for larger pieces of texts , such as documents , would perform in document clustering or filtering tasks .


We propose a new framework for abstractive text summarization based on a sequence-to-sequence oriented encoderdecoder model equipped with a deep recurrent generative decoder ( DRGN ) .

Latent structure information implied in the target summaries is learned based on a recurrent latent random model for improving the summarization quality .

Neural variational inference is employed to address the intractable posterior inference for the recurrent latent variables .

Abstractive summaries are generated based on both the generative latent variables and the discriminative deterministic states .

Extensive experiments on some benchmark datasets in different languages show that DRGN achieves improvements over the state-ofthe-art methods .

Automatic summarization is the process of automatically generating a summary that retains the most important content of the original text document ( Edmundson , 1969 ; Luhn , 1958 ; Nenkova and McKeown , 2012 ) .

Different from the common extraction-based and compression-based methods , abstraction-based methods aim at constructing new sentences as summaries , thus they require a deeper understanding of the text and the capability of generating new sentences , which provide an obvious advantage in improving the focus of a summary , reducing the redundancy , and keeping a good compression rate ( Bing et al . , 2015 ; Rush et al . , 2015 ; Nallapati et al . , 2016 ) .

The work described in this paper is supported by a grant from the Grant Council of the Hong Kong Special Administrative Region , China ( Project Code : 14203414 ) .

Some previous research works show that human-written summaries are more abstractive ( Jing and McKeown , 2000 ) .

Moreover , our investigation reveals that people may naturally follow some inherent structures when they write the abstractive summaries .

To illustrate this observation , we show some examples in Figure 1 , which are some top story summaries or headlines from the channel " Technology " of CNN .

After analyzing the summaries carefully , we can find some common structures from them , such as " What " , " What-Happened " , " Who Action What " , etc .

For example , the summary " Apple sues Qualcomm for nearly $ 1 billion " can be structuralized as " Who ( Apple ) Action ( sues )

Similarly , the summaries " [ Twitter ] [ fixes ]

[ botched @POTUS account transfer ] " , " [ Uber ] [ to pay ]

[ $ 20 million ] for misleading drivers " , and " [ Bipartisan bill ] aims to [ reform ]

[ H-1B visa system ] " also follow the structure of " Who Action What " .

The summary " The emergence of the ' cyber cold war " ' matches with the structure of " What " , and the summary " St. Louis ' public library computers hacked " follows the structure of " What-Happened " .

Intuitively , if we can incorporate the latent structure information of summaries into the abstractive summarization model , it will improve the quality of the generated summaries .

However , very few existing works specifically consider the latent structure information of summaries in their summarization models .

Although a very popular neural network based sequence-to-sequence ( seq2seq ) framework has been proposed to tackle the abstractive summarization problem ( Lopyrev , 2015 ; Rush et al . , 2015 ; Nallapati et al . , 2016 ) , the calculation of the internal decoding states is entirely deterministic .

The deterministic transformations in these discriminative models lead to limitations on the representation ability of the latent structure information .

Miao and Blunsom ( 2016 ) extended the seq2seq framework and proposed a generative model to capture the latent summary information , but they did not consider the recurrent dependencies in their generative model leading to limited representation ability .

To tackle the above mentioned problems , we design a new framework based on sequenceto-sequence oriented encoder-decoder model equipped with a latent structure modeling component .

We employ Variational Auto-Encoders ( VAEs ) ( Kingma and Welling , 2013 ; Rezende et al . , 2014 ) as the base model for our generative framework which can handle the inference problem associated with complex generative modeling .

However , the standard framework of VAEs is not designed for sequence modeling related tasks .

Inspired by ( Chung et al . , 2015 ) , we add historical dependencies on the latent variables of VAEs and propose a deep recurrent generative decoder ( DRGD ) for latent structure modeling .

Then the standard discriminative deterministic decoder and the recurrent generative decoder are integrated into a unified decoding framework .

The target summaries will be decoded based on both the discriminative deterministic variables and the generative latent structural information .

All the neural parameters are learned by back-propagation in an end-to-end training paradigm .

The main contributions of our framework are summarized as follows : ( 1 ) We propose a sequence-to-sequence oriented encoder-decoder model equipped with a deep recurrent generative decoder ( DRGD ) to model and learn the latent structure information implied in the target summaries of the training data .

( 2 ) Both the generative latent structural information and the discriminative deterministic variables are jointly considered in the generation process of the abstractive summaries .

( 3 ) Experimental results on some benchmark datasets in different languages show that our framework achieves better performance than the state-of-the-art models .

We propose a deep recurrent generative decoder ( DRGD ) to improve the abstractive summarization performance .

The model is a sequenceto-sequence oriented encoder-decoder framework equipped with a latent structure modeling component .

Abstractive summaries are generated based on both the latent variables and the deterministic states .


Multi-layer multi-head self-attention mechanism is widely applied in modern neural language models .

Attention redundancy has been observed among attention heads but has not been deeply studied in the literature .

Using BERT-base model as an example , this paper provides a comprehensive study on attention redundancy which is helpful for model interpretation and model compression .

We analyze the attention redundancy with Five-Ws and How .

( What ) We define and focus the study on redundancy matrices generated from pre-trained and fine-tuned BERT-base model for GLUE datasets .

( How ) We use both token-based and sentence-based distance functions to measure the redundancy .

( Where ) Clear and similar redundancy patterns ( cluster structure ) are observed among attention heads .

( When ) Redundancy patterns are similar in both pre-training and fine-tuning phases .

( Who ) We discover that redundancy patterns are task-agnostic .

Similar redundancy patterns even exist for randomly generated token sequences .

( " Why " ) We also evaluate influences of the pre-training dropout ratios on attention redundancy .

Based on the phaseindependent and task-agnostic attention redundancy patterns , we propose a simple zero-shot pruning method as a case study .

Experiments on fine-tuning GLUE tasks verify its effectiveness .

The comprehensive analyses on attention redundancy make model understanding and zero-shot model pruning promising .

Multi-layer multi-head self-attention architectures ( Transformer ( Vaswani et al . , 2017 ) ) are widely applied in modern language models , such as BERT ( Devlin et al . , 2019 ) , RoBERTa ( Liu et al . , 2019 ) , OpenAI GPT ( Radford et al . , 2018 ) , GPT-2 ( Radford et al . , 2019 ) and ERNIE2.0 ( Sun et al . , 2019 ) , to name a few .

Redundancy phenomenon is discovered among attention heads .

It demonstrates that many attention heads generate very similar attention matrices ( Clark et al . , 2019 ; Kovaleva et al . , 2019 ) .

We take the pre-trained BERT-base model as an example .

It learns 12-layer-12-head self-attention matrices describing dependencies between each pair of tokens in a sentence .

Then for each token , there are 144 attention vectors .

We use Jensen-Shannon distance to measure the relationship between each pair of vectors .

Then for one sentence ( consisting of a sequence of tokens ) , the token-averaged distance is utilized to imply the redundancy between each pair of attention matrices .

Smaller distance values reflect more redundancy .

Figure 1 shows the redundancy ( distance ) among 144 × 144 pairs of attention matrices averaged over 1000 randomly sampled sentences .

We can see clear redundancy patterns ( clusters with smaller distance areas ) in consecutive attention layers .

Analyzing the attention redundancy helps to interpret the multi-layer multi-head self-attention architecture .

Various studies have attempted to re-veal the relationship among attention heads .

Examples are attention visualization ( Vig and Belinkov , 2019 ) , common attention patterns ( Kovaleva et al . , 2019 ) , attention head pruning ( Voita et al . , 2019 ) , and probing test ( Clark et al . , 2019 ) .

Existing works either focus on the 12 × 12 attention matrices and their effects on ( pre-training or / and finetuning ) performances or focus on linguistic features extracted by latent token vectors and attention matrices .

Though the redundancy phenomenon was discovered , no existing work studies the attention redundancy pattern itself ( i.e. , the 144 × 144 distance matrix in Figure 1 ) deeply .

This motivates us to conduct a comprehensive and complementary study on the attention redundancy phenomenon .

In this paper , we take the BERT-base model as a representative model to analyze the attention redundancy with Five Ws and How .

As far as we know , many of the following discoveries are new to the research community .

Given a distance function , we define the pairwise distance matrix ( ∈ R 144×144 ) of the 12 × 12 attention matrices of BERT-base model as attention redundancy matrix .

In this paper , we obtain redundancy matrices from both pre-trained and finetuned BERT-base model for GLUE tasks as the research objects .

Except for the two token-based measures , Jensen-Shannon distance ( Clark et al . , 2019 ) and cosine similarity ( Kovaleva et al . , 2019 ) used in literature , we employ two more token-based distance function and three sentence-based ones to measure attention redundancy and analyze their similar redundancy patterns ( please refer to Section 4.1 for more details ) .

The purpose is to alleviate the measuring bias of just using one distance function .

Sentence-based distances directly measure the relationship between two attention matrices without averaging over tokens .

We visualize the redundancy patterns using various distance functions .

We find common hierarchical cluster structures in the set of token-based redundancy matrices and the set of sentence-based redundancy matrices , respectively .

Attention heads of earlier , middle , and deeper attention layers are clearly clustered in the redundancy matrices .

We also demonstrate that highly correlated similar redundancy patterns exist in redundancy matrices generated based on different type of distances .

Common redundancy patterns are discovered in both the pre-trained phase and fine-tuned phases .

For any downstream task with any distance function , we notice highly correlated attention redundancy patterns between two phases .

Who ( which task ) has attention redundancy ?

We surprisingly realize that the redundancy is task-agnostic .

The redundancy patterns are highly correlated across different tasks .

We even randomly generate token sequences as input in the pre-trained BERT-base model .

Very similar attention redundancy patterns occur as well .

Based on this astonishing discovery , as a case study application , we propose a simple zero-shot head-pruning strategy based on clustering results using redundancy matrices .

Compared to other complex pruning strategies , e.g. , ( Tang et al . , 2019 ; Jiao et al . , 2019 ; Fan et al . , 2019 ; Wang et al . , 2019 ; McCarley , 2019 ) , the most important is that without knowing any data of fine-tuning tasks , this pruning can be effectively and efficiently conducted just based on some randomly generated token sequences with the pre-trained BERT-base model .

The only effort is to compute one or several redundancy matrices .

Results reflect that for most GLUE tasks , the proposed pruning strategy based on redundancy matrices can prune up to 75 % to 85 % of attention heads while keeping comparable fine-tuning performances .

" Why " does the phase-independent and taskagnostic attention redundancy happen ?

It 's hard to tell the reason of the redundancy patterns ( that 's why we use the quoted " Why " ) .

However , we conduct experiments to evaluate the effects on attention redundancy of dropout ratios in the pre-training phase which are suspected as one of reasons ( Clark et al . , 2019 ) .

When we use sentence-based distance , a monotonic trend is found .

Attention heads tend to be more redundant when increasing dropout ratios .

When we use token-based distances , a complex " N"-shape effect exists .

We also notice that the redundancy is more sensitive to dropouts in hidden linear transformations than to dropouts in the self-attention mechanism .

We believe that above-mentioned new findings in this paper make the redundancy analyses a promis-ing research direction in model interpretation and model compression , and so on .

Using BERT-base model as an example , we comprehensively investigated the attention redundancy in multi-layer multi-head self-attention based language models .

The redundancy was measured by distance functions at token level and sentence level .

At both levels , we found that many heads are not distinct from each other , and clear clustering effects were observed .

We discovered that the attention redundancy is phase-independent and taskagnostic .

Specifically , compared to a pre-trained model , the redundancy patterns do not change much after fine-tuning on multiple downstream tasks .

We also shown complex influences on redundancy of dropout ratios in hidden transformations and self-attention .

Based on these discoveries , we design a zero-shot strategy to prune attention heads .

Compared to existing methods , the zero-shot pruning is simple and robust ( task-agnostic ) .

In the near future , we are interested in experimenting this method over more self-attention based pre-trained language models and more downstream tasks .

This is the appendix for NAACL-HLT 2021 paper : Yuchen Bian , Jiaji Huang , Xingyu Cai , Jiahong Yuan , and Kenneth Church .

On Attention Redundancy : A Comprehensive Study .

Redundancy phenomenon is discovered among attention heads .

The redundancy was measured by distance functions at token level and sentence level .

Experiments on fine-tuning GLUE tasks verify its effectiveness .

We discovered that the attention redundancy is phase-independent and taskagnostic .


Comprehending lyrics , as found in songs and poems , can pose a challenge to human and machine readers alike .

This motivates the need for systems that can understand the ambiguity and jargon found in such creative texts , and provide commentary to aid readers in reaching the correct interpretation .

We introduce the task of automated lyric annotation ( ALA ) .

Like text simplification , a goal of ALA is to rephrase the original text in a more easily understandable manner .

However , in ALA the system must often include additional information to clarify niche terminology and abstract concepts .

To stimulate research on this task , we release a large collection of crowdsourced annotations for song lyrics .

We analyze the performance of translation and retrieval models on this task , measuring performance with both automated and human evaluation .

We find that each model captures a unique type of information important to the task .

Song lyrics and poetry often make use of ambiguity , symbolism , irony , and other stylistic elements to evoke emotive responses .

These characteristics sometimes make it challenging to interpret obscure lyrics , especially for readers or listeners who are unfamiliar with the genre .

To address this problem , several online lyric databases have been created where users can explain , contextualize , or discuss lyrics .

Examples include MetroLyrics1 and Genius.com2 .

To be without a home Like a complete unknown ,

We presented and released the Genius dataset to study the task of Automated Lyric Annotation .

As a first investigation , we studied automatic generation of context independent annotations as machine translation and information retrieval .

Our baseline system tests indicate that our corpus is suitable to train machine translation systems .

Standard SMT models are capable of rephrasing and simplifying song lyrics but tend to keep close to the structure of the song lyric .

Seq2Seq models demonstrated potential to generate more fluent and informative text , dissimilar to the lyric .

A large fraction of the annotations is heavily based on context and background knowledge ( CS ) , one of their most appealing aspects .

As future work we suggest injection of structured and unstructured external knowledge ( Ahn et al . , 2016 ) and explicit modeling of references ( Yang et al . , 2016 ) .


Most statistical translation systems are based on phrase translation pairs , or " blocks " , which are obtained mainly from word alignment .

We use blocks to infer better word alignment and improved word alignment which , in turn , leads to better inference of blocks .

We propose two new probabilistic models based on the innerouter segmentations and use EM algorithms for estimating the models ' parameters .

The first model recovers IBM Model-1 as a special case .

Both models outperform bidirectional IBM Model-4 in terms of word alignment accuracy by 10 % absolute on the F-measure .

Using blocks obtained from the models in actual translation systems yields statistically significant improvements in Chinese-English SMT evaluation .

Today 's statistical machine translation systems rely on high quality phrase translation pairs to acquire state-of-the-art performance , see ( Koehn et al . , 2003 ; Zens and Ney , 2004 ; Och and Ney , 2003 ) .

Here , phrase pairs , or " blocks " are obtained automatically from parallel sentence pairs via the underlying word alignments .

Word alignments traditionally are based on IBM Models 1 - 5 ( Brown et al . , 1993 ) or on HMMs ( Vogel et al . , 1996 ) .

Automatic word alignment is challenging in that its accuracy is not yet close to inter-annotator agreement in some language pairs : for Chinese-English , inter-annotator agreement exceeds 90 on F-measure whereas IBM Model-4 or HMM accuracy is typically below 80s .

HMMs assume that words " close-in-source " are aligned to words " close-in-target " .

While this locality assumption is generally sound , HMMs do have limitations : the self-transition probability of a state ( word ) is the only control on the duration in the state , the length of the phrase aligned to the word .

Also there is no natural way to control repeated non-contiguous visits to a state .

Despite these problems , HMMs remain attractive for their speed and reasonable accuracy .

We propose a new method for localizing word alignments .

We use blocks to achieve locality in the following manner : a block in a sentence pair is a source phrase aligned to a target phrase .

We assume that words inside the source phrase can not align to words outside the target phrase and that words outside the source phrase can not align to words inside the target phrase .

Furthermore , a block divides the sentence pair into two smaller regions : the inner part of the block , which corresponds to the source and target phrase in the block , and the outer part of the block , which corresponds to the remaining source and target words in the parallel sentence pair .

The two regions are non-overlapping ; and each of them is shorter than the original parallel sentence pair .

The regions are thus easier to align than the original sentence pairs ( e.g. , using IBM Model-1 ) .

While the model uses a single block to split the sentence pair into two independent regions , it is not clear which block we should select for this purpose .

Therefore , we treat the splitting block as a hidden variable .

This proposed approach is far simpler than treating the entire sentence as a sequence of nonoverlapping phrases ( or chunks ) and considering such sequential segmentation either explicitly or implicitly .

For example , ( Marcu and Wong , 2002 ) for a joint phrase based model , ( Huang et al . , 2003 ) for a translation memory system ; and ( Watanabe et al . , 2003 ) for a complex model of insertion , deletion and head-word driven chunk reordering .

Other approaches including ( Watanabe et al . , 2002 ) treat extracted phrase-pairs as new parallel data with limited success .

Typically , they share a similar architecture of phrase level segmentation , reordering , translation as in ( Och and Ney , 2002 ; Koehn and Knight , 2002 ; Yamada and Knight , 2001 ) .

The phrase level interaction has to be taken care of for the non-overlapping sequential segmentation in a complicated way .

Our models model such interactions in a soft way .

The hidden blocks are allowed to overlap with each other , while each block induced two non-overlapping regions , i.e. the model brackets the sentence pair into two independent parts which are generated synchronously .

In this respect , it resembles bilingual bracketing ( Wu , 1997 ) , but our model has more lexical items in the blocks with many-to-many word alignment freedom in both inner and outer parts .

We present our localization constraints using blocks for word alignment in Section 2 ; we detail our two new probabilistic models and their EM training algorithms in Section 3 ; our baseline system , a maximum-posterior inference for word alignment , is explained in Section 4 ; experimental results of alignments and translations are in Section 5 ; and Section 6 contains discussion and conclusions .

Our main contributions are two novel Inner-Outer Bracket models based on segmentations induced by hidden blocks .

Modeling the Inner-Outer hidden segmentations , we get significantly improved word alignments for both the small training set and the large training set over the widely-practiced bidirectional IBM Model-4 alignment .

We also show significant improvements in translation quality using our proposed bracket models .

Robustness to noisy blocks merits further investigation .


We propose an approach for extracting relations between entities from biomedical literature based solely on shallow linguistic information .

We use a combination of kernel functions to integrate two different information sources : ( i ) the whole sentence where the relation appears , and ( ii ) the local contexts around the interacting entities .

We performed experiments on extracting gene and protein interactions from two different data sets .

The results show that our approach outperforms most of the previous methods based on syntactic and semantic information .

Information Extraction ( IE ) is the process of finding relevant entities and their relationships within textual documents .

Applications of IE range from Semantic Web to Bioinformatics .

For example , there is an increasing interest in automatically extracting relevant information from biomedical literature .

Recent evaluation campaigns on bio-entity recognition , such as BioCreAtIvE and JNLPBA 2004 shared task , have shown that several systems are able to achieve good performance ( even if it is a bit worse than that reported on news articles ) .

However , relation identification is more useful from an applicative perspective but it is still a considerable challenge for automatic tools .

In this work , we propose a supervised machine learning approach to relation extraction which is applicable even when ( deep ) linguistic processing is not available or reliable .

In particular , we explore a kernel-based approach based solely on shallow linguistic processing , such as tokeniza-tion , sentence splitting , Part-of-Speech ( PoS ) tagging and lemmatization .

Kernel methods ( Shawe-Taylor and Cristianini , 2004 ) show their full potential when an explicit computation of the feature map becomes computationally infeasible , due to the high or even infinite dimension of the feature space .

For this reason , kernels have been recently used to develop innovative approaches to relation extraction based on syntactic information , in which the examples preserve their original representations ( i.e. parse trees ) and are compared by the kernel function ( Zelenko et al . , 2003 ; Culotta and Sorensen , 2004 ; Zhao and Grishman , 2005 ) .

Despite the positive results obtained exploiting syntactic information , we claim that there is still room for improvement relying exclusively on shallow linguistic information for two main reasons .

First of all , previous comparative evaluations put more stress on the deep linguistic approaches and did not put as much effort on developing effective methods based on shallow linguistic information .

A second reason concerns the fact that syntactic parsing is not always robust enough to deal with real-world sentences .

This may prevent approaches based on syntactic features from producing any result .

Another related issue concerns the fact that parsers are available only for few languages and may not produce reliable results when used on domain specific texts ( as is the case of the biomedical literature ) .

For example , most of the participants at the Learning Language in Logic ( LLL ) challenge on Genic Interaction Extraction ( see Section 4.2 ) were unable to successfully exploit linguistic information provided by parsers .

It is still an open issue whether the use of domainspecific treebanks ( such as the Genia treebank 1 ) can be successfully exploited to overcome this problem .

Therefore it is essential to better investigate the potential of approaches based exclusively on simple linguistic features .

In our approach we use a combination of kernel functions to represent two distinct information sources : the global context where entities appear and their local contexts .

The whole sentence where the entities appear ( global context ) is used to discover the presence of a relation between two entities , similarly to what was done by Bunescu and Mooney ( 2005b ) .

Windows of limited size around the entities ( local contexts ) provide useful clues to identify the roles of the entities within a relation .

The approach has some resemblance with what was proposed by Roth and Yih ( 2002 ) .

The main difference is that we perform the extraction task in a single step via a combined kernel , while they used two separate classifiers to identify entities and relations and their output is later combined with a probabilistic global inference .

We evaluated our relation extraction algorithm on two biomedical data sets ( i.e. the AImed corpus and the LLL challenge data set ; see Section 4 ) .

The motivations for using these benchmarks derive from the increasing applicative interest in tools able to extract relations between relevant entities in biomedical texts and , consequently , from the growing availability of annotated data sets .

The experiments show clearly that our approach consistently improves previous results .

Surprisingly , it outperforms most of the systems based on syntactic or semantic information , even when this information is manually annotated ( i.e. the LLL challenge ) .

The good results obtained using only shallow linguistic features provide a higher baseline against which it is possible to measure improvements obtained using methods based on deep linguistic processing .

In the near future , we plan to extend our work in several ways .

First , we would like to evaluate the contribution of syntactic information to relation extraction from biomedical literature .

With this aim , we will integrate the output of a parser ( possibly trained on a domain-specific resource such the Genia Treebank ) .

Second , we plan to test the portability of our model on ACE and MUC data sets .

Third , we would like to use a named entity recognizer instead of assuming that entities are already extracted or given by a dictionary .

Our long term goal is to populate databases and ontologies by extracting information from large text collections such as Medline .


Domain adaptation is an important problem in named entity recognition ( NER ) .

NER classifiers usually lose accuracy in the domain transfer due to the different data distribution between the source and the target domains .

The major reason for performance degrading is that each entity type often has lots of domainspecific term representations in the different domains .

The existing approaches usually need an amount of labeled target domain data for tuning the original model .

However , it is a labor-intensive and time-consuming task to build annotated training data set for every target domain .

We present a domain adaptation method with latent semantic association ( LaSA ) .

This method effectively overcomes the data distribution difference without leveraging any labeled target domain data .

LaSA model is constructed to capture latent semantic association among words from the unlabeled corpus .

It groups words into a set of concepts according to the related context snippets .

In the domain transfer , the original term spaces of both domains are projected to a concept space using LaSA model at first , then the original NER model is tuned based on the semantic association features .

Experimental results on English and Chinese corpus show that LaSA-based domain adaptation significantly enhances the performance of NER .

Named entities ( NE ) are phrases that contain names of persons , organizations , locations , etc .

NER is an important task in information extraction and natural language processing ( NLP ) applications .

Supervised learning methods can effectively solve NER problem by learning a model from manually labeled data ( Borthwick , 1999 ; Sang and Meulder , 2003 ; Gao et al . , 2005 ; Florian et al . , 2003 ) .

However , empirical study shows that NE types have different distribution across domains ( Guo et al . , 2006 ) .

Trained NER classifiers in the source domain usually lose accuracy in a new target domain when the data distribution is different between both domains .

Domain adaptation is a challenge for NER and other NLP applications .

In the domain transfer , the reason for accuracy loss is that each NE type often has various specific term representations and context clues in the different domains .

For example , { " economist " , " singer " , " dancer " , " athlete " , " player " , " philosopher " , ... } are used as context clues for NER .

However , the distribution of these representations are varied with domains .

We expect to do better domain adaptation for NER by exploiting latent semantic association among words from different domains .

Some approaches have been proposed to group words into " topics " to capture important relationships between words , such as Latent Semantic Indexing ( LSI ) ( Deerwester et al . , 1990 ) , probabilistic Latent Semantic Indexing ( pLSI ) ( Hofmann , 1999 ) , Latent Dirichlet Allocation ( LDA ) ( Blei et al . , 2003 ) .

These models have been successfully employed in topic modeling , dimensionality reduction for text categorization ( Blei et al . , 2003 ) , ad hoc IR ( Wei and Croft . , 2006 ) , and so on .

In this paper , we present a domain adaptation method with latent semantic association .

We focus on capturing the hidden semantic association among words in the domain adaptation .

We introduce the LaSA model to overcome the distribution difference between the source domain and the target domain .

LaSA model is constructed from the unlabeled corpus at first .

It learns latent semantic association among words from their related context snippets .

In the domain transfer , words in the corpus are associated with a low-dimension concept space using LaSA model , then the original NER model is tuned using these generated semantic association features .

The intuition behind our method is that words in one concept set will have similar semantic features or latent semantic association , and share syntactic and semantic context in the corpus .

They can be considered as behaving in the same way for discriminative learning in the source and target domains .

The proposed method associates words from different domains on a semantic level rather than by lexical occurrence .

It can better bridge the domain distribution gap without any labeled target domain samples .

Experimental results on English and Chinese corpus show that LaSA-based adaptation significantly enhances NER performance across domains .

The rest of this paper is organized as follows .

Section 2 briefly describes the related works .

Section 3 presents a domain adaptation method based on latent semantic association .

Section 4 illustrates how to learn LaSA model from the unlabeled corpus .

Section 5 shows experimental results on large-scale English and Chinese corpus across domains , respectively .

We present a domain adaptation method with LaSA model in this paper .

LaSA model captures latent semantic association among words from the unlabeled corpus .

It better groups words into a set of concepts according to the related context snippets .

LaSAbased domain adaptation method projects words to a low-dimension concept feature space in the transfer .

It effectively overcomes the data distribution gap across domains without using any labeled target domain data .

Experimental results on English and Chinese corpus show that LaSA-based domain adaptation significantly enhances the performance of NER across domains .

Especially , LaSA model effectively increases the accuracy of each NE type recognition in the domain transfer .

Moreover , LaSA-based domain adaptation method works well across languages .

To further reduce the accuracy loss , we will explore informative sampling to capture fine-grained data difference in the domain transfer .

LaSA model is constructed from the unlabeled corpus at first .

We present a domain adaptation method with LaSA model in this paper .


Manual evaluation of translation quality is generally thought to be excessively time consuming and expensive .

We explore a fast and inexpensive way of doing it using Amazon 's Mechanical Turk to pay small sums to a large number of non-expert annotators .

For $ 10 we redundantly recreate judgments from a WMT08 translation task .

We find that when combined non-expert judgments have a high-level of agreement with the existing gold-standard judgments of machine translation quality , and correlate more strongly with expert judgments than Bleu does .

We go on to show that Mechanical Turk can be used to calculate human-mediated translation edit rate ( HTER ) , to conduct reading comprehension experiments with machine translation , and to create high quality reference translations .

Conventional wisdom holds that manual evaluation of machine translation is too time-consuming and expensive to conduct .

Instead , researchers routinely use automatic metrics like Bleu ( Papineni et al . , 2002 ) as the sole evidence of improvement to translation quality .

Automatic metrics have been criticized for a variety of reasons ( Babych and Hartley , 2004 ; Callison-Burch et al . , 2006 ; Chiang et al . , 2008 ) , and it is clear that they only loosely approximate human judgments .

Therefore , having people evaluate translation output would be preferable , if it were more practical .

In this paper we demonstrate that the manual evaluation of translation quality is not as expensive or as time consuming as generally thought .

We use Amazon 's Mechanical Turk , an online labor market that is designed to pay people small sums of money to complete human intelligence teststasks that are difficult for computers but easy for people .

Non-expert annotators produce judgments that are very similar to experts and that have a stronger correlation than Bleu .

• Mechanical Turk can be used for complex tasks like human-mediated translation edit rate ( HTER ) and creating multiple reference translations .

• Evaluating translation quality through reading comprehension , which is rarely done , can be easily accomplished through creative use of Mechanical Turk .

2 Related work Snow et al . ( 2008 ) examined the accuracy of labels created using Mechanical Turk for a variety of natural language processing tasks .

These tasks included word sense disambiguation , word similarity , textual entailment , and temporal ordering of events , but not machine translation .

Snow et al . measured the quality of non-expert annotations by comparing them against labels that had been previously created by expert annotators .

They report inter-annotator agreement between expert and non-expert annotators , and show that the average of many non-experts converges on performance of a single expert for many of their tasks .

Although it is not common for manual evaluation results to be reported in conference papers , several large-scale manual evaluations of machine translation quality take place annually .

These include public forums like the NIST MT Evaluation Workshop , IWSLT and WMT , as well as the project-specific Go / No Go evaluations for the DARPA GALE program .

Various types of human judgments are used .

NIST collects 5-point fluency and adequacy scores ( LDC , 2005 ) , IWSLT and WMT collect relative rankings ( Callison-Burch et al . , 2008 ; Paul , 2006 ) , and DARPA evaluates using HTER ( Snover et al . , 2006 ) .

The details of these are provided later in the paper .

Public evaluation campaigns provide a ready source of goldstandard data that non-expert annotations can be compared to .

Mechanical Turk is an inexpensive way of gathering human judgments and annotations for a wide variety of tasks .

In this paper we demonstrate that it is feasible to perform manual evaluations of machine translation quality using the web service .

The low cost of the non-expert labor found on Mechanical Turk is cheap enough to collect redundant annotations , which can be utilized to ensure translation quality .

By combining the judgments of many non-experts we are able to achieve the equivalent quality of experts .

The suggests that manual evaluation of translation quality could be straightforwardly done to validate performance improvements reported in conference papers , or even for mundane tasks like tracking incremental system updates .

This challenges the conventional wisdom which has long held that automatic metrics must be used since manual evaluation is too costly and timeconsuming .

We have shown that Mechanical Turk can be used creatively to produce quite interesting things .

We showed how a reading comprehension test could be created , administered , and graded , with only very minimal intervention .

We believe that it is feasible to use Mechanical Turk for a wide variety of other machine translated tasks like creating word alignments for sentence pairs , verifying the accuracy of document-and sentence-alignments , performing non-simulated active learning experiments for statistical machine translation , even collecting training data for low resource languages like Urdu .

The cost of using Mechanical Turk is low enough that we might consider attempting quixotic things like human-in-the-loop minimum error rate training ( Zaidan and Callison-Burch , 2009 ) , or doubling the amount of training data available for Urdu .


State-of-the-art networks that model relations between two pieces of text often use complex architectures and attention .

In this paper , instead of focusing on architecture engineering , we take advantage of small amounts of labelled data that model semantic phenomena in text to encode matching features directly in the word representations .

This greatly boosts the accuracy of our reference network , while keeping the model simple and fast to train .

Our approach also beats a tree kernel model that uses similar input encodings , and neural models which use advanced attention and compare-aggregate mechanisms .

Modeling a match between pieces of text is at the core of many NLP tasks .

Recently , manual feature engineering methods have been shadowed by neural network approaches .

These networks model the interaction of two pieces of text , or word-toword interactions across sentences , using sophisticated attention mechanisms ( Wang et al . , 2016a ; Santos et al . , 2016 ) and compare-aggregate frameworks ( He and Lin , 2016 ; Wang et al . , 2017 ) .

Architectural complexity is tied to longer training times 1 .

Meaningful features may take long time to emerge by only leveraging word representations and the training data of the task at hand .

This is especially problematic with little data , as it often happens in question answering ( QA ) tasks , e.g. , answer sentence selection ( Wang et al . , 2007 ; Yang et al . , 2015 ) .

Thus , effective word representations are crucial in neural network models to get state-of-the-art performance .

In this work , we try to answer the following research questions : ( i ) in addition to lexical links , can we incorporate higher-level semantic links between the words in a question and a candidate answer passage , and ( ii ) can we show that such information has an impact on the quality of our model , and also allows us to keep the architecture simple ?

We show that modeling semantic relations improves the performance of a neural network for answer sentence selection with ( i ) a little number of semantic annotations , and ( ii ) a little increase in training time w.r.t .

In this paper , we presented a neural network that models semantic links between questions and answers , in addition to lexical links .

The annotations for establishing such links are produced by a set of fast neural components for question analysis , trained on publicly available datasets .

The evaluation on two QA datasets shows that our approach can achieve state-of-the-art performance using a simple CNN , leading to a low complexity and training time .

Our approach is an interesting first step towards a future architecture , in which we will jointly optimize the semantic annotators and the answer sentence selection model , in an end-to-end fashion .


Language representations are known to carry certain associations ( e.g. , gendered connotations ) which may lead to invalid and harmful predictions in downstream tasks .

While existing methods are effective at mitigating such unwanted associations by linear projection , we argue that they are too aggressive : not only do they remove such associations , they also erase information that should be retained .

To address this issue , we propose OS-CAR ( Orthogonal Subspace Correction and Rectification ) , a balanced approach of mitigation that focuses on disentangling associations between concepts that are deemed problematic , instead of removing concepts wholesale .

We develop new measurements for evaluating information retention relevant to the debiasing goal .

Our experiments on genderoccupation associations show that OSCAR is a well-balanced approach that ensures that semantic information is retained in the embeddings and unwanted associations are also effectively mitigated .

Keeping in mind that removing bias and retaining information have to be done in synergy , we present how to obtain aggregated measurements for these two components .

Word embeddings are used extensively across natural language processing ( NLP ) and succinctly capture not only the syntactic and semantic structure of language , but also word meaning in context .

As such , word embeddings are essential building blocks for today 's state-of-the-art in NLP .

But they are also known to capture a significant amount of stereotypical associations ( e.g. , Bolukbasi et al . , 2016 ; Zhao et al . , 2017 ; Dev and Phillips , 2019 ; Sun et al . , 2019 ) related to gender , race , nationality , or religion , which can manifest in unwanted and/or potentially harmful ways in downstream tasks ( Webster et al . , 2018 ; Zhao et al . , 2019 ; Dev et al . , 2020 ) .

Such potentially problematic associations , when embedded in word representations , can lead to incorrect and unfair decisions about large groups of people .

While the term " bias " has many meanings , in this paper we use it to refer to these unwanted stereotypical associations .

Existing methods to mitigate these effects either require expensive retraining of vectors ( Lemoine et al . , 2018 ) which can be inefficient , or projecting out information contained along an entire subspace representing a protected concept ( such as gender or race ) in the embedding space ( e.g. , Bolukbasi et al . , 2016 ; Dev and Phillips , 2019 ; Ravfogel et al . , 2020 ) .

Projective approaches are difficult to control as they are either insufficient : removing a subspace can still leave residual bias ( Gonen and Goldberg , 2019 ; Lauscher et al . , 2020 ) , or too aggressive : in the case of gender , also unnecessarily altering the association between the word pregnant and words like female and mother .

In tasks such as coreference resolution , removing such associations could hinder reference resolution .

To quantify how much valid information is retained , we look at the output space of models .

Following Dev et al . ( 2020 ) , we use Natural Language Inference ( NLI ) as an effective quantitative probe .

Here , we construct the hypothesis by making minimal edits to a premise , and observe model prediction conditioned on such changes .

For example , Premise : A matriarch sold a watch .

Here , the objective is to determine if the hypothesis is entailed by the premise , contradicted by it , or neither ( neutral to it ) .

A GloVe-based NLI ( Parikh et al . , 2016 ) model , without any explicit form of bias mitigation , predicts label entail with a high probability of 97 % ; the notion of a matriarch being a woman is correctly identified by the model .

However , after projective debiasing , the model classifies the pair as neutral with a probability 62 % while the probability of the label entail drops to much lower at 16 % .

That is , aggressive mitigation of gender representations erases valid gender associations .

Ideally , we should correct problematic associations without erasing valid ones .

To this end , we propose OSCAR ( Othogonal Subspace Correction and Rectification ) which orthogonalizes and rectifies identified subspaces of concepts that are incorrectly associated in an embedding space .

Embeddings outside the subspace are stretched in a graded manner or untouched .

We argue that mitigating unwanted stereotypical associations should go beyond information removal ( e.g. , projecting out features ) , and should also preserve pertinent associations .

We present OSCAR1 , a completely different method from the existing projective approaches ; it uses orthogonalization of subspaces desired not to have interdependence , and so minimal change is made to embeddings to prevent loss of desired associations .

We develop a combination of tests based on NLI to evaluate both bias mitigation and information retention .

Our experiments show that OSCAR is a wellbalanced approach that mitigates biases as good as projective approaches while retaining more valid associations .

Our contributions , focusing specifically on representations rather than classifiers , are important given the preponderance of distributed representations of text across NLP .

Predictions from systems that use such representations , if unchecked , could lead to real-world decisions ( involving , e.g. hiring ) that " systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others " ( Friedman and Nissenbaum , 1996 ) .

In an effort to prevent such transition of representational harms into allocational harms ( cf . Crawford , 2017 ; Abbasi et al . , 2019 ; Blodgett et al . , 2020 ) , we look at mitigating stereotyping biases at the source , i.e. the embedding space .

2 Bias , Gender and NLP Bias and Related Work .

Social biases in tasks using machine learning have the potential to cause harms ( Barocas and Selbst , 2016 ) and are widely studied .

Such biases in language technologies have been detected ( Bolukbasi et al . , 2016 ; Gonen and Goldberg , 2019 ) , measured ( Caliskan et al . , 2017 ; Webster et al . , 2018 ; Lauscher et al . , 2020 ) and mitigated ( Zhao et al . , 2018 ; Ravfogel et al . , 2020 ) .

Most work focuses on gender bias , and in particular , the stereotypical associations of occupations to males and females ( Rudinger et al . , 2018 ; De-Arteaga et al . , 2019 ; Gaut et al . , 2019 ; Dev et al . , 2020 ) .

In this work too , bias refers to stereotypical associations in language representations .

The concept of gender is a complex one ( cf .

In our experiments , we identify a subspace in the representation associated with gender , typically derived by terms specifically associated with notions of male and female .

This aligns with prominent recent NLP research ( Caliskan et al . , 2017 ; Bolukbasi et al . , 2016 ; Dev and Phillips , 2019 ; Ravfogel et al . , 2020 ) that try to measure and mitigate gender bias ( either implicitly or explicitly ) , directly focusing on male versus female associations .

The underlying goal of this work is to disassociate two concepts .

We mainly use concepts male-vsfemale and occupations to demonstrate this .

There is explicit and measurable bias encoded in word vector representations ( Bolukbasi et al . , 2016 ) by the male-vs-female relation to occupations , and standard evaluation metrics ( Caliskan et al . , 2017 ; Dev et al . , 2020 ) are designed to measure this .

As such many of our experiments are geared towards these male-vs-female issues .

The notion of gender is not restricted to male and female .

But because " conceptualizations of gender as binary are reinforced linguistically in English through pronoun conventions " ( Blodgett , 2021 ) , gender notions beyond male and female are either missing or poorly represented .

As a result , their associations with other concepts are also not robust .

Quantifying this effect , and evaluating its attenuation , is beyond the scope of this paper ; however the techniques we study for disassociating two concepts are amenable to such tasks in general .

We remark that a potential harm towards nonmale / female people arises through omission ( Cao and Daumé III , 2020 ) , and as a result , we explicitly encourage further research on this topic .

sign of our evaluation dataset in § 3.1 , then present evaluation metrics .

Then , we will discuss how to measure biases in § 3.2 and information retention in § 3.3 .

To be comprehensive , we will use metrics that are both intrinsic to embedding geometry and extrinsic ones that focus on downstream NLI model outputs .

The propagation of undesirable and stereotypical associations learned from data into decisions made by language models maintains a vicious cycle of biases .

Combating biases before deploying representations is thus extremely vital .

Word embeddings capture a lot of information implicitly in relatively few dimensions .

These implicit associations are what makes them state-of-the-art at tackling different language modeling tasks .

Breaking down these associations for bias rectification , thus , has to be done carefully so as to not destroy the structure of the embeddings .

OSCAR 's rectification of associations helps integrate both these aspects , allowing it to be more suitable at making word embeddings more usable .

Moreover , being computationally lightweight and sub-differentiable , it is simple to apply adaptively without extensive retraining .

OSCAR dissociates concepts which may be otherwise lost or overlapped in distributed representations by re-orthogonalizing them .

We envision that this method can be used for many types of unwanted associations ( age-ability , religion-virtue perception , etc ) .

Since it only decouples specific associations , informative components of these features will remain unchanged .

All the experiments in this paper are based on English embeddings , models and tests ; extending our analysis to other languages is important future work .

Further , we be-lieve OSCAR can extend in a straightforward way beyond NLP to other distributed , vectorized representations ( e.g. , of images , graphs , or spatial data ) which can also exhibit stereotypical associations .

As such , word embeddings are essential building blocks for today 's state-of-the-art in NLP .

We develop new measurements for evaluating information retention relevant to the debiasing goal .

Embeddings outside the subspace are stretched in a graded manner or untouched .


Humor plays an important role in human languages and it is essential to model humor when building intelligence systems .

Among different forms of humor , puns perform wordplay for humorous effects by employing words with double entendre and high phonetic similarity .

However , identifying and modeling puns are challenging as puns usually involved implicit semantic or phonological tricks .

In this paper , we propose Pronunciation-attentive Contextualized Pun Recognition ( PCPR ) to perceive human humor , detect if a sentence contains puns and locate them in the sentence .

PCPR derives contextualized representation for each word in a sentence by capturing the association between the surrounding context and its corresponding phonetic symbols .

Extensive experiments are conducted on two benchmark datasets .

Results demonstrate that the proposed approach significantly outperforms the state-of-the-art methods in pun detection and location tasks .

In-depth analyses verify the effectiveness and robustness of PCPR .

During the last decades , social media has promoted the creation of a vast amount of humorous web contents ( Nijholt et al . , 2017 ) .

Automatic recognition of humor has become an important task in the area of figurative language processing , which can benefit various downstream NLP applications such as dialogue systems , sentiment analysis , and machine translation ( Melby and Warner , 1995 ; Augello et al . , 2008 ; Ghosh et al . , 2015 ; Bertero and Fung , 2016 ; Blinov et al . , 2019 ) .

However , humor is one of the most complicated behaviors in natural language semantics and sometimes it is even difficult for humans to interpret .

In most cases , understanding humor requires adequate background knowledge and a rich context .

Puns are a form of humorous approaches using the different meanings of identical words or words with similar pronunciations to explain texts or utterances .

Homographic puns rely on multiple interpretations of the same word .

As shown in Table 1 , the phrase all right means good condition or opposite to left ; the word reaction means chemical change or action .

The two meanings of the same expression are consistent with its context , which creates a humorous pun in both sentences when there is a clear contrast between two meanings .

On the other hand , heterographic puns take advantage of phonologically same or similar words .

For example , the word pairs sale and sail , weak and week in Table 1 have the same or similar pronunciations .

The sentences are funny because both words fit the same context .

Understanding puns is a big fish to fry for deep comprehension of complex semantics .

These two forms of puns have been studied in literature from different angles .

To recognize puns in a sentence , word sense disambiguation techniques ( WSD ) ( Navigli , 2009 ) have been employed to identify the equitable intention of words in utterances ( Pedersen , 2017 ) .

External knowledge bases such as WordNet ( Miller , 1998b ) have been applied in determining word senses of pun words ( Oele and Evang , 2017 ) .

However , these methods can not tackle heterographic puns with distinct word spellings and knowledge bases that only contain a limited vocabulary .

To resolve the issues of sparseness and heterographics , the word embedding techniques ( Mikolov et al . , 2013 ; Pennington et al . , 2014 ) provide flexible representations to model puns ( Hurtado et al . , 2017 ; Indurthi and Oota , 2017 ; Cai et al . , 2018 ) .

However , a word may have different meanings regarding its contexts .

Especially , an infrequent meaning of the word might be utilized for creating a pun .

Therefore , static word embeddings are insufficient to represent words .

In addition , some puns are created by replacing a word with another word with the same or similar pronunciation as examples shown in Table 1 .

Therefore , to recognize puns , it is essential to model the association between words in the sentence and the pronunciation of words .

Despite existing approaches attempt to leverage phonological structures to understand puns ( Doogan et al . , 2017 ; Jaech et al . , 2016 ) , there is a lack of a general framework to model these two types of signals in a whole .

In this paper , we propose Pronunciation-attentive Contextualized Pun Recognition ( PCPR ) to jointly model the contextualized word embeddings and phonological word representations for pun recognition .

To capture the phonological structures of words , we break each word into a sequence of phonemes as its pronunciation so that homophones can have similar phoneme sets .

For instance , the phonemes of the word pun are { P , AH , N } .

In PCPR , we construct a pronunciation attentive module to identify important phonemes of each word , which can be applied in other tasks related to phonology .

We jointly encode the contextual and phonological features into a self-attentive embedding to tackle both pun detection and location tasks .

We summarize our contributions as following .

• To the best of our knowledge , PCPR is the first work to jointly model contextualized word embeddings and pronunciation embeddings to recognize puns .

Both contexts and phonological properties are beneficial to pun recognition .

In this paper , we propose a novel approach , PCPR , for pun detection and location by leveraging a contextualized word encoder and modeling phonemes as word pronunciations .

Moreover , we would love to apply the proposed model to other problems , such as general humor recognition , irony discovery , and sarcasm detection , as the future work .


Part-of-speech ( POS ) tag distributions are known to exhibit sparsity -a word is likely to take a single predominant tag in a corpus .

Recent research has demonstrated that incorporating this sparsity constraint improves tagging accuracy .

However , in existing systems , this expansion come with a steep increase in model complexity .

This paper proposes a simple and effective tagging method that directly models tag sparsity and other distributional properties of valid POS tag assignments .

In addition , this formulation results in a dramatic reduction in the number of model parameters thereby , enabling unusually rapid training .

Our experiments consistently demonstrate that this model architecture yields substantial performance gains over more complex tagging counterparts .

On several languages , we report performance exceeding that of more complex state-of-the art systems .

Since the early days of statistical NLP , researchers have observed that a part-of-speech tag distribution exhibits " one tag per discourse " sparsity -words are likely to select a single predominant tag in a corpus , even when several tags are possible .

Simply assigning to each word its most frequent associated tag in a corpus achieves 94.6 % accuracy on the WSJ portion of the Penn Treebank .

This distributional sparsity of syntactic tags is not unique to English 1

The source code for the work presented in this paper is available at http://groups.csail.mit.edu / rbg / code / typetagging/.

-similar results have been observed across multiple languages .

Clearly , explicitly modeling such a powerful constraint on tagging assignment has a potential to significantly improve the accuracy of an unsupervised part-of-speech tagger learned without a tagging dictionary .

In practice , this sparsity constraint is difficult to incorporate in a traditional POS induction system ( Mérialdo , 1994 ; Johnson , 2007 ; Gao and Johnson , 2008 ; Grac ¸a et al . , 2009 ; Berg-Kirkpatrick et al . , 2010 ) .

These sequence models-based approaches commonly treat token-level tag assignment as the primary latent variable .

By design , they readily capture regularities at the token-level .

However , these approaches are ill-equipped to directly represent type-based constraints such as sparsity .

Previous work has attempted to incorporate such constraints into token-level models via heavy-handed modifications to inference procedure and objective function ( e.g. , posterior regularization and ILP decoding ) ( Grac ¸a et al . , 2009 ; Ravi and Knight , 2009 ) .

In most cases , however , these expansions come with a steep increase in model complexity , with respect to training procedure and inference time .

In this work , we take a more direct approach and treat a word type and its allowed POS tags as a primary element of the model .

The model starts by generating a tag assignment for each word type in a vocabulary , assuming one tag per word .

Then , tokenlevel HMM emission parameters are drawn conditioned on these assignments such that each word is only allowed probability mass on a single assigned tag .

In this way we restrict the parameterization of a token-level HMM to reflect lexicon sparsity .

This model admits a simple Gibbs sampling algorithm where the number of latent variables is proportional to the number of word types , rather than the size of a corpus as for a standard HMM sampler ( Johnson , 2007 ) .

There are two key benefits of this model architecture .

First , it directly encodes linguistic intuitions about POS tag assignments : the model structure reflects the one-tag-per-word property , and a typelevel tag prior captures the skew on tag assignments ( e.g. , there are fewer unique determiners than unique nouns ) .

Second , the reduced number of hidden variables and parameters dramatically speeds up learning and inference .

We evaluate our model on seven languages exhibiting substantial syntactic variation .

On several languages , we report performance exceeding that of state-of-the art systems .

Our analysis identifies three key factors driving our performance gain : 1 ) selecting a model structure which directly encodes tag sparsity , 2 ) a type-level prior on tag assignments , and 3 ) a straightforward naïve-Bayes approach to incorporate features .

The observed performance gains , coupled with the simplicity of model implementation , makes it a compelling alternative to existing more complex counterparts .

We have presented a method for unsupervised partof-speech tagging that considers a word type and its allowed POS tags as a primary element of the model .

This departure from the traditional token-based tagging approach allows us to explicitly capture typelevel distributional properties of valid POS tag as-signments as part of the model .

The resulting model is compact , efficiently learnable and linguistically expressive .

Our empirical results demonstrate that the type-based tagger rivals state-of-the-art tag-level taggers which employ more sophisticated learning mechanisms to exploit similar constraints .

In this paper , we make a simplifying assumption of one-tag-per-word .

This assumption , however , is not inherent to type-based tagging models .

A promising direction for future work is to explicitly model a distribution over tags for each word type .

We hypothesize that modeling morphological information will greatly constrain the set of possible tags , thereby further refining the representation of the tag lexicon .