Sentiment classification aims to automatically predict sentiment polarity ( e.g. , positive or negative ) of user generated sentiment data ( e.g. , reviews , blogs ) .

Due to the mismatch among different domains , a sentiment classifier trained in one domain may not work well when directly applied to other domains .

Thus , domain adaptation for sentiment classification algorithms are highly desirable to reduce the domain discrepancy and manual labeling costs .

To address the above challenge , we propose a novel domain adaptation method , called Bi-Transferring Deep Neural Networks ( BTDNNs ) .

The proposed BTDNNs attempts to transfer the source domain examples to the target domain , and also transfer the target domain examples to the source domain .

The linear transformation of BTDNNs ensures the feasibility of transferring between domains , and the distribution consistency between the transferred domain and the desirable domain is constrained with a linear data reconstruction manner .

As a result , the transferred source domain is supervised and follows similar distribution as the target domain .

Therefore , any supervised method can be used on the transferred source domain to train a classifier for sentiment classification in a target domain .

We conduct experiments on a benchmark composed of reviews of 4 types of Amazon products .

Experimental results show that our proposed approach significantly outperforms the several baseline methods , and achieves an accuracy which is competitive with the state-of-the-art method for domain adaptation .

With the rise of social media ( e.g. , blogs and social networks etc . ) , more and more user generated sentiment data have been shared on the Web ( Pang et al . , 2002 ; Pang and Lee , 2008 ; Liu , 2012 ; Zhou et al . , 2011 ) .

They exist in the form of user reviews on shopping or opinion sites , in posts of blogs / questions or customer feedbacks .

This has created a surge of research in sentiment classification ( or sentiment analysis ) , which aims to automatically determine the sentiment polarity ( e.g. , positive or negative ) of user generated sentiment data ( e.g. , reviews , blogs , questions ) .

Machine learning algorithms have been proved promising and widely used for sentiment classification ( Pang et al . , 2002 ;

However , the performance of these models relies on manually labeled training data .

In many practical cases , we may have plentiful labeled data in the source domain , but very few or no labeled data in the target domain with a different data distribution .

For example , we may have many labeled books reviews , but we are interested in detecting the polarity of electronics reviews .

Reviews for different products might have different vocabularies , thus classifiers trained on one domain often fail to produce satisfactory results when transferring to another domain .

This has motivated much research on cross-domain ( domain adaptation ) sentiment classification which transfers the knowledge from the source domain to the target domain ( Thomas et al . , 2006 ; Snyder and Barzilay , 2007 ; Blitzer et al . , 2006 ; Blitzer et al . , 2007 ; Daume III , 2007 ; Li and Zong , 2008 ; Li et al . , 2009 ; Pan et al . , 2010 ; Kumar et al . , 2010 ; Glorot et al . , 2011 ; Chen et al . , 2011a ; Chen et al . , 2012 ; Li et al . , 2012 ; Xia et al . , 2013a ; Li et al . , 2013 ; Zhou et al . , 2015a ; Zhuang et al . , 2015 ) .

Depending on whether the labeled data are available for the target domain , cross-domain sen-timent classification can be divided into two categories : supervised domain adaptation and unsupervised domain adaptation .

In scenario of supervised domain adaptation , labeled data is available in the target domain but the number is usually too small to train a good sentiment classifier , while in unsupervised domain adaptation only unlabeled data is available in the target domain , which is more challenging .

This work focuses on the unsupervised domain adaptation problem of which the essence is how to employ the unlabeled data of target domain to guide the model learning from the labeled source domain .

The fundamental challenge of cross-domain sentiment classification lies in that the source domain and the target domain have different data distribution .

Recent work has investigated several techniques for alleviating the domain discrepancy : instance-weight adaptation ( Huang et al . , 2007 ; Jiang and Zhai , 2007 ; Li and Zong , 2008 ; Mansour et al . , 2009 ; Dredze et al . , 2010 ; Chen et al . , 2011b ; Chen et al . , 2011a ; Chen et al . , 2012 ; Li et al . , 2013 ; Xia et al . , 2013a ) and feature representation adaptation ( Thomas et al . , 2006 ; Snyder and Barzilay , 2007 ; Blitzer et al . , 2006 ; Blitzer et al . , 2007 ; Li et al . , 2009 ; Pan et al . , 2010 ; Zhou et al . , 2015a ; Zhuang et al . , 2015 ) .

The first kind of methods assume that some training data in the source domain are very useful for the target domain and these data can be used to train models for the target domain after re-weighting .

In contrast , feature representation approaches attempt to develop an adaptive feature representation that is effective in reducing the difference between domains .

Recently , some efforts have been initiated on learning robust feature representations with deep neural networks ( DNNs ) in the context of crossdomain sentiment classification ( Glorot et al . , 2011 ; Chen et al . , 2012 ) .

Glorot et al . ( 2011 ) proposed to learn robust feature representations with stacked denoising auto-encoders ( SDAs ) ( Vincent et al . , 2008 ) .

Denoising auto-encoders are onelayer neural networks that are optimized to reconstruct input data from partial and random corruption .

These denoisers can be stacked into deep learning architectures .

The outputs of their intermediate layers are then used as input features for SVMs ( Fan et al . , 2008 ) .

Chen et al . ( 2012 ) proposed a marginalized SDA ( mSDA ) that addressed the two crucial limitations of SDAs : high computational cost and lack of scalability to highdimensional features .

However , these methods learn the unified domain-invariable feature representations by combining the source domain data and that of the target domain data together , which can not well characterize the domain-specific features as well as the commonality of domains .

To this end , we propose a Bi-Transferring Deep Neural Networks ( BTDNNs ) which can transfer the source domain examples to the target domain and also transfer the target domain examples to the source domain , as shown in Figure 1 .

In BTDNNs , the linear transformation makes the feasibility of transferring between domains , and the linear data reconstruction manner ensures the distribution consistency between the transferred domain and the desirable domain .

Specifically , our BTDNNs has one common encoder f c , two decoders g s and g t which can map an example to the source domain and the target domain respectively .

As a result , the source domain can be transferred to the target domain along with its sentiment label , and any supervised method can be used on the transferred source domain to train a classifier for sentiment classification in the target domain , as the transferred source domain data share the similar distribution as the target domain .

Experimental results show that the proposed approach significantly outperforms several baselines , and achieves an accuracy which is competitive with the state-of-the-art method for cross-domain sentiment classification .

The remainder of this paper is organized as follows .

Section 3 describes our proposed bi-transferring deep neural networks ( BTDNNs ) .

Section 4 presents the experimental results .

In Section 5 , we conclude with ideas for future research .

In this paper , we propose a novel Bi-Transferring Deep Neural Networks ( BTDNNs ) for crossdomain sentiment classification .

Experimental results show that BTDNNs significantly outperforms the several baselines , and achieves an accuracy which is competitive with the state-of-the-art method for sentiment classification adaptation .

There are some ways in which this research could be continued .

First , since deep learning may obtain better generalization on large-scale data sets ( Bengio , 2009 ) , a straightforward path of the future research is to apply the proposed BTDNNs for domain adaptation on a much larger industrial-strength data set of 22 domains ( Glorot et al . , 2011 ) .

Second , we will try to investigate the use of the proposed approach for other kinds of data set , such as 20 newsgroups and Reuters-21578 ( Li et al . , 2012 ; Zhuang et al . , 2013 ) .


This paper describes a dependency structure analysis of Japanese sentences based on the maximum entropy models .

Our model is created by learning the weights of some features from a training corpus to predict the dependency between bunsetsus or phrasal units .

The dependency accuracy of our system is 87.2 % using the Kyoto University corpus .

We discuss the contribution of each feature set and the relationship between the number of training data and the accuracy .

Dependency structure analysis is one of the basic techniques in Japanese sentence analysis .

The Japanese dependency structure is usually represented by the relationship between phrasal units called ' bunsetsu . '

In the first step , a dependency matrix is prepared .

Each element of the matrix represents how likely one bunsetsu is to depend on the other .

In the second step , an optimal set of dependencies for the entire sentence is found .

In this paper , we will mainly discuss the first step , a model for estimating dependency likelihood .

So far there have been two different approaches to estimating the dependency likelihood , One is the rule-based approach , in which the rules are created by experts and likelihoods are calculated by some means , including semiautomatic corpusbased methods but also by manual assignment of scores for rules .

However , hand-crafted rules have the following problems .

• They have a problem with their coverage .

Because there are many features to find correct dependencies , it is difficult to find them manually .

They also have a problem with their consistency , since many of the features compete with each other and humans can not create consistent rules or assign consistent scores .

As syntactic characteristics differ across different domains , the rules have to be changed when the target domain changes .

It is costly to create a new hand-made rule for each domain .

At / other approach is a fully automatic corpusbased approach .

This approach has the potential to overcome the problems of the rule-based approach .

It automatically learns the likelihoods of dependencies from a tagged corpus and calculates the best dependencies for an input sentence .

This approach is taken by some other systems ( Collins , 1996 ; Fujio and Matsumoto , 1998 ; Haruno et ah , 1998 ) .

The parser proposed by Ratnaparkhi ( Ratnaparkhi , 1997 ) is considered to be one of the most accurate parsers in English .

Its probability estimation is based on the maximum entropy models .

This model learns the weights of given features from a training corpus .

The weights are calculated based on the frequencies of the features in the training data .

The set of features is defined by a human .

In our model , we use features of bunsetsu , such as character strings , parts of speech , and inflection types of bunsetsu , as well as information between bunsetsus , such as the existence of punctuation , and the distance between bunsetsus .

The probabilities of dependencies are estimated from the model by using those features in input sentences .

We assume that the overall dependencies in a whole sentence can be determined as the product of the probabilities of all the dependencies in the sentence .

Now , we briefly describe the algorithm of dependency analysis .

It is said that Japanese dependencies have the following characteristics .

( 1 ) Dependencies are directed from left to right ( 2 ) Dependencies do not cross ( 3 ) A bunsetsu , except for the rightmost one , depends on only one bunsetsu ( 4 ) In many cases , the left context is not necessary to determine a dependency 1

The analysis method proposed in this paper is designed to utilize these features .

Based on these properties , we detect the dependencies in a sentence by analyzing it backwards ( from right to left ) .

In the past , such a backward algorithm has been used with rule-based parsers ( e.g. , ( Fujita , 1988 ) ) .

We applied it to our statistically based approach .

Because of the statistical property , we can incorporate a beam search , an effective way of limiting the search space in a backward analysis .

This paper described a Japanese dependency structure analysis based on the maximum entropy model .

The probabilities of dependencies between bunsetsus are estimated by this model .

The dependency accuracy of our system was 87.2 % using the Kyoto University corpus .

In our experiments without the feature sets shown in Tables 1 and 2 , we found that some basic and combined features strongly contribute to improve the accuracy .

Investigating the relationship between the number of training data and the accuracy , we found that good accuracy can be achieved even with a very small set of training data .

We believe that the maximum entropy framework has suitable characteristics for overcoming the data sparseness problem .

In particular , we are interested in how to deal with coordinate structures , since that seems to be the largest problem at the moment .

The set of features is defined by a human .


In this paper , drawing intuition from the Turing test , we propose using adversarial training for open-domain dialogue generation : the system is trained to produce sequences that are indistinguishable from human-generated dialogue utterances .

We cast the task as a reinforcement learning ( RL ) problem where we jointly train two systems , a generative model to produce response sequences , and a discriminator-analagous to the human evaluator in the Turing test-to distinguish between the human-generated dialogues and the machine-generated ones .

The outputs from the discriminator are then used as rewards for the generative model , pushing the system to generate dialogues that mostly resemble human dialogues .

In addition to adversarial training we describe a model for adversarial evaluation that uses success in fooling an adversary as a dialogue evaluation metric , while avoiding a number of potential pitfalls .

Experimental results on several metrics , including adversarial evaluation , demonstrate that the adversarially-trained system generates higher-quality responses than previous baselines .

Open domain dialogue generation ( Ritter et al . , 2011 ; Sordoni et al . , 2015 ; Xu et al . , 2016 ; Wen et al . , 2016 ; Li et al . , 2016b ; Serban et al . , 2016c Serban et al . , , 2017 ) ) aims at generating meaningful and coherent dialogue responses given the dialogue history .

Prior systems , e.g. , phrase-based machine translation systems ( Ritter et al . , 2011 ; Sordoni et al . , 2015 ) or end-to-end neural systems ( Shang et al . , 2015 ; Vinyals and Le , 2015 ; Li et al . , 2016a ; Yao et al . , 2015 ; Luan et al . , 2016 ) approximate such a goal by predicting the next dialogue utterance given the dialogue history using the maximum likelihood estimation ( MLE ) objective .

Despite its success , this over-simplified training objective leads to problems : responses are dull , generic ( Sordoni et al . , 2015 ; Serban et al . , 2016a ; Li et al . , 2016a ) , repetitive , and short-sighted ( Li et al . , 2016d ) .

Solutions to these problems require answering a few fundamental questions : what are the crucial aspects that characterize an ideal conversation , how can we quantitatively measure them , and how can we incorporate them into a machine learning system ?

For example , Li et al . ( 2016d ) manually define three ideal dialogue properties ( ease of answering , informativeness and coherence ) and use a reinforcement-learning framework to train the model to generate highly rewarded responses .

Yu et al . ( 2016b ) use keyword retrieval confidence as a reward .

However , it is widely acknowledged that manually defined reward functions ca n't possibly cover all crucial aspects and can lead to suboptimal generated utterances .

A good dialogue model should generate utterances indistinguishable from human dialogues .

Such a goal suggests a training objective resembling the idea of the Turing test ( Turing , 1950 ) .

We borrow the idea of adversarial training ( Goodfellow et al . , 2014 ; Denton et al . , 2015 ) in computer vision , in which we jointly train two models , a generator ( a neural SEQ2SEQ model ) that defines the probability of generating a dialogue sequence , and a discriminator that labels dialogues as human-generated or machine-generated .

This discriminator is analogous to the evaluator in the Turing test .

We cast the task as a reinforcement learning problem , in which the quality of machinegenerated utterances is measured by its ability to fool the discriminator into believing that it is a human-generated one .

The output from the discriminator is used as a reward to the generator , pushing it to generate utterances indistinguishable from human-generated dialogues .

The idea of a Turing test-employing an evaluator to distinguish machine-generated texts from human-generated ones-can be applied not only to training but also testing , where it goes by the name of adversarial evaluation .

Adversarial evaluation was first employed in Bowman et al . ( 2016 ) to evaluate sentence generation quality , and preliminarily studied for dialogue generation by Kannan and Vinyals ( 2016 ) .

In this paper , we discuss potential pitfalls of adversarial evaluations and necessary steps to avoid them and make evaluation reliable .

Experimental results demonstrate that our approach produces more interactive , interesting , and non-repetitive responses than standard SEQ2SEQ models trained using the MLE objective function .

In this paper , drawing intuitions from the Turing test , we propose using an adversarial training approach for response generation .

We cast the model in the framework of reinforcement learning and train a generator based on the signal from a discriminator to generate response sequences indistinguishable from human-generated dialogues .

We observe clear performance improvements on multiple metrics from the adversarial training strategy .

The adversarial training model should theoretically benefit a variety of generation tasks in NLP .

Unfortunately , in preliminary experiments applying the same training paradigm to machine translation , we did not observe a clear performance boost .

We conjecture that this is because the adversarial training strategy is more beneficial to tasks in which there is a big discrepancy between the distributions of the generated sequences and the reference target sequences .

In other words , the adversarial approach is more beneficial on tasks in which entropy of the targets is high .

Exploring this relationship further is a focus of our future work .

Zhou Yu , Ziyu Xu , Alan W Black , and Alex I Rudnicky . 2016b .

Strategy and policy learning for nontask-oriented conversational systems .

In 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue .

Yuan Zhang , Regina Barzilay , and Tommi Jaakkola . 2017 .

Aspect-augmented adversarial networks for domain adaptation .


Query segmentation is the process of taking a user 's search-engine query and dividing the tokens into individual phrases or semantic units .

Identification of these query segments can potentially improve both document-retrieval precision , by first returning pages which contain the exact query segments , and document-retrieval recall , by allowing query expansion or substitution via the segmented units .

We train and evaluate a machine-learned query segmentation system that achieves 86 % segmentationdecision accuracy on a gold standard set of segmented noun phrase queries , well above recently published approaches .

Key enablers of this high performance are features derived from previous natural language processing work in noun compound bracketing .

For example , token association features beyond simple N-gram counts provide powerful indicators of segmentation .

Billions of times every day , people around the world communicate with Internet search engines via a small text box on a web page .

The user provides a sequence of words to the search engine , and the search engine interprets the query and tries to return web pages that not only contain the query tokens , but that are also somehow about the topic or idea that the query terms describe .

Recent years have seen a widespread recognition that the user is indeed providing natural language text to the search engine ; query tokens are not independent , unordered symbols to be matched on a web document but rather ordered words and phrases with syntactic relationships .

For example , Zhai ( 1997 ) pointed out that indexing on single-word symbols is not able to distinguish a search for " bank terminology " from one for " terminology bank . "

The reader can submit these queries to a current search engine to confirm that modern indexing does recognize the effect of token order on query meaning in some way .

Accurately interpreting query semantics also depends on establishing relationships between the query tokens .

For example , consider the query " two man power saw . "

There are a number of possible interpretations of this query , and these can be expressed through a number of different segmentations or bracketings of the query terms : One simple way to make use of these interpretations in search would be to put quotation marks around the phrasal segments to require the search engine to only find pages with exact phrase matches .

If , as seems likely , the searcher is seeking pages about the large , mechanically-powered two-man saws used by lumberjacks and sawyers to cut big trees , then the first segmentation is correct .

Indeed , a phrasal search for " two man power saw " on Google does find the device of interest .

So does the second interpretation , but along with other , less-relevant pages discussing competitions involving " two-man handsaw , two-woman handsaw , power saw log bucking , etc . "

The top document returned for the third interpretation , meanwhile , describes a man on a rampage at a subway station with two cordless power saws , while the fourth interpretation finds pages about topics ranging from hockey 's thrilling two-man power play advantage to the man power situation during the Second World War .

Clearly , choosing the right segmentation means finding the right documents faster .

Query segmentation can also help if insufficient pages are returned for the original query .

A technique such as query substitution or expansion ( Jones et al . , 2006 ) can be employed using the segmented units .

For example , we could replace the sexist " two man " modifier with the politically-correct " two person " phrase in order to find additional relevant documents .

Without segmentation , expanding via the individual words " two , " " man , " " power , " or " saw " could produce less sensible results .

In this paper , we propose a data-driven , machinelearned approach to query segmentation .

Similar to previous segmentation approaches described in Section 2 , we make a decision to segment or not to segment between each pair of tokens in the query .

Unlike previous work , we view this as a classification task where the decision parameters are learned discriminatively from gold standard data .

In Section 3 , we describe our approach and the features we use .

Section 4 describes our labelled data , as well as the specific tools used for our experiments .

Section 5 provides the results of our evaluation , and shows the strong gains in performance possible using a wide set of features within a discriminative framework .

We have developed a novel approach to search query segmentation and evaluated this approach on actual user queries , reducing error by 56 % over a recent comparison approach .

Gains in performance were made possible by both leveraging recent progress in feature engineering for noun compound bracketing , as well as using a flexible , discriminative incorporation of association information , beyond the decisionboundary tokens .

We have created and made available a set of manually-segmented user queries , and thus provided a new testing platform for other researchers in this area .

Our initial formulation of query segmentation as a structured learning problem , and our leveraging of association statistics beyond the decision boundary , also provides powerful tools for noun compound bracketing researchers to both move beyond three-word compounds and to adopt discriminative feature weighting techniques .

The positive results achieved on this important application should encourage further inter-disciplinary collaboration between noun compound interpretation and information retrieval researchers .

For example , analysing the semantics of multiword expressions may allow for more-focused query expansion ; knowing to expand " bank manager " to include pages describing a " manager of the bank , " but not doing the same for non-compositional phrases like " real estate " or " private investigator , " requires exactly the kind of techniques being developed in the noun compound interpretation community .

Thus for query expansion , as for query segmentation , work in natural language processing has the potential to make a real and immediate impact on search-engine technology .

The next step in this research is to directly investigate how query segmentation affects search performance .

For such an evaluation , we would need to know , for each possible segmentation ( including no segmentation ) , the document retrieval performance .

This could be the proportion of returned documents that are deemed to be relevant to the original query .

Exactly such an evaluation was recently used by Kumaran and Allan ( 2007 ) for the related task of query contraction .

Of course , a dataset with queries and retrieval scores may serve for more than evaluation ; it may provide the examples used by the learning module .

That is , the parameters of the contraction or segmentation scoring function could be discriminatively set to optimize the retrieval of the training set queries .

A unified framework for query contraction , segmentation , and expansion , all based on discriminatively optimizing retrieval performance , is a very appealing future research direction .

In this framework , the size of the training sets would not be limited by human annotation resources , but by the number of queries for which retrieved-document relevance judgments are available .

Generating more training examples would allow the use of more powerful , finer-grained lexical features for classification .


In recent years , research in natural language processing has increasingly focused on normalizing SMS messages .

Different well-defined approaches have been proposed , but the problem remains far from being solved : best systems achieve a 11 % Word Error Rate .

This paper presents a method that shares similarities with both spell checking and machine translation approaches .

The normalization part of the system is entirely based on models trained from a corpus .

Evaluated in French by 10-fold-cross validation , the system achieves a 9.3 % Word Error Rate and a 0.83 BLEU score .

Introduced a few years ago , Short Message Service ( SMS ) offers the possibility of exchanging written messages between mobile phones .

These messages often greatly deviate from traditional spelling conventions .

As shown by specialists ( Thurlow and Brown , 2003 ; Fairon et al . , 2006 ; Bieswanger , 2007 ) , this variability is due to the simultaneous use of numerous coding strategies , like phonetic plays ( 2m1 read ' demain ' , " tomorrow " ) , phonetic transcriptions ( kom instead of ' comme ' , " like " ) , consonant skeletons ( tjrs for ' toujours ' , " always " ) , misapplied , missing or incorrect separators ( j esper for ' j'espère ' , " I hope " ; j'croibi1k , instead of ' je crois bien que ' , " I am pretty sure that " ) , etc .

These deviations are due to three main factors : the small number of characters allowed per text message by the service ( 140 bytes ) , the constraints of the small phones ' keypads and , last but not least , the fact that people mostly communicate between friends and relatives in an informal register .

Whatever their causes , these deviations considerably hamper any standard natural language processing ( NLP ) system , which stumbles against so many Out-Of-Vocabulary words .

For this reason , as noted by Sproat et al . ( 2001 ) , an SMS normalization must be performed before a more conventional NLP process can be applied .

As defined by Yvon ( 2008 ) , " SMS normalization consists in rewriting an SMS text using a more conventional spelling , in order to make it more readable for a human or for a machine . "

The SMS normalization we present here was developed in the general framework of an SMSto-speech synthesis system1 .

This paper , however , only focuses on the normalization process .

Evaluated in French , our method shares similarities with both spell checking and machine translation .

The machine translation-like module of the system performs the true normalization task .

It is entirely based on models learned from an SMS corpus and its transcription , aligned at the character-level in order to get parallel corpora .

Two spell checking-like modules surround the normalization module .

The first one detects unambiguous tokens , like URLs or phone numbers , to keep them out of the normalization .

The second one , applied on the normalized parts only , identifies non-alphabetic sequences , like punctuations , and labels them with the corresponding token .

This greatly helps the system 's print module to follow the basic rules of typography .

Section 2 proposes an overview of the state of the art .

Section 3 presents the general architecture of our system , while Section 4 focuses on how we learn and combine our normalization models .

Section 5 evaluates the system and compares it to previous works .

Section 6 draws conclusions and considers some future possible improvements of the method .

In this paper , we presented an SMS normalization framework based on finite-state machines and developed in the context of an SMS-to-speech synthesis system .

With the intention to avoid wrong modifications of special tokens and to handle word boundaries as easily as possible , we designed a method that shares similarities with both spell checking and machine translation .

Our Evaluated by ten-fold cross-validation , the system seems efficient , and the performance in terms of BLEU score and WER are quite encouraging .

However , the SER remains too high , which emphasizes the fact that the system needs several improvements .

First of all , the model should take phonetic similarities into account , because SMS messages contain a lot of phonetic plays .

The phonetic model , for instance , should know that o , au , eau , . . .

, aux can all be pronounced [ o ] , while è , ais , ait , . . .

However , unlike Kobus et al . ( 2008a ) , we feel that this model must avoid the normalization step in which the graphemic sequence is converted into phonemes , because this conversion prevents the next steps from knowing which graphemes were in the initial sequence .

Instead , we propose to learn phonetic similarities from a dictionary of words with phonemic transcriptions , and to build graphemes-to-graphemes rules .

These rules could then be automatically weighted , by learning their frequencies from our aligned corpora .

Furthermore , this model should be able to allow for timbre variation , like [ e]-[E ] , in order to allow similarities between graphemes frequently confused in French , like ai ( [ e ] ) and ais / ait / aient ( [ E ] ) .

Last but not least , the graphemes-tographemes rules should be contextualized , in order to reduce the complexity of the model .

It would also be interesting to test the impact of another lexical language model , learned on non-SMS sentences .

Indeed , the lexical model must be learned from sequences of standard written forms , an obvious prerequisite that involves a major drawback when the corpus is made of SMS sentences : the corpus must first be transcribed , an expensive process that reduces the amount of data on which the model will be trained .

For this reason , we propose to learn a lexical model from non-SMS sentences .

However , the corpus of external sentences should still share two important features with the SMS language : it should mimic the oral language and be as spontaneous as possible .

With this in mind , our intention is to gather sentences from Internet forums .

But not just any forum , because often forums share another feature with the SMS language : their language is noisy .

Thus , the idea is to choose a forum asking its members to pay attention to spelling mistakes and grammatical errors , and to avoid the use of the SMS language .


Beam search is a go-to strategy for decoding neural sequence models .

The algorithm can naturally be viewed as a subset optimization problem , albeit one where the corresponding set function does not reflect interactions between candidates .

Empirically , this leads to sets often exhibiting high overlap , e.g. , strings may differ by only a single word .

Yet in use-cases that call for multiple solutions , a diverse or representative set is often desired .

To address this issue , we propose a reformulation of beam search , which we call determinantal beam search .

Determinantal beam search has a natural relationship to determinantal point processes ( DPPs ) , models over sets that inherently encode intra-set interactions .

By posing iterations in beam search as a series of subdeterminant maximization problems , we can turn the algorithm into a diverse subset selection process .

In a case study , we use the string subsequence kernel to explicitly encourage n-gram coverage in text generated from a sequence model .

We observe that our algorithm offers competitive performance against other diverse set generation strategies in the context of language generation , while providing a more general approach to optimizing for diversity .

The decoding of neural sequence models is a fundamental component of many tasks in NLP .

Yet , many proposed decoding methods aim to produce only a single solution ; further , decoding strategies that provide a set , such as beam search , admit high overlap between solutions .

Such approaches fail to reflect that for many NLP tasks,1 there can be multiple correct solutions-or that we may desire a diverse set of solutions .

As it stands , standard beam search chooses items based purely on individual scores , with no means for encoding interaction between candidates ; this is the limitation which we attempt to address in this work .

We derive determinantal beam search , a novel generalization of beam search that casts subset selection as the subdeterminant optimization problem .

Specifically , we formulate each iteration of beam search as a subdeterminant maximization problem parameterized by a positive semi-definite matrix that encodes interactions between the possible candidates ; standard beam search is recovered by a specific diagonal matrix .

This framing creates a natural paradigm for taking the relationships between candidates during the decoding process , and can thus assign higher scores to diversified sets ; we show how this approach relates to k-determinantal point processes ( DPPs ) .

Given the wealth of research on efficient kernel computation ( Rousu and Shawe-Taylor , 2005 ; Farhan et al . , 2017 ) and DPP inference strategies ( Li et al . , 2016 ; Han et al . , 2017 ; Chen et al . , 2018 ) , we find the impact on runtime to be quite reasonable in comparison to standard decoding techniques .

In a case study on neural machine translation ( NMT ) , we demonstrate how to make use of the string subsequence kernel ( Lodhi et al . , 2002 ) to encode the notion of n-gram diversity in the language generation process , allowing us to derive an elegant diverse beam search .

Under this scheme , we observe that determinantal beam search generates more diverse sets than standard beam search with minimal trade-off in terms of BLEU .

We see improved performance over stochastic beam search ( SBS ; Kool et al . , 2019 ) , which is reported to encourage diversity , and a slight improvement over Vijayakumar et al . ( 2018 ) 's diverse beam search ( DBS ) while providing a more general approach to optimizing for intra-set diversity .

We propose determinantal beam search ( DetBS ): a new way of framing beam search that allows us to optimize set generation for diversity and coverage rather than simply individual scores .

Formally , we redefine beam search as an iterative subdeterminant maximization problems where we select the approximately maximizing set according to the PSD matrix parameterizing our score function .

This gives us the ability to encode the notion of intra-set diversity into the beam search optimization problem .

We discuss and experiment with efficient methods for inference and kernel computation that make DetBS an efficient decoding strategy in practice .

We use DetBS in the context of language generation , where we explicitly encourage n-gram coverage through the string subsequence kernel .

In our NMT experiments , we find DetBS generates much more diverse sets of strings than standard beam search and stochastic beam search with a small tradeoff in median BLEU .

We observe competitive performance compared with diverse beam search .


This paper describes a computational approach to resolving the true referent of a named mention of a person in the body of an email .

A generative model of mention generation is used to guide mention resolution .

Results on three relatively small collections indicate that the accuracy of this approach compares favorably to the best known techniques , and results on the full CMU Enron collection indicate that it scales well to larger collections .

The increasing prevalence of informal text from which a dialog structure can be reconstructed ( e.g. , email or instant messaging ) , raises new challenges if we are to help users make sense of this cacophony .

Large collections offer greater scope for assembling evidence to help with that task , but they pose additional challenges as well .

With well over 100,000 unique email addresses in the CMU version of the Enron collection ( Klimt and Yang , 2004 ) , common names ( e.g. , John ) might easily refer to any one of several hundred people .

In this paper , we associate named mentions in unstructured text ( i.e. , the body of an email and/or the subject line ) to modeled identities .

We see at least two direct applications for this work : ( 1 ) helping searchers who are unfamiliar with the contents of an email collection ( e.g. , historians or lawyers ) better understand the context of emails that they find , and ( 2 ) augmenting more typical social networks ( based on senders and recipients ) with additional links based on references found in unstructured text .

Most approaches to resolving identity can be decomposed into four sub-problems : ( 1 ) finding a reference that requires resolution , ( 2 ) identifying candidates , ( 3 ) assembling evidence , and ( 4 ) choosing among the candidates based on the evidence .

For the work reported in this paper , we rely on the user to designate references requiring resolution ( which we model as a predetermined set of mention-queries for which the correct referent is known ) .

Candidate identification is a computational expedient that permits the evidence assembly effort to be efficiently focused ; we use only simple techniques for that task .

Our principal contributions are the approaches we take to evidence generation ( leveraging three ways of linking to other emails where evidence might be found : reply chains , social interaction , and topical similarity ) and our approach to choosing among candidates ( based on a generative model of reference production ) .

We evaluate the effectiveness of our approach on four collections , three of which have previously reported results for comparison , and one that is considerably larger than the others .

The remainder of this paper is as follows .

Section 3 then describes our approach to modeling identity and ranking candidates .

Section 4 presents results , and Section 5 concludes .

We have presented an approach to mention resolution in email that flexibly makes use of expanding contexts to accurately resolve the identity of a given mention .

Our approach focuses on four naturally occurring contexts in email , including a message , a thread , other emails with senders and/or recipients in common , and other emails with significant topical content in common .

Our approach outperforms previously reported techniques and it scales well to larger collections .

Moreover , our results serve to highlight the importance of social context when resolving mentions in social media , which is an idea that deserves more attention generally .

In future work , we plan to extend our test collection with mention queries that must be resolved in the " long tail " of the identity distribution where less evidence is available .

We are also interested in exploring iterative approaches to jointly resolving mentions .


Recently , neural network based sentence modeling methods have achieved great progress .

Among these methods , the recursive neural networks ( RecNNs ) can effectively model the combination of the words in sentence .

However , RecNNs need a given external topological structure , like syntactic tree .

In this paper , we propose a gated recursive neural network ( GRNN ) to model sentences , which employs a full binary tree ( FBT ) structure to control the combinations in recursive structure .

By introducing two kinds of gates , our model can better model the complicated combinations of features .

Experiments on three text classification datasets show the effectiveness of our model .

Recently , neural network based sentence modeling approaches have been increasingly focused on for their ability to minimize the efforts in feature engineering , such as Neural Bag-of-Words ( NBoW ) , Recurrent Neural Network ( RNN ) ( Mikolov et al . , 2010 ) , Recursive Neural Network ( RecNN ) ( Pollack , 1990 ; Socher et al . , 2013b ; Socher et al . , 2012 ) and Convolutional Neural Network ( CNN ) ( Kalchbrenner et al . , 2014 ; Hu et al . , 2014 ) .

Among these methods , recursive neural networks ( RecNNs ) have shown their excellent abilities to model the word combinations in sentence .

However , RecNNs require a pre-defined topological structure , like parse tree , to encode sentence , which limits the scope of its application .

Cho et al . ( 2014 ) proposed the gated recursive convolutional neural network ( grConv ) by utilizing the directed acyclic graph ( DAG ) structure instead of parse tree *

Left is a GRNN using a directed acyclic graph ( DAG ) structure .

Right is a GRNN using a full binary tree ( FBT ) structure .

( The green nodes , gray nodes and white nodes illustrate the positive , negative and neutral sentiments respectively . )

However , DAG structure is relatively complicated .

The number of the hidden neurons quadraticly increases with the length of sentences so that grConv can not effectively deal with long sentences .

Inspired by grConv , we propose a gated recursive neural network ( GRNN ) for sentence modeling .

Different with grConv , we use the full binary tree ( FBT ) as the topological structure to recursively model the word combinations , as shown in Figure 1 .

The number of the hidden neurons linearly increases with the length of sentences .

Another difference is that we introduce two kinds of gates , reset and update gates ( Chung et al . , 2014 ) , to control the combinations in recursive structure .

With these two gating mechanisms , our model can better model the complicated combinations of features and capture the long dependency interactions .

In our previous works , we have investigated several different topological structures ( tree and directed acyclic graph ) to recursively model the semantic composition from the bottom layer to the top layer , and applied them on Chinese word segmentation ( Chen et al . , 2015a ) and dependency parsing ( Chen et al . , 2015b ) tasks .

However , these structures are not suitable for modeling sentences .

i P(•|x i ; θ ) w 2 w 1 ( i ) ( i ) … … … … … … … … w 3 w 4 w 5 ( i ) ( i ) ( i ) 0 0 0

Figure 2 : Architecture of Gated Recursive Neural Network ( GRNN ) .

In this paper , we adopt the full binary tree as the topological structure to reduce the model complexity .

Experiments on the Stanford Sentiment Treebank dataset ( Socher et al . , 2013b ) and the TREC questions dataset ( Li and Roth , 2002 ) show the effectiveness of our approach .

In this paper , we propose a gated recursive neural network ( GRNN ) to recursively summarize the meaning of sentence .

GRNN uses full binary tree as the recursive topological structure instead of an external syntactic tree .

In addition , we introduce two kinds of gates to model the complicated combinations of features .

In future work , we would like to investigate the other gating mechanisms for better modeling the feature combinations .


We explore a hybrid approach for Chinese definitional question answering by combining deep linguistic analysis with surface pattern learning .

We answer four questions in this study : 1 ) How helpful are linguistic analysis and pattern learning ?

2 ) What kind of questions can be answered by pattern matching ?

3 ) How much annotation is required for a pattern-based system to achieve good performance ?

4 ) What linguistic features are most useful ?

Extensive experiments are conducted on biographical questions and other definitional questions .

Major findings include : 1 ) linguistic analysis and pattern learning are complementary ; both are required to make a good definitional QA system ; 2 ) pattern matching is very effective in answering biographical questions while less effective for other definitional questions ; 3 ) only a small amount of annotation is required for a pattern learning system to achieve good performance on biographical questions ; 4 ) the most useful linguistic features are copulas and appositives ; relations also play an important role ; only some propositions convey vital facts .

Due to the ever increasing large amounts of online textual data , learning from textual data is becoming more and more important .

Traditional document retrieval systems return a set of relevant documents and leave the users to locate the specific information they are interested in .

Question answering , which combines traditional document retrieval and information extraction , solves this problem directly by returning users the specific answers .

Research in textual question answering has made substantial advances in the past few years ( Voorhees , 2004 ) .

Most question answering research has been focusing on factoid questions where the goal is to return a list of facts about a concept .

Definitional questions , however , remain largely unexplored .

Definitional questions differ from factoid questions in that the goal is to return the relevant " answer nuggets " of information about a query .

Identifying such answer nuggets requires more advanced language processing techniques .

Definitional QA systems are not only interesting as a research challenge .

They also have the potential to be a valuable complement to static knowledge sources like encyclopedias .

This is because they create definitions dynamically , and thus answer definitional questions about terms which are new or emerging ( Blair-Goldensoha et al . , 2004 ) .

One success in factoid question answering is pattern based systems , either manually constructed ( Soubbotin and Soubbotin , 2002 ) or machine learned ( Cui et al . , 2004 ) .

However , it is unknown whether such pure pattern based systems work well on definitional questions where answers are more diverse .

Deep linguistic analysis has been found useful in factoid question answering ( Moldovan et al . , 2002 ) and has been used for definitional questions ( Xu et al . , 2004 ; Harabagiu et al . , 2003 ) .

Linguistic analy-sis is useful because full parsing captures long distance dependencies between the answers and the query terms , and provides more information for inference .

However , merely linguistic analysis may not be enough .

First , current state of the art linguistic analysis such as parsing , co-reference , and relation extraction is still far below human performance .

Errors made in this stage will propagate and lower system accuracy .

Second , answers to some types of definitional questions may have strong local dependencies that can be better captured by surface patterns .

Thus we believe that combining linguistic analysis and pattern learning would be complementary and be beneficial to the whole system .

Work in combining linguistic analysis with patterns include Weischedel et al . ( 2004 ) and Jijkoun et al . ( 2004 ) where manually constructed patterns are used to augment linguistic features .

However , manual pattern construction critically depends on the domain knowledge of the pattern designer and often has low coverage ( Jijkoun et al . , 2004 ) .

Automatic pattern derivation is more appealing ( Ravichandran and Hovy , 2002 ) .

In this work , we explore a hybrid approach to combining deep linguistic analysis with automatic pattern learning .

We are interested in answering the following four questions for Chinese definitional question answering : How helpful are linguistic analysis and pattern learning in definitional question answering ?

If pattern learning is useful , what kind of question can pattern matching answer ?

How much human annotation is required for a pattern based system to achieve reasonable performance ?

We have explored a hybrid approach for definitional question answering by combining deep linguistic analysis and surface pattern learning .

For the first time , we have answered four questions regarding Chinese definitional QA : deep linguistic analysis and automatic pattern learning are complementary and may be combined ; patterns are powerful in answering biographical questions ; only a small amount of annotation ( 2 days ) is required to obtain good performance in a biographical QA system ; copulas and appositions are the most useful linguistic features ; relation extraction also helps .

Answering " What-is " questions is more challenging than answering " Who-is " questions .

To improve the performance on " What-is " questions , we could divide " What-is " questions into finer classes such as organization , location , disease , and general substance , and process them specifically .

Our current pattern matching is based on simple POS tagging which captures only limited syntactic information .

We generalize words to their corresponding POS tags .

Another possible improvement is to generalize using automatically derived word clusters , which provide semantic information .


This work concerns automatic topic segmentation of email conversations .

We present a corpus of email threads manually annotated with topics , and evaluate annotator reliability .

To our knowledge , this is the first such email corpus .

We show how the existing topic segmentation models ( i.e. , Lexical Chain Segmenter ( LCSeg ) and Latent Dirichlet Allocation ( LDA ) ) which are solely based on lexical information , can be applied to emails .

By pointing out where these methods fail and what any desired model should consider , we propose two novel extensions of the models that not only use lexical information but also exploit finer level conversation structure in a principled way .

Empirical evaluation shows that LCSeg is a better model than LDA for segmenting an email thread into topical clusters and incorporating conversation structure into these models improves the performance significantly .

With the ever increasing popularity of emails and web technologies , it is very common for people to discuss issues , events , agendas or tasks by email .

Effective processing of the email contents can be of great strategic value .

In this paper , we study the problem of topic segmentation for emails , i.e. , grouping the sentences of an email thread into a set of coherent topical clusters .

Adapting the standard definition of topic ( Galley et al . , 2003 ) to conversations / emails , we consider a topic is something about which the participant(s ) discuss or argue or express their opinions .

For example , in the email thread shown in Figure 1 , according to the majority of our annotators , participants discuss three topics ( e.g. , ' telecon cancellation ' , ' TAG document ' , and ' responding to I18N ' ) .

Multiple topics seem to occur naturally in social interactions , whether synchronous ( e.g. , chats , meetings ) or asynchronous ( e.g. , emails , blogs ) conversations .

In multi-party chat ( Elsner and Charniak , 2008 ) report an average of 2.75 discussions active at a time .

In our email corpus , we found an average of 2.5 topics per thread .

Topic segmentation is often considered a prerequisite for other higher-level conversation analysis and applications of the extracted structure are broad , encompassing : summarization ( Harabagiu and Lacatusu , 2005 ) , information extraction and ordering ( Allan , 2002 ) , information retrieval ( Dias et al . , 2007 ) , and intelligent user interfaces ( Dredze et al . , 2008 ) .

While extensive research has been conducted in topic segmentation for monologues ( e.g. , ( Malioutov and Barzilay , 2006 ) , ( Choi et al . , 2001 ) ) and synchronous dialogs ( e.g. , ( Galley et al . , 2003 ) , ( Hsueh et al . , 2006 ) ) , none has studied the problem of segmenting asynchronous multi-party conversations ( e.g. , email ) .

Therefore , there is no reliable annotation scheme , no standard corpus , and no agreedupon metrics available .

Also , it is our key hypothesis that , because of its asynchronous nature , and the use of quotation ( Crystal , 2001 ) , topics in an email thread often do not change in a sequential way .

As a result , we do not expect models which have proved successful in monologue or dialog to be as effective when they are applied to email conversations .

Our contributions in this paper aim to remedy these problems .

First , we present an email corpus annotated with topics and evaluate annotator agreement .

Second , we adopt a set of metrics to measure the local and global structural similarity between two annotations from the work on multi-party chat disentanglement ( Elsner and Charniak , 2008 ) .

Third , we show how the two state-of-the-art topic segmentation methods ( i.e. , LCSeg and LDA ) which are solely based on lexical information and make strong assumptions on the resulting topic models , can be effectively applied to emails , by having them to consider , in a principled way , a finer level structure of the underlying conversations .

Experimental results show that both LCSeg and LDA benefit when they are extended to consider the conversational structure .

When comparing the two methods , we found that LCSeg is better than LDA and this advantage is preserved when they are extended to incorporate conversational structure .

In this paper we presented an email corpus annotated for topic segmentation .

We extended LDA and LC-Seg models by incorporating the fragment quotation graph , a fine-grain model of the conversation , which is based on the analysis of quotations .

Empirical evaluation shows that the fragment quotation graph helps both these models to perform significantly better than their basic versions , with LCSeg+FQG being the best performer .


Scientific literature analysis needs fine-grained named entity recognition ( NER ) to provide a wide range of information for scientific discovery .

For example , chemistry research needs to study dozens to hundreds of distinct , fine-grained entity types , making consistent and accurate annotation difficult even for crowds of domain experts .

On the other hand , domain-specific ontologies and knowledge bases ( KBs ) can be easily accessed , constructed , or integrated , which makes distant supervision realistic for fine-grained chemistry NER .

In distant supervision , training labels are generated by matching mentions in a document with the concepts in the knowledge bases ( KBs ) .

However , this kind of KB-matching suffers from two major challenges : incomplete annotation and noisy annotation .

We propose CHEMNER , an ontologyguided , distantly-supervised method for finegrained chemistry NER to tackle these challenges .

It leverages the chemistry type ontology structure to generate distant labels with novel methods of flexible KB-matching and ontology-guided multi-type disambiguation .

It significantly improves the distant label generation for the subsequent sequence labeling model training .

We also provide an expertlabeled , chemistry NER dataset with 62 finegrained chemistry types ( e.g. , chemical compounds and chemical reactions ) .

Experimental results show that CHEMNER is highly effective , outperforming substantially the stateof-the-art NER methods ( with .25 absolute F1 score improvement ) .

Named entity recognition ( NER ) is a fundamental step in scientific literature analysis to build AI-driven systems for molecular discovery , synthetic strategy designing , and manufacturing ( Xie et al . , 2013 ; Szklarczyk et al . , 2015 ; Huang et al . , 2015 ; Szklarczyk et al . , 2017 ; de Almeida et al . , 2019 ) .

It aims to locate and classify entity mentions ( e.g. , " Suzuki-Miyaura cross-coupling reactions " ) from unstructured text into pre-defined categories ( e.g. , " coupling reactions " ) .

In the chemistry domain , previous NER studies are mostly focused on one coarse-grained entity type ( i.e. , chemicals ) ( Krallinger et al . , 2015 ; He et al . , 2020 ; Watanabe et al . , 2019 ) and rely on large amounts of manuallyannotated data for training deep learning models ( Chiu and Nichols , 2016 ; Ma and Hovy , 2016 ; Lample et al . , 2016 ; Wang et al . , 2019b ; Devlin et al . , 2019 ; Liu et al . , 2019 ) .

In real-world applications , it is important to recognize chemistry entities on diverse and finegrained types ( e.g. , " inorganic phophorus compounds " , " coupling reactions " and " catalysts " ) to provide a wide range of information for scientific discovery .

It will need dozens to hundreds of distinct types , making consistent and accurate annotation difficult even for domain experts .

On the other hand , the domain-specific ontologies and knowledge bases ( KBs ) can be easily accessed , constructed , or integrated , which makes distant supervision realistic for fine-grained chemistry NER .

Still , challenges exist for correctly recognizing the entity boundaries and accurately typing entities with distant supervision .

In distant supervision , training labels are generated by matching the mentions in a document with the concepts in the knowledge bases ( KBs ) .

( 1 ) incomplete annotation where a mention in a document can be matched only partially or missed completely due to an incomplete coverage of the KBs ( Figure 1a ) , and ( 2 ) noisy annotation where a mention can be erroneously matched due to the potential matching of multiple entity types in the KBs ( Figure 1b ) .

Due to the complex name structures ( e.g. , nested naming structures and long chemical formulas ) of chemical entities , these challenges lead to severe low-precision and low-recall for finegrained chemistry NER with distant supervision .

Several studies have attempted to address the incomplete annotation problem in distantlysupervised NER .

For example , AutoNER ( Shang et al . , 2018b ) introduces an " unknown " type that can be skipped during training to reduce the effect of false negative labeling with distant supervision .

BOND ( Liang et al . , 2020 ) leverages the power of pre-trained language models and a self-training approach to iteratively incorporate more training labels and improve the NER performance .

However , previous methods assume a high precision and reasonable coverage of KB-matching for distant label generation .

For example , the KB-matching on the CoNLL03 dataset ( Liang et al . , 2020 ) reported over 80 % on precision and over 60 % on recall .

These methods do not work well with fine-grained chemistry NER that has severe low precision and low recall with KB-matching .

Previous studies also largely ignore the noisy annotation problem by simply discarding those multi-labels during the KBmatching process ( Liang et al . , 2020 ) .

However , the noisy labels can not be simply ignored for the chemistry entities because they consist of a large portion of distant training labels .

We observe that more than 60 % of the entities have multiple labels during KB-matching in the chemistry domain .

We propose CHEMNER , an ontology-guided , distantly-supervised NER method for fine-grained chemistry NER .

Taking an input corpus , a chemistry type ontology and associated entity dictionaries collected from the KBs , we develop a novel flexible KB-matching method with TF-IDF-based majority voting to resolve the incomplete annota-tion problem .

Then we develop a novel ontologyguided multi-type disambiguation method to resolve the noisy annotation problem .

Taking the output from the above two steps as distant supervision , we further train a sequence labeling model to cover additional entities .

CHEMNER significantly improves the distant label generation for the subsequent NER model training .

We also provide an expert-labeled , chemistry NER dataset with 62 finegrained chemistry types ( e.g. , chemical compounds and chemical reactions ) .

Experimental results show that CHEMNER is highly effective , achieving substantially better performance ( with .25 absolute F1 score improvement ) compared with the state-ofthe-art NER methods .

We have released our data and code to benefit future studies1 .

We propose CHEMNER , an ontology-guided , distantly-supervised method for fine-grained chemistry NER .

We also provide an expert labeled , chemistry NER dataset with 62 finegrained chemistry types ( e.g. , chemical compounds and chemical reactions ) .

Experimental results show that CHEMNER is highly effective , outperforming substantially the state-of-the-art NER methods on fine-grained chemistry NER .

Although achieving great performance , there is still large room for improvement of CHEMNER .

In the future , we plan to further refine and enrich the type ontology and incorporate more information in the dictionaries ( e.g. , chemical structures in the KBs ) for a better NER performance .

We also plan to apply our finegrained NER method to other scientific domains .


Constrained decoding is of great importance not only for speed but also for translation quality .

Previous efforts explore soft syntactic constraints which are based on constituent boundaries deduced from parse trees of the source language .

We present a new framework to establish soft constraints based on a more natural alternative : translation boundary rather than constituent boundary .

We propose simple classifiers to learn translation boundaries for any source sentences .

The classifiers are trained directly on word-aligned corpus without using any additional resources .

We report the accuracy of our translation boundary classifiers .

We show that using constraints based on translation boundaries predicted by our classifiers achieves significant improvements over the baseline on large-scale Chinese-to-English translation experiments .

The new constraints also significantly outperform constituent boundary based syntactic constrains .

It has been known that phrase-based decoding ( phrase segmentation / translation / reordering ( Chiang , 2005 ) ) should be constrained to some extent not only for transferring the NP-hard problem ( Knight , 1999 ) into a tractable one in practice but also for improving translation quality .

For example , Xiong et al . ( 2008 ) find that translation quality can be significantly improved by either prohibiting reorderings around punctuation or restricting reorderings within a 15-word window .

Recently , more linguistically motivated constraints are introduced to improve phrase-based decoding .

( Cherry , 2008 ) and ( Marton and Resnik , 2008 ) introduce syntactic constraints into the standard phrase-based decoding ( Koehn et al . , 2003 ) and hierarchical phrase-based decoding ( Chiang , 2005 ) respectively by using a counting feature which accumulates whenever hypotheses violate syntactic boundaries of source-side parse trees .

( Xiong et al . , 2009 ) further presents a bracketing model to include thousands of context-sensitive syntactic constraints .

All of these approaches achieve their improvements by guiding the phrase-based decoder to prefer translations which respect source-side parse trees .

One major problem with such constituent boundary based constraints is that syntactic structures of the source language do not necessarily reflect translation structures where the source and target language correspond to each other .

In this paper , we investigate building classifiers that directly address the problem of translation boundary , rather than extracting constituent boundary from sourceside parsers built for a different purpose .

A translation boundary is a position in the source sequence which begins or ends a translation zone1 spanning multiple source words .

In a translation zone , the source phrase is translated as a unit .

Reorderings which cross translation zones are not desirable .

Inspired by ( Roark and Hollingshead , 2008 ) which introduces classifiers to decide if a word can begin / end a multi-word constituent , we build two discriminative classifiers to tag each word in the source sequence with a binary class label .

The first classifier decides if a word can begin a multi-sourceword translation zone ; the second classifier decides if a word can end a multi-source-word translation zone .

Given a partial translation covering source sequence ( i , j ) with start word c i and end word c j2 , this translation can be penalized if the first classifier decides that the start word c i can not be a beginning translation boundary or the second classifier decides that the end word c j can not be an ending translation boundary .

In such a way , we can guide the decoder to boost hypotheses that respect translation boundaries and therefore the common translation structure shared by the source and target language , rather than the syntactic structure of the source language .

We report the accuracy of such classifiers by comparing their outputs with " gold " translation boundaries obtained from reference translations on the development set .

We integrate translation boundary based constraints into phrase-based decoding and display that they improve translation quality significantly in large-scale experiments .

Furthermore , we confirm that they also significantly outperform constituent boundary based syntactic constraints .

In this paper , we have presented a simple approach to learn translation boundaries on source sentences .

The learned translation boundaries are used to constrain phrase-based decoding in a soft manner .

The whole approach has several properties .

• First , it is based on a simple classification task that can achieve considerably high accuracy when taking translation divergences into account using simple models and features .

• Second , the classifier output can be straightforwardly used to constrain phrase-based decoder .

• Finally , we have empirically shown that , to build soft constraints for phrase-based decoding , translation boundary predicted by our classifier is a better choice than constituent boundary deduced from source-side parse tree .

Future work in this direction will involve trying different methods to define more informative translation boundaries , such as a boundary to begin / end a swapping .

We would also like to investigate new methods to incorporate automatically learned translation boundaries more efficiently into decoding in an attempt to further improve search in both speed and accuracy .


Information Extraction ( IE ) systems that extract role fillers for events typically look at the local context surrounding a phrase when deciding whether to extract it .

Often , however , role fillers occur in clauses that are not directly linked to an event word .

We present a new model for event extraction that jointly considers both the local context around a phrase along with the wider sentential context in a probabilistic framework .

Our approach uses a sentential event recognizer and a plausible role-filler recognizer that is conditioned on event sentences .

We evaluate our system on two IE data sets and show that our model performs well in comparison to existing IE systems that rely on local phrasal context .

Information Extraction ( IE ) systems typically use extraction patterns ( e.g. , Soderland et al . ( 1995 ) , Riloff ( 1996 ) , Yangarber et al . ( 2000 ) , Califf and Mooney ( 2003 ) ) or classifiers ( e.g. , Freitag ( 1998 ) , Freitag and McCallum ( 2000 ) , Chieu et al . ( 2003 ) , Bunescu and Mooney ( 2004 ) ) to extract role fillers for events .

Most IE systems consider only the immediate context surrounding a phrase when deciding whether to extract it .

For tasks such as named entity recognition , immediate context is usually sufficient .

But for more complex tasks , such as event extraction , a larger field of view is often needed to understand how facts tie together .

Most IE systems are designed to identify role fillers that appear as arguments to event verbs or nouns , either explicitly via syntactic relations or implicitly via proximity ( e.g. , John murdered Tom or the murder of Tom by John ) .

But many facts are presented in clauses that do not contain event words , requiring discourse relations or deep structural analysis to associate the facts with event roles .

For example , consider the sentences below :

and 30 were injured in India after terrorists launched an attack on the Taj Hotel .

in Mexico City and its surrounding suburbs in a Swine Flu outbreak .

after a tractor-trailer collided with a bus in Arkansas .

in Baghdad last night in a resurgence of bomb attacks in the capital city .

and $ 50 million in damage was caused by a hurricane that hit Miami on Friday .

to make way for modern , safer bridges that will be constructed early next year .

These examples illustrate a common phenomenon in text where information is not explicitly stated as filling an event role , but readers have no trouble making this inference .

The role fillers above ( seven people , two bridges ) occur as arguments to verbs that reveal state information ( death , destruction ) but are not event-specific ( i.e. , death and destruction can result from a wide variety of incident types ) .

IE systems often fail to extract these role fillers because these systems do not recognize the immediate context as being relevant to the specific type of event that they are looking for .

We propose a new model for information extraction that incorporates both phrasal and sentential evidence in a unified framework .

Our unified probabilistic model , called GLACIER , consists of two components : a model for sentential event recognition and a model for recognizing plausible role fillers .

The Sentential Event Recognizer offers a probabilistic assessment of whether a sentence is discussing a domain-relevant event .

The Plausible Role-Filler Recognizer is then conditioned to identify phrases as role fillers based upon the assumption that the surrounding context is discussing a relevant event .

This unified probabilistic model allows the two components to jointly make decisions based upon both the local evidence surrounding each phrase and the " peripheral vision " afforded by the sentential event recognizer .

Section 2 positions our research with respect to related work .

Section 3 presents our unified probabilistic model for information extraction .

Section 4 shows experimental results on two IE data sets , and Section 5 discusses directions for future work .

We presented a unified model for IE that balances the influence of sentential context with local contextual evidence to improve the performance of event-based IE .

Our experimental results showed that using sentential contexts indeed produced better results on two IE data sets .

Our current model uses supervised learning , so one direction for future work is to adapt the model for weakly supervised learning .

We also plan to incorporate discourse features and investigate even wider contexts to capture broader discourse effects .


This work introduces AM-FM , a semantic framework for machine translation evaluation .

Based upon this framework , a new evaluation metric , which is able to operate without the need for reference translations , is implemented and evaluated .

The metric is based on the concepts of adequacy and fluency , which are independently assessed by using a cross-language latent semantic indexing approach and an n-gram based language model approach , respectively .

Comparative analyses with conventional evaluation metrics are conducted on two different evaluation tasks ( overall quality assessment and comparative ranking ) over a large collection of human evaluations involving five European languages .

Finally , the main pros and cons of the proposed framework are discussed along with future research directions .

Evaluation has always been one of the major issues in Machine Translation research , as both human and automatic evaluation methods exhibit very important limitations .

On the one hand , although highly reliable , in addition to being expensive and time consuming , human evaluation suffers from inconsistency problems due to inter-and intraannotator agreement issues .

On the other hand , while being consistent , fast and cheap , automatic evaluation has the major disadvantage of requiring reference translations .

This makes automatic evaluation not reliable in the sense that good translations not matching the available references are evaluated as poor or bad translations .

The main objective of this work is to propose and evaluate AM-FM , a semantic framework for assessing translation quality without the need for reference translations .

The proposed framework is theoretically grounded on the classical concepts of adequacy and fluency , and it is designed to account for these two components of translation quality in an independent manner .

First , a cross-language latent semantic indexing model is used for assessing the adequacy component by directly comparing the output translation with the input sentence it was generated from .

Second , an n-gram based language model of the target language is used for assessing the fluency component .

Both components of the metric are evaluated at the sentence level , providing the means for defining and implementing a sentence-based evaluation metric .

Finally , the two components are combined into a single measure by implementing a weighted harmonic mean , for which the weighting factor can be adjusted for optimizing the metric performance .

The rest of the paper is organized as follows .

Section 2 , presents some background work and the specific dataset that has been used in the experimental work .

Section 3 , provides details on the proposed AM-FM framework and the specific metric implementation .

Section 4 presents the results of the conducted comparative evaluations .

Finally , section 5 presents the main conclusions and relevant issues to be dealt with in future research .

This work presented AM-FM , a semantic framework for translation quality assessment .

Two comparative evaluations with standard metrics have been conducted over a large collection of humangenerated scores involving different languages .

Although the obtained performance is below standard metrics , the proposed method has the main advantage of not requiring reference translations .

Notice that a monolingual version of AM-FM is also possible by using monolingual latent semantic indexing ( Landauer et al . , 1998 ) along with a set of reference translations .

A detailed evaluation of a monolingual implementation of AM-FM can be found in Banchs and Li ( 2011 ) .

As future research , we plan to study the impact of different dataset sizes and vector space model parameters for improving the performance of the AM component of the metric .

This will include the study of learning curves based on the amount of training data used , and the evaluation of different vector model construction strategies , such as removing stop-words and considering bigrams and word categories in addition to individual words .

Finally , we also plan to study alternative uses of AM-FM within the context of statistical machine translation as , for example , a metric for MERT optimization , or using the AM component alone as an additional feature for decoding , rescoring and/or confidence estimation .


Video Question Answering is a task which requires an AI agent to answer questions grounded in video .

This task entails three key challenges : ( 1 ) understand the intention of various questions , ( 2 ) capturing various elements of the input video ( e.g. , object , action , causality ) , and ( 3 ) cross-modal grounding between language and vision information .

We propose Motion-Appearance Synergistic Networks ( MASN ) , which embed two crossmodal features grounded on motion and appearance information and selectively utilize them depending on the question 's intentions .

MASN consists of a motion module , an appearance module , and a motion-appearance fusion module .

The motion module computes the action-oriented cross-modal joint representations , while the appearance module focuses on the appearance aspect of the input video .

Finally , the motion-appearance fusion module takes each output of the motion module and the appearance module as input , and performs question-guided fusion .

As a result , MASN achieves new state-of-the-art performance on the TGIF-QA and MSVD-QA datasets .

We also conduct qualitative analysis by visualizing the inference results of MASN .

The code is available at https://github.com/ ahjeongseo / MASN-pytorch .

Recently , research in natural language processing and computer vision has made significant progress in artificial intelligence ( AI ) .

Thanks to this , visionlanguage tasks such as image captioning ( Xu et al . , 2015 ) , visual question answering ( VQA ) ( Antol et al . , 2015 ; Goyal et al . , 2017 ) , and visual commonsense reasoning ( VCR ) ( Zellers et al . , 2019 ) have been introduced to the research community , along with some benchmark datasets .

In particular , video question answering ( video QA ) tasks ( Xu et al . , 2016 ; Jang et al . , 2017 ; Lei et al . , 2018 ;

Yu et al . , 2019 ; Choi et al . , 2020 ) have been proposed with the goal of reasoning over higher-level visionlanguage interactions .

In contrast to QA tasks based on static images , the questions presented in the video QA dataset vary from frame-level questions regarding the appearance of objects ( e.g. , what is the color of the hat ? )

to questions regarding action and causality ( e.g. , what does the man do after opening a door ? ) .

There are three crucial challenges in video QA : ( 1 ) understand the intention of various questions , ( 2 ) capturing various elements of the input video ( e.g. , object , action , and causality ) , and ( 3 ) crossmodal grounding between language and vision information .

To tackle these challenges , previous studies ( Li et al . , 2019 ; Jiang et al . , 2020 ; Huang et al . , 2020 ) have mainly explored this task by jointly embedding the features from the pre-trained word embedding model ( Pennington et al . , 2014 ) and the object detection models ( He et al . , 2016 ; Ren et al . , 2016 ) .

However , as discussed in ( Gao et al . , 2018 ) , the use of the visual features extracted from the object detection models suffers from motion analysis since the object detection model lacks temporal modeling .

To enforce the motion analysis , a few approaches ( Xu et al . , 2017 ; Gao et al . , 2018 ) have employed additional visual features ( Tran et al . , 2015 ) ( i.e. , motion features ) which were widely used in the action recognition domain , but their reasoning capability is still limited .

They typically employed recurrent models ( e.g. , LSTM ) to embed a long sequence of the visual features .

Due to the problem of long-term dependency in recurrent models ( Bengio et al . , 1993 ) , their proposed methods may fail to learn dependencies between distant features .

In this paper , we propose Motion-Appearance The output from the fusion module is used to derive answers .

For question features , the word-level representation F Q is integrated with the visual features in the VQ interaction submodule .

The last hidden units q from the bi-LSTM are used to combine appearance and motion features .

Synergistic Networks ( MASN ) for video question answering which consist of three kinds of modules : the motion module , the appearance module , and the motion-appearance fusion module .

As shown in Figure 1 , the motion module and the appearance module aim to embed rich cross-modal representations .

These two modules have the same architecture except that the motion module takes the motion features extracted from I3D as visual features and the appearance module utilizes the appearance features extracted from ResNet .

Each of these modules first constructs the object graphs via graph convolutional networks ( GCN ) to compute the relationships among objects in each visual feature .

Then , the vision-question interaction module performs cross-modal grounding between the output of the GCNs and the question features .

The motion module and the appearance module each yield cross-modal representations of the motion and the appearance aspects of the input video respectively .

The motion-appearance fusion module finally integrates these two features based on the question features .

The main contributions of our paper are as follows .

First , we propose Motion-Appearance Synergistic Networks ( MASN ) for video question answering based on three modules , the motion module , the appearance module , and the motionappearance fusion module .

Second , we validate MASN on the large-scale video question answering datasets TGIF-QA , MSVD-QA , and MSRVTT-QA .

MASN achieves the new state-of-the-art performance on TGIF-QA and MSVD-QA .

We perform ablation studies to validate the effectiveness of our proposed methods .

Finally , we conduct a qualitative analysis of MASN by visualizing inference results .

In this paper , we proposed a Motion-Appearance Synergistic Networks to fuse and create a synergy between motion and appearance features .

Through the Motion and Appearance modules , MASN manages to find motion and appearance clues to solve the question , while modulating the amount of information used of each type through the Fusion module .

Experimental results on three benchmark datasets show the effectiveness of our proposed MASN architecture compared to other models .


This paper describes a novel approach to the semantic relation detection problem .

Instead of relying only on the training instances for a new relation , we leverage the knowledge learned from previously trained relation detectors .

Specifically , we detect a new semantic relation by projecting the new relation 's training instances onto a lower dimension topic space constructed from existing relation detectors through a three step process .

First , we construct a large relation repository of more than 7,000 relations from Wikipedia .

Second , we construct a set of non-redundant relation topics defined at multiple scales from the relation repository to characterize the existing relations .

Similar to the topics defined over words , each relation topic is an interpretable multinomial distribution over the existing relations .

Third , we integrate the relation topics in a kernel function , and use it together with SVM to construct detectors for new relations .

The experimental results on Wikipedia and ACE data have confirmed that backgroundknowledge-based topics generated from the Wikipedia relation repository can significantly improve the performance over the state-of-theart relation detection approaches .

Detecting semantic relations in text is very useful in both information retrieval and question answering because it enables knowledge bases to be leveraged to score passages and retrieve candidate answers .

To extract semantic relations from text , three types of approaches have been applied .

Rule-based methods ( Miller et al . , 2000 ) employ a number of linguistic rules to capture relation patterns .

Featurebased methods ( Kambhatla , 2004 ; Zhao and Grishman , 2005 ) transform relation instances into a large amount of linguistic features like lexical , syntactic and semantic features , and capture the similarity between these feature vectors .

Recent results mainly rely on kernel-based approaches .

Many of them focus on using tree kernels to learn parse tree structure related features ( Collins and Duffy , 2001 ; Culotta and Sorensen , 2004 ; Bunescu and Mooney , 2005 ) .

Other researchers study how different approaches can be combined to improve the extraction performance .

For example , by combining tree kernels and convolution string kernels , ( Zhang et al . , 2006 ) achieved the state of the art performance on ACE ( ACE , 2004 ) , which is a benchmark dataset for relation extraction .

Although a large set of relations have been identified , adapting the knowledge extracted from these relations for new semantic relations is still a challenging task .

Most of the work on domain adaptation of relation detection has focused on how to create detectors from ground up with as little training data as possible through techniques such as bootstrapping ( Etzioni et al . , 2005 ) .

We take a different approach , focusing on how the knowledge extracted from the existing relations can be reused to help build detectors for new relations .

We believe by reusing knowledge one can build a more cost effective relation detector , but there are several challenges associated with reusing knowledge .

The first challenge to address in this approach is how to construct a relation repository that has suffi-cient coverage .

In this paper , we introduce a method that automatically extracts the knowledge characterizing more than 7,000 relations from Wikipedia .

Wikipedia is comprehensive , containing a diverse body of content with significant depth and grows rapidly .

Wikipedia 's infoboxes are particularly interesting for relation extraction .

They are short , manually-created , and often have a relational summary of an article : a set of attribute / value pairs describing the article 's subject .

Another challenge is how to deal with overlap of relations in the repository .

For example , Wikipedia authors may make up a name when a new relation is needed without checking if a similar relation has already been created .

We refine the relation repository based on an unsupervised multiscale analysis of the correlations between existing relations .

This method is parameter free , and able to produce a set of non-redundant relation topics defined at multiple scales .

Similar to the topics defined over words ( Blei et al . , 2003 ) , we define relation topics as multinomial distributions over the existing relations .

The relation topics extracted in our approach are interpretable , orthonormal to each other , and can be used as basis relations to re-represent the new relation instances .

The third challenge is how to use the relation topics for a relation detector .

We map relation instances in the new domains to the relation topic space , resulting in a set of new features characterizing the relationship between the relation instances and existing relations .

By doing so , background knowledge from the existing relations can be introduced into the new relations , which overcomes the limitations of the existing approaches when the training data is not sufficient .

Our work fits in to a class of relation extraction research based on " distant supervision " , which studies how knowledge and resources external to the target domain can be used to improve relation extraction .

( Mintz et al . , 2009 ; Jiang , 2009 ; Chan and Roth , 2010 ) .

One distinction between our approach and other existing approaches is that we represent the knowledge from distant supervision using automatically constructed topics .

When we test on new instances , we do not need to search against the knowledge base .

In addition , our topics also model the indirect relationship between relations .

Such information can not be directly found from the knowledge base .

The contributions of this paper are three-fold .

Firstly , we extract a large amount of training data for more than 7,000 semantic relations from Wikipedia ( Wikipedia , 2011 ) and DBpedia ( Auer et al . , 2007 ) .

A key part of this step is how we handle noisy data with little human effort .

Secondly , we present an unsupervised way to construct a set of relation topics at multiple scales .

This step is parameter free , and results in a nonredundant , multiscale relation topic space .

Thirdly , we design a new kernel for relation detection by integrating the relation topics into the relation detector construction .

The experimental results on Wikipedia and ACE data ( ACE , 2004 ) have confirmed that background-knowledge-based features generated from the Wikipedia relation repository can significantly improve the performance over the state-of-the-art relation detection approaches .

This paper proposes a novel approach to create detectors for new relations integrating the knowledge extracted from the existing relations .

Firstly , we pro- vide an automatic way to collect training data for more than 7,000 relations from Wikipedia and DBpedia .

Different from the topics defined over words , relation topics are defined over the existing relations .

Thirdly , we design a new kernel for relation detection by integrating the relation topics in the representation of the relation instances .

By leveraging the knowledge extracted from the Wikipedia relation repository , our approach significantly improves the performance over the state-of-the-art approaches on ACE data .

This paper makes use of all DBpedia relations to create relation topics .

It is possible that using a subset of them ( more related to the target relations ) might improve the performance .