This work investigates design choices in modeling a discourse scheme for improving opinion polarity classification .

For this , two diverse global inference paradigms are used : a supervised collective classification framework and an unsupervised optimization framework .

Both approaches perform substantially better than baseline approaches , establishing the efficacy of the methods and the underlying discourse scheme .

We also present quantitative and qualitative analyses showing how the improvements are achieved .

The importance of discourse in opinion analysis is being increasingly recognized ( Polanyi and Zaenen , 2006 ) .

Motivated by the need to enable discourse-based opinion analysis , previous research ( Asher et al . , 2008 ; Somasundaran et al . , 2008 ) developed discourse schemes and created manually annotated corpora .

However , it was not known whether and how well these linguistic ideas and schemes can be translated into effective computational implementations .

In this paper , we first investigate ways in which an opinion discourse scheme can be computationally modeled , and then how it can be utilized to improve polarity classification .

Specifically , the discourse scheme we use is from Somasundaran et al . ( 2008 ) , which was developed to support a global , interdependent polarity interpretation .

To achieve discourse-based global inference , we explore two different frameworks .

The first is a supervised framework that learns interdependent opinion interpretations from training data .

The second is an unsupervised optimization framework which uses constraints to express the ideas of coherent opinion interpretation embodied in the scheme .

For the supervised framework , we use Iterative Collective Classification ( ICA ) , which facilitates machine learning using relational information .

The unsupervised optimization is implemented as an Integer Linear Programming ( ILP ) problem .

Via our implementations , we aim to empirically test if discourse-based approaches to opinion analysis are useful .

Our results show that both of our implementations achieve significantly better accuracies in polarity classification than classifiers using local information alone .

This confirms the hypothesis that the discourse-based scheme is useful , and also shows that both of our design choices are effective .

We also find that there is a difference in the way ICA and ILP achieve improvements , and a simple hybrid approach , which incorporates the strengths of both , is able to achieve significant overall improvements over both .

Our analyses show that even when our discourse-based methods bootstrap from noisy classifications , they can achieve good improvements .

The rest of this paper is organized as follows : we discuss related work in Section 2 and the discourse scheme in Section 3 .

We present our discourse-based implementations in Section 4 , experiments in Section 5 , discussions in Section 6 and conclusions in Section 7 .

This work focuses on the first step to ascertain whether discourse relations are useful for improving opinion polarity classification , whether they can be modeled and what modeling choices can be used .

To this end , we explored two distinct paradigms : the supervised ICA and the unsupervised ILP .

We showed that both of our approaches are effective in exploiting discourse relations to significantly improve polarity classification .

We found that there is a difference in how ICA and ILP achieve improvements , and that combining the two in a hybrid approach can lead to further overall improvement .

Quantitatively , we showed that our approach is able to achieve a large increase in recall of the polar categories without harming the precision , which results in the performance improvements .

Qualitatively , we illustrated how , even if the bootstrapping process is noisy , the optimization and discourse constraints effectively rectify the misclassifications .

The improvements of our diverse global inference approaches indicate that discourse information can be adapted in different ways to augment and improve existing opinion analysis techniques .

The automation of the discourse-relation recognition is the next step in this research .

The behavior of ICA and ILP can change , depending on the automation of discourse level recognition .

The implementation and comparison of the two methods under full automation is the focus of our future work .

They investigate the influences of different discourse schemes on opinion polarity classification performance when coupled with supervised and unsupervised inference to achieve discourse-based global inference.

opinion polarity classification
unsupervised
discourse


Compositional embedding models build a representation ( or embedding ) for a linguistic structure based on its component word embeddings .

We propose a Feature-rich Compositional Embedding Model ( FCM ) for relation extraction that is expressive , generalizes to new domains , and is easy-to-implement .

The key idea is to combine both ( unlexicalized ) handcrafted features with learned word embeddings .

The model is able to directly tackle the difficulties met by traditional compositional embeddings models , such as handling arbitrary types of sentence annotations and utilizing global information for composition .

We test the proposed model on two relation extraction tasks , and demonstrate that our model outperforms both previous compositional models and traditional feature rich models on the ACE 2005 relation extraction task , and the SemEval 2010 relation classification task .

The combination of our model and a loglinear classifier with hand-crafted features gives state-of-the-art results .

We made our implementation available for general use 1 .

Two common NLP feature types are lexical properties of words and unlexicalized linguistic / structural interactions between words .

Prior work on relation extraction has extensively studied how to design such features by combining discrete lexical properties ( e.g. the identity of a word , ⇤ ⇤ Gormley and Yu contributed equally .

1 https://github.com / mgormley / pacaya its lemma , its morphological features ) with aspects of a word 's linguistic context ( e.g. whether it lies between two entities or on a dependency path between them ) .

While these help learning , they make generalization to unseen words difficult .

An alternative approach to capturing lexical information relies on continuous word embeddings2 as representative of words but generalizable to new words .

Embedding features have improved many tasks , including NER , chunking , dependency parsing , semantic role labeling , and relation extraction ( Miller et al . , 2004 ; Turian et al . , 2010 ; Koo et al . , 2008 ; Roth and Woodsend , 2014 ; Sun et al . , 2011 ; Plank and Moschitti , 2013 ; Nguyen and Grishman , 2014 ) .

Embeddings can capture lexical information , but alone they are insufficient : in state-of-the-art systems , they are used alongside features of the broader linguistic context .

In this paper , we introduce a compositional model that combines unlexicalized linguistic context and word embeddings for relation extraction , a task in which contextual feature construction plays a major role in generalizing to unseen data .

Our model allows for the composition of embeddings with arbitrary linguistic structure , as expressed by hand crafted features .

In the following sections , we begin with a precise construction of compositional embeddings using word embeddings in conjunction with unlexicalized features .

Various feature sets used in prior work ( Turian et al . , 2010 ; Nguyen and Grishman , 2014 ; Hermann et al . , 2014 ; Roth and Woodsend , 2014 )

A feature that depends on the embedding for this context word could generalize to other lexical indicators of the same relation ( e.g. " operating " ) that do n't appear with ART during training .

But lexical information alone is insufficient ; relation extraction requires the identification of lexical roles : where a word appears structurally in the sentence .

In ( 2 ) , the word " of " between " suburbs " and " Baghdad " suggests that the first entity is part of the second , yet the earlier occurrence after " direction " is of no significance to the relation .

Even finer information can be expressed by a word 's role on the dependency path between entities .

In ( 3 ) we can distinguish the word " died " from other irrelevant words that do n't appear between the entities .

tured as special cases of this construction .

Adding these compositional embeddings directly to a standard log-linear model yields a special case of our full model .

We then treat the word embeddings as parameters giving rise to our powerful , efficient , and easy-to-implement log-bilinear model .

The model capitalizes on arbitrary types of linguistic annotations by better utilizing features associated with substructures of those annotations , including global information .

We choose features to promote different properties and to distinguish different functions of the input words .

First , it decomposes the annotated sentence into substructures ( i.e. a word and associated annotations ) .

Second , it extracts features for each substructure ( word ) , and combines them with the word 's embedding to form a substructure embedding .

Third , we sum over substructure embeddings to form a composed annotated sentence embedding , which is used by a final softmax layer to predict the output label ( relation ) .

The result is a state-of-the-art relation extractor for unseen domains from ACE 2005 ( Walker et al . , 2006 ) and the relation classification dataset from SemEval-2010 Task 8 ( Hendrickx et al . , 2010 ) .

Contributions This paper makes several contributions , including :

We introduce the Yu and Dredze ( 2015 ) and Yu et al . ( 2015 ) .

Additionally , we have extended FCM to incorporate a low-rank embedding of the features ( Yu et al . , 2015 ) , which focuses on fine-grained relation extraction for ACE and ERE .

This paper obtains better results than the low-rank extension on ACE coarse-grained relation extraction .

We have presented FCM , a new compositional model for deriving sentence-level and substructure embeddings from word embeddings .

Compared to existing compositional models , FCM can easily handle arbitrary types of input and handle global information for composition , while remaining easy to implement .

We have demonstrated that FCM alone attains near state-of-the-art performances on several relation extraction tasks , and in combination with traditional feature based loglinear models it obtains state-of-the-art results .

Our next steps in improving FCM focus on enhancements based on task-specific embeddings or loss functions as in Hashimoto et al . ( 2015 ; dos Santos et al . ( 2015 ) .

Moreover , as the model provides a general idea for representing both sentences and sub-structures in language , it has the potential to contribute useful components to various tasks , such as dependency parsing , SRL and paraphrasing .

Also as kindly pointed out by one anonymous reviewer , our FCM can be applied to the TAC-KBP ( Ji et al . , 2010 ) tasks , by replacing the training objective to a multi-instance multilabel one ( e.g. Surdeanu et al . ( 2012 ) ) .

We plan to explore the above applications of FCM in the future .

They propose a compositional embedding model which combines hand-crafted features with word embeddings by summing over substructure embeddings to form a composed annotated sentence embedding.


Chinese short text matching usually employs word sequences rather than character sequences to get better performance .

However , Chinese word segmentation can be erroneous , ambiguous or inconsistent , which consequently hurts the final matching performance .

To address this problem , we propose neural graph matching networks , a novel sentence matching framework capable of dealing with multi-granular input information .

Instead of a character sequence or a single word sequence , paired word lattices formed from multiple word segmentation hypotheses are used as input and the model learns a graph representation according to an attentive graph matching mechanism .

Experiments on two Chinese datasets show that our models outperform the state-of-the-art short text matching models .

Short text matching ( STM ) is a fundamental task of natural language processing ( NLP ) .

It is usually recognized as a paraphrase identification task or a sentence semantic matching task .

Given a pair of sentences , a matching model is to predict their semantic similarity .

It is widely used in question answer systems and dialogue systems ( Gao et al . , 2019 ; Yu et al . , 2014 ) .

The recent years have seen advances in deep learning methods for text matching ( Mueller and Thyagarajan , 2016 ; Gong et al . , 2017 ; Chen et al . , 2017 ; Lan and Xu , 2018 ) .

However , almost all of these models are initially proposed for English text matching .

Applying them for Chinese text matching , we have two choices .

One is to take Chinese characters as the input of models .

Another is first to segment each sentence into words , and then to take these words as input tokens .

Although character-based models can overcome the * Kai Yu is the corresponding author .

problem of data sparsity to some degree ( Li et al . , 2019 ) , the main drawback of these models is that explicit word information is not fully exploited , which can be potentially useful for semantic matching .

However , word-based models often suffer some potential issues caused by word segmentation .

As shown in Figure 1 , the character sequence " 南 京 市 长 江 大 桥(South

Capital City Long River Big Bridge ) " has two different meanings with different word segmentation .

The first one refers to a bridge ( Segment-1 , Segment-2 ) , and the other refers to a person ( Segment-3 ) .

The ambiguity may be eliminated with more context .

Additionally , the segmentation granularity of different tools is different .

For example , " 长江大 桥(Yangtze River Bridge ) " in Segment-1 is divided into two words " 长江(Yangtze River ) " and " 大 桥(Bridge ) " in Segment-2 .

It has been shown that multi-granularity information is important for text matching ( Lai et al . , 2019 ) .

Here we propose a neural graph matching method ( GMN ) for Chinese short text matching .

Instead of segmenting each sentence into a word sequence , we keep all possible segmentation paths to form a word lattice graph , as shown in Figure 1 .

GMN takes a pair of word lattice graphs as input and updates the representations of nodes according to the graph matching attention mechanism .

Also , GMN can be combined with pre-trained language models , e.g. BERT ( Devlin et al . , 2019 ) .

It can be regarded as a method to integrate word information in these pre-trained language models during the fine-tuning phase .

The experiments on two Chinese Datasets show that our model outperforms not only previous state-of-the-art models but also the pre-trained model BERT as well as some variants of BERT .

In this paper , we propose a neural graph matching model for Chinese short text matching .

It takes a pair of word lattices as input instead of word or character sequences .

The utilization of word lattice can provide more multi-granularity information and avoid the error propagation issue of word segmentation .

Additionally , our model and the pre-training model are complementary .

It can be regarded as a flexible method to introduce word information into BERT during the fine-tuning phase .

The experimental results show that our model outperforms the state-of-the-art text matching models as well as some BERT-based models .

They propose neural graph matching networks inspired which can take multiple word segmentation hypotheses into account using a word lattice graph.


Chinese Pinyin input method is very important for Chinese language information processing .

Users may make errors when they are typing in Chinese words .

In this paper , we are concerned with the reasons that cause the errors .

Inspired by the observation that pressing backspace is one of the most common user behaviors to modify the errors , we collect 54 , 309 , 334 error-correction pairs from a realworld data set that contains 2 , 277 , 786 users via backspace operations .

In addition , we present a comparative analysis of the data to achieve a better understanding of users ' input behaviors .

Comparisons with English typos suggest that some language-specific properties result in a part of Chinese input errors .

Unlike western languages , Chinese is unique due to its logographic writing system .

Chinese users can not directly type in Chinese words using a QW-ERTY keyboard .

Pinyin is the official system to transcribe Chinese characters into the Latin alphabet .

Based on this transcription system , Pinyin input methods have been proposed to assist users to type in Chinese words ( Chen , 1997 ) .

The typical way to type in Chinese words is in a sequential manner ( Wang et al . , 2001 ) .

Assume users want to type in the Chinese word " 什 么(what ) " .

First , they mentally generate and type in corresponding Pinyin " shenme " .

Then , a Chinese Pinyin input method displays a list of Chinese words which share that Pinyin , as shown in Fig . 1 .

Users visually search the target word from candidates and select numeric key " 1 " to get the result .

The last two steps do not exist in typing process of English words , which indicates that it is more complicated for Chinese users to type in Chinese words .

Chinese users may make errors when they are typing in Chinese words .

As shown in Fig . 2 , a user may mistype " shenme " as " shenem " .

Typical Chinese Pinyin input method can not return the right word .

Users may not realize that an error occurs and select the first candidate word " 什恶魔 " ( a meaningless word ) as the result .

This greatly limits user experience since users have to identify errors and modify them , or can not get the right word .

In this paper , we analyze the reasons that cause errors in Chinese Pinyin input method .

This analysis is helpful in enhancing the user experience and the performance of Chinese Pinyin input method .

In practice , users press backspace on the keyboard to modify the errors , they delete the mistyped word and re-type in the correct word .

Motivated by this ob-servation , we can extract error-correction pairs from backspace operations .

These error-correction pairs are of great importance in Chinese spelling correction task which generally relies on sets of confusing words .

We extract 54 , 309 , 334 error-correction pairs from user input behaviors and further study them .

Our comparative analysis of Chinese and English typos suggests that some language-specific properties of Chinese lead to a part of input errors .

To the best of our knowledge , this paper is the first one which analyzes user input behaviors in Chinese Pinyin input method .

The rest of this paper is organized as follows .

Section 3 introduces how we collect errors in Chinese Pinyin input method .

In Section 4 , we investigate the reasons that result in these errors .

Section 5 concludes the whole paper and discusses future work .

In this paper , we study user input behaviors in Chinese Pinyin input method from backspace operations .

We aim at analyzing the reasons that cause these errors .

Users signal that they are very likely to make errors if they press backspace on the keyboard .

Then they modify the errors and type in the correct words they want .

Different from the previous research , we extract abundant Pinyin-correction and Chinese word-correction pairs from backspace operations .

Compared with English typos , we observe some language-specific properties in Chinese have impact on errors .

All in all , user behaviors ( Zheng et al . , 2009 ; Zheng et al . , 2010 ; Zheng et al . , 2011b ) in Chinese Pinyin input method provide novel perspectives for natural language processing tasks .

Below we sketch three possible directions for the future work : ( 1 ) we should consider position features in analyzing Pinyin errors .

For example , it is less likely that users make errors in the first letter of an input Pinyin .

( 2 ) we aim at designing a selfadaptive input method that provide error-tolerant features ( Chen and Lee , 2000 ; Zheng et al . , 2011a ) .

( 3 ) we want to build a Chinese spelling correction system based on extracted error-correction pairs .

They build large Chinese error-correction pairs by exploiting the action of pressing the backspace key and perform analysis to understand users' input behaviors.

understand user input behavior
press backspace
chinese


We present an inexact search algorithm for the problem of predicting a two-layered dependency graph .

The algorithm is based on a k-best version of the standard cubictime search algorithm for projective dependency parsing , which is used as the backbone of a beam search procedure .

This allows us to handle the complex nonlocal feature dependencies occurring in bistratal parsing if we model the interdependency between the two layers .

We apply the algorithm to the syntacticsemantic dependency parsing task of the CoNLL-2008 Shared Task , and we obtain a competitive result equal to the highest published for a system that jointly learns syntactic and semantic structure .

Numerous linguistic theories assume a multistratal model of linguistic structure , such as a layer of surface syntax , deep syntax , and shallow semantics .

Examples include Meaning-Text Theory ( Mel'čuk , 1988 ) , Discontinuous Grammar ( Buch-Kromann , 2006 ) , Extensible Dependency Grammar ( Debusmann et al . , 2004 ) , and the Functional Generative Description ( Sgall et al . , 1986 ) which forms the theoretical foundation of the Prague Dependency Treebank ( Hajič , 1998 ) .

In the statistical NLP community , the most widely used grammatical resource is the Penn Treebank ( Marcus et al . , 1993 ) .

This is a purely syntactic resource , but we can also include this treebank in the category of multistratal resources since the PropBank ( Palmer et al . , 2005 ) and NomBank ( Meyers et al . , 2004 ) projects have annotated shallow semantic structures on top of it .

Dependency-converted versions of the Penn Treebank , PropBank and NomBank were used in the CoNLL-2008 Shared Task ( Surdeanu et al . , 2008 ) , in which the task of the participants was to produce a bistratal dependency structure consisting of surface syntax and shallow semantics .

Producing a consistent multistratal structure is a conceptually and computationally complex task , and most previous methods have employed a purely pipeline-based decomposition of the task .

This includes the majority of work on shallow semantic analysis ( Gildea and Jurafsky , 2002 , inter alia ) .

Nevertheless , since it is obvious that syntax and semantics are highly interdependent , it has repeatedly been suggested that the problems of syntactic and semantic analysis should be carried out simultaneously rather than in a pipeline , and that modeling the interdependency between syntax and semantics would improve the quality of all the substructures .

The purpose of the CoNLL-2008 Shared Task was to study the feasibility of a joint analysis of syntax and semantics , and while most participating systems used a pipeline-based approach to the problem , there were a number of contributions that attempted to take the interdependence between syntax and semantics into account .

The top-performing system in the task ( Johansson and Nugues , 2008 ) applied a very simple reranking scheme by means of a k-best syntactic output , similar to previous attempts ( Gildea and Jurafsky , 2002 ; Toutanova et al . , 2005 ) to improve semantic role labeling performance by using mul-tiple parses .

The system by Henderson et al . ( 2008 ) extended previous stack-based algorithms for dependency parsing by using two separate stacks to build the syntactic and semantic graphs .

Lluís and Màrquez ( 2008 ) proposed a model that simultaneously predicts syntactic and semantic links , but since its search algorithm could not take the syntactic-semantic interdependencies into account , a pre-parsing step was still needed .

In addition , before the CoNLL-2008 shared task there have been a few attempts to jointly learn syntactic and semantic structure ; for instance , Merlo and Musillo ( 2008 ) appended semantic role labels to the phrase tags in a constituent treebank and applied a conventional constituent parser to predict constituent structure and semantic roles .

In this paper , we propose a new approximate search method for bistratal dependency analysis .

The search method is based on a beam search procedure that extends a k-best version of the standard cubic-time search algorithm for projective dependency parsing .

This is similar to the search method for constituent parsing used by Huang ( 2008 ) , who referred to it as cube pruning , inspired by an idea from machine translation decoding ( Chiang , 2007 ) .

The cube pruning approach , which is normally used to solve the arg max problem , was also recently extended to summing problems , which is needed in some learning algorithms ( Gimpel and Smith , 2009 ) .

We apply the algorithm on the CoNLL-2008 Shared Task data , and obtain the same evaluation score as the best previously published system that simultaneously learns syntactic and semantic structure ( Titov et al . , 2009 ) .

In this paper , we have presented a new approximate search method to solve the problem of jointly predicting the two layers in a bistratal dependency graph .

The algorithm shows competitive performance on the treebank used in the CoNLL-2008 Shared Task , a bistratal treebank consisting of a surface-syntactic and a shallow semantic layer .

In addition to the syntactic-semantic task that we have described in this paper , we believe that our method can be used in other types of multistratal syntactic frameworks , such as a representation of surface and deep syntax as in Meaning-Text Theory ( Mel'čuk , 1988 ) .

The optimization problem that we set out to solve is intractable , but we have shown that reasonable performance can be achieved with an inexact , beam search-based search method .

This is not obvious : it has previously been shown that using an inexact search procedure when the learning algorithm assumes that the search is exact may lead to slow convergence or even divergence ( Kulesza and Pereira , 2008 ) , but this does not seem to be a problem in our case .

While we used a beam search method as the method of approximation , other methods are certainly possible .

An interesting example is the recent system by Smith and Eisner ( 2008 ) , which used loopy belief propagation in a dependency parser using highly complex features , while still maintaining cubic-time search complexity .

An obvious drawback of our approach compared to traditional pipeline-based semantic role labeling methods is that the speed of the algorithm is highly dependent on the size of the interdependency feature representation Φ i .

Also , extracting these features is fairly complex , and it is of critical importance to implement the feature extraction procedure efficiently since it is one of the bottlenecks of the algorithm .

It is plausible that our performance suffers from the absence of other frequently used syntax-based features such as dependent-of-dependent and voice .

It is thus highly dubious that a joint modeling of syntactic and semantic structure is worth the additional implementational effort .

So far , no system using tightly integrated syntactic and semantic processing has been competitive with the best systems , which have been either completely pipelinebased ( Che et al . , 2008 ; Ciaramita et al . , 2008 ) or employed only a loose syntactic-semantic coupling ( Johansson and Nugues , 2008 ) .

It has been conjectured that modeling the semantics of the sentence would also help in syntactic disambiguation ; however , it is likely that this is already implicitly taken into account by the lexical features present in virtually all modern parsers .

In addition , a problem that our beam search method has in common with the constituent parsing method by Huang ( 2008 ) is that highly nonlocal features must be computed late .

In our case , this means that if there is a long distance between a predicate and an argument , the secondary link between them will be unlikely to influence the final search result .

They propose an inexact search algorithm that predicts jointly the two layers in a bistratal dependency graph based on a beam search procedure.


This work describes an answer ranking engine for non-factoid questions built using a large online community-generated question-answer collection ( Yahoo ! Answers ) .

We show how such collections may be used to effectively set up large supervised learning experiments .

Furthermore we investigate a wide range of feature types , some exploiting NLP processors , and demonstrate that using them in combination leads to considerable improvements in accuracy .

The problem of Question Answering ( QA ) has received considerable attention in the past few years .

Nevertheless , most of the work has focused on the task of factoid QA , where questions match short answers , usually in the form of named or numerical entities .

Thanks to international evaluations organized by conferences such as the Text REtrieval Conference ( TREC)1 or the Cross Language Evaluation Forum ( CLEF ) Workshop2 , annotated corpora of questions and answers have become available for several languages , which has facilitated the development of robust machine learning models for the task .

The situation is different once one moves beyond the task of factoid QA .

Comparatively little research has focused on QA models for non-factoid questions such as causation , manner , or reason questions .

Because virtually no training data is available for this problem , most automated systems train either on small hand-annotated corpora built in house ( Higashinaka and Isozaki , 2008 ) or on question-answer pairs harvested from Frequently Asked Questions ( FAQ ) lists or similar resources ( Soricut and Brill , 2006 ) .

None of these situations is ideal : the cost of building the training corpus in the former setup is high ; in the latter scenario the data tends to be domain-specific , hence unsuitable for the learning of open-domain models .

On the other hand , recent years have seen an explosion of user-generated content ( or social media ) .

Of particular interest in our context are communitydriven question-answering sites , such as Yahoo !

Answers 3 , where users answer questions posed by other users and best answers are selected manually either by the asker or by all the participants in the thread .

The data generated by these sites has significant advantages over other web resources : ( a ) it has a high growth rate and it is already abundant ; ( b ) it covers a large number of topics , hence it offers a better approximation of open-domain content ; and ( c ) it is available for many languages .

Community QA sites , similar to FAQs , provide large number of questionanswer pairs .

Nevertheless , this data has a significant drawback : it has high variance of quality , i.e. , answers range from very informative to completely irrelevant or even abusive .

Table 1 shows some examples of both high and low quality content .

In this paper we address the problem of answer ranking for non-factoid questions from social media content .

Our research objectives focus on answering the following two questions :

1 . Is it possible to learn an answer ranking model for complex questions from such noisy data ?

This is an interesting question because a positive answer indicates that a plethora of training data is readily available to QA researchers and system developers .

2 . Which features are most useful in this scenario ?

Are similarity models as effective as models that learn question-to-answer transformations ?

Does syntactic and semantic information help ?

For generality , we focus only on textual features extracted from the answer text and we ignore all meta data information that is not generally available .

Notice that we concentrate on one component of a possible social-media QA system .

In addition to answer ranking , a complete system would have to search for similar questions already answered ( Jeon et al . , 2005 ) , and rank content quality using " social " features such as the authority of users ( Jeon et al . , 2006 ; Agichtein et al . , 2008 ) .

This is not the focus of our work : here we investigate the problem of learning an answer ranking model capable of dealing with complex questions , using a large number of , possible noisy , question-answer pairs .

By focusing exclusively on textual content we increase the portability of our approach to other collections where " social " features might not available , e.g. , Web search .

We describe our approach , including all the features explored for answer modeling , in Section 2 .

We introduce the corpus used in our empirical analysis in Section 3 .

We detail our experiments and analyze the results in Section 4 .

We overview related work in Section 5 and conclude the paper in Section 6 .

In this work we described an answer ranking engine for non-factoid questions built using a large community-generated question-answer collection .

On one hand , this study shows that we can effectively exploit large amounts of available Web data to do research on NLP for non-factoid QA systems , without any annotation or evaluation cost .

This provides an excellent framework for large-scale experimentation with various models that otherwise might be hard to understand or evaluate .

On the other hand , we expect the outcome of this process to help several applications , such as open-domain QA on the Web and retrieval from social media .

For example , on the Web our ranking system could be combined with a passage retrieval system to form a QA system for complex questions .

On social media , our system should be combined with a component that searches for similar questions already answered ; this output can possibly be filtered further by a content-quality module that explores " social " features such as the authority of users , etc .

We show that the best ranking performance is obtained when several strategies are combined into a single model .

We obtain the best results when similarity models are aggregated with features that model question-to-answer transformations , frequency and density of content , and correlation of QA pairs with external collections .

While the features that model question-to-answer transformations provide most benefits , we show that the combination is crucial for improvement .

Lastly , we show that syntactic dependency parsing and coarse semantic disambiguation yield a small , yet statistically significant performance increase on top of the traditional bag-of-words and n-gram representation .

We obtain these results using only off-the-shelf NLP processors that were not adapted in any way for our task .

They propose an answer ranking engine for non-factoid questions using a large online community-generated question-answer collection which covers a large number of domains and languages.


Disfluencies in spontaneous speech are known to be associated with prosodic disruptions .

However , most algorithms for disfluency detection use only word transcripts .

Integrating prosodic cues has proved difficult because of the many sources of variability affecting the acoustic correlates .

This paper introduces a new approach to extracting acoustic-prosodic cues using text-based distributional prediction of acoustic cues to derive vector z-score features ( innovations ) .

We explore both early and late fusion techniques for integrating text and prosody , showing gains over a high-accuracy text-only model .

Speech disfluencies are frequent events in spontaneous speech .

The rate of disfluencies varies with the speaker and context ; one study observed disfluencies once in every 20 words , affecting up to one third of utterances ( Shriberg , 1994 ) .

Disfluencies are important to account for , both because of the challenge that the disrupted grammatical flow poses for natural language processing of spoken transcripts and because of the information that they provide about the speaker .

Most work on disfluency detection builds on the framework that annotates a disfluency in terms of a reparandum followed by an interruption point ( + ) , an optional interregnum ( { } ) , and then the repair , if any .

A few simple examples are given below : Based on the similarity / differences between the reparandum and the repair , disfluencies are often categorized into three types : repetition ( the first example ) , rephrase ( the next example ) , and restart ( the last example ) .

The interruption point is associated with a disruption in the realization of a prosodic phrase , which could involve cutting words off or elongation associated with hesitation , followed by a prosodic reset at the start of the repair .

There may also be emphasis in the repair to highlight the correction .

Researchers have been working on automatic disfluency detection for many years ( Lickley , 1994 ; Shriberg et al . , 1997 ; Charniak and Johnson , 2001 ; Johnson and Charniak , 2004 ; Lease et al . , 2006 ;

Qian and Liu , 2013 ; Zayats et al . , 2016 ) , motivated in part by early work on parsing speech that assumed reliable detection of the interruption point ( Nakatani and Hirschberg , 1994 ; Shriberg and Stolcke , 1997 ; Liu et al . , 2006 ) .

The first efforts to integrate prosody with word cues for disfluency detection ( Baron et al . , 2002 ; Snover et al . , 2004 ) found gains from using prosody , but word cues played the primary role .

In subsequent work ( Qian and Liu , 2013 ; Honnibal and Johnson , 2014 ; Wang et al . , 2017 ) , more effective models of word transcripts have been the main source of performance gains .

The success of recent neural network systems raises the question of what the role is for prosody in future work .

In the next section , we hypothesize where prosody might help and look at the relative frequency of these cases and the performance of a high accuracy disfluency detection algorithm in these contexts .

With the premise that there is a potential for prosody to benefit disfluency detection , we then propose a new approach to extracting prosodic features .

A major challenge for all efforts to incorporate prosodic cues in spoken language understanding is the substantial variability in the acoustic correlates of prosody .

For example , duration cues are expected to be useful -disfluencies are often associated with duration lengthening related to hesitation .

However , duration varies with phonetic context , word function , prosodic phrase structure , speaking rate , etc .

To account for some of this variability , various feature normalization techniques are used , but typically these account for only limited contexts , e.g. phonetic context for duration or speaker pitch range for fundamental frequency .

In our work , we introduce a mechanism for normalization using the full sentence context .

We train a sequential neural prediction model to estimate distributions of acoustic features for each word , given the word sequence of a sentence .

Then , the actual observed acoustic feature is used to find the prediction error , normalized by the estimated variance .

We refer to the resulting features as innovations , which can be thought of as a non-linear version of the innovations in a Kalman filter .

The innovations will be large when the acoustic cues do not reflect the expected prosodic structure , such as during hesitations , disfluencies , and contrastive or emphatic stress .

The idea is to provide prosodic cues that are less redundant with the textual cues .

We assess the new prosodic features in experiments on disfluency detection using the Switchboard corpus , exploring both early and late fusion techniques to integrate innovations with text features .

Our analysis shows that prosody does help with detecting some of the more difficult types of disfluencies .

This paper has three main contributions .

First , our analysis of a high performance disfluency detection algorithm confirms hypotheses about contexts where text-only models have high error rates .

Second , we introduce a novel representation of prosodic cues , i.e. the innovation vector resulting from predicting prosodic cues given the whole sentence context .

Analyses of the innovation distributions show expected patterns of prosodic cues at interruption points .

Finally , we demonstrate improved disfluency detection performance on Switchboard by integrating prosody and textbased features in a neural network architecture , while comparing early and late fusion approaches .

In this paper , we introduce a novel approach to extracting acoustic-prosodic cues with the goal of improving disfluency detection , but also with the intention of impacting spoken language processing more generally .

Our initial analysis of a textonly disfluency detection system shows that despite high performance of such models , there exists a big gap in the performance of text-based approaches for some types of disfluencies , such as restarts and non-trivial or long rephrases .

Thus , prosody cues , which can be indicative of interruption points , have a potential to contribute towards detection of more difficult types of disfluencies .

Since the acoustic-prosodic cues carry information related to multiple phenomena , it can be difficult to isolate the cues that are relevant to specific events , such as interruption points .

In this work , we introduce a novel approach where we extract relevant acoustic-prosodic information using textbased distributional prediction of acoustic cues to derive vector z-score features , or innovations .

The innovations point to irregularities in prosody flow that are not predicted by the text , helping to better isolate signals relevant to disfluency detection that are not simply redundant with textual cues .

We explore both early and late fusion approaches to combine innovations with text-based features .

Our experiments show that innovation features are better predictors of disfluencies compared to the original acoustic cues .

Our analysis of the errors and of the innovation features point to a limitation of the current work , which is in the modeling of F0 features .

The current model obtains word-based F0 ( and energy ) features by simply averaging the values over the duration of the word , which loses any distinctions between rising and falling F0 .

By leveraging polynomial contour models , we expect to improve both intonation and energy features , which we hope will reduce some of the false detections associated with emphasis and unexpected fluent phrase boundaries .

An important next step is to test the system using ASR rather than hand transcripts .

It is possible that errors in the transcripts could hurt the residual prediction , but if prosody is used to refine the recognition hypothesis , this could actually lead to improved recognition .

Finally , we expect that the innovation model of prosody can benefit other NLP tasks , such as sarcasm and intent detection , as well as detecting paralinguist information .

They propose a sequential neural model to estimate distributions of acoustic features for each word, given entire sentences and use them to find prediction errors.


In this work , the problem of extracting phrase translation is formulated as an information retrieval process implemented with a log-linear model aiming for a balanced precision and recall .

We present a generic phrase training algorithm which is parameterized with feature functions and can be optimized jointly with the translation engine to directly maximize the end-to-end system performance .

Multiple data-driven feature functions are proposed to capture the quality and confidence of phrases and phrase pairs .

Experimental results demonstrate consistent and significant improvement over the widely used method that is based on word alignment matrix only .

Phrase has become the standard basic translation unit in Statistical Machine Translation ( SMT ) since it naturally captures context dependency and models internal word reordering .

In a phrase-based SMT system , the phrase translation table is the defining component which specifies alternative translations and their probabilities for a given source phrase .

In learning such a table from parallel corpus , two related issues need to be addressed ( either separately or jointly ): which pairs are considered valid translations and how to assign weights , such as probabilities , to them .

The first problem is referred to as phrase pair extraction , which identifies phrase pairs that are supposed to be translations of each other .

Methods have been proposed , based on syntax , that take advantage of linguistic constraints and alignment of grammatical structure , such as in Yamada and Knight ( 2001 ) and Wu ( 1995 ) .

The most widely used approach derives phrase pairs from word alignment matrix ( Och and Ney , 2003 ; Koehn et al . , 2003 ) .

Other methods do not depend on word alignments only , such as directly modeling phrase alignment in a joint generative way ( Marcu and Wong , 2002 ) , pursuing information extraction perspective ( Venugopal et al . , 2003 ) , or augmenting with modelbased phrase pair posterior ( Deng and Byrne , 2005 ) .

Using relative frequency as translation probability is a common practice to measure goodness of a phrase pair .

Since most phrases appear only a few times in training data , a phrase pair translation is also evaluated by lexical weights ( Koehn et al . , 2003 ) or term weighting ( Zhao et al . , 2004 ) as additional features to avoid overestimation .

The translation probability can also be discriminatively trained such as in Tillmann and Zhang ( 2006 ) .

The focus of this paper is the phrase pair extraction problem .

As in information retrieval , precision and recall issues need to be addressed with a right balance for building a phrase translation table .

High precision requires that identified translation candidates are accurate , while high recall wants as much valid phrase pairs as possible to be extracted , which is important and necessary for online translation that requires coverage .

In the word-alignment derived phrase extraction approach , precision can be improved by filtering out most of the entries by using a statistical significance test ( Johnson et al . , 2007 ) .

On the other hand , there are valid translation pairs in the training corpus that are not learned due to word alignment errors as shown in Deng and Byrne ( 2005 ) .

We would like to improve phrase translation accuracy and at the same time extract as many as possible valid phrase pairs that are missed due to incorrect word alignments .

One approach is to leverage underlying word alignment quality such as in Ayan and Dorr ( 2006 ) .

In this work , we present a generic discriminative phrase pair extraction framework that can integrate multiple features aiming to identify correct phrase translation candidates .

A significant deviation from most other approaches is that the framework is parameterized and can be optimized jointly with the decoder to maximize translation performance on a development set .

Within the general framework , the main work is on investigating useful metrics .

We employ features based on word alignment models and alignment matrix .

We also propose information metrics that are derived from both bilingual and monolingual perspectives .

All these features are data-driven and independent of languages .

The proposed phrase extraction framework is general to apply linguistic features such as semantic , POS tags and syntactic dependency .

In this paper , the problem of extracting phrase translation is formulated as an information retrieval process implemented with a log-linear model aiming for a balanced precision and recall .

We have presented a generic phrase translation extraction procedure which is parameterized with feature functions .

It can be optimized jointly with the translation engine to directly maximize the end-to-end translation performance .

Multiple feature functions were investigated .

Our experimental results on IWSLT Chinese-English corpus have demonstrated consistent and significant improvement over the widely used word alignment matrix based extraction method .

They propose an algorithm parameterized with feature functions which can be jointly optimized to maximize final translation quality to obtain a balanced precision and recall.


Embedding methods which enforce a partial order or lattice structure over the concept space , such as Order Embeddings ( OE ) ( Vendrov et al . , 2016 ) , are a natural way to model transitive relational data ( e.g. entailment graphs ) .

However , OE learns a deterministic knowledge base , limiting expressiveness of queries and the ability to use uncertainty for both prediction and learning ( e.g. learning from expectations ) .

Probabilistic extensions of OE ( Lai and Hockenmaier , 2017 ) have provided the ability to somewhat calibrate these denotational probabilities while retaining the consistency and inductive bias of ordered models , but lack the ability to model the negative correlations found in real-world knowledge .

In this work we show that a broad class of models that assign probability measures to OE can never capture negative correlation , which motivates our construction of a novel box lattice and accompanying probability measure to capture anticorrelation and even disjoint concepts , while still providing the benefits of probabilistic modeling , such as the ability to perform rich joint and conditional queries over arbitrary sets of concepts , and both learning from and predicting calibrated uncertainty .

We show improvements over previous approaches in modeling the Flickr and WordNet entailment graphs , and investigate the power of the model .

Structured embeddings based on regions , densities , and orderings have gained popularity in recent years for their inductive bias towards the essential asymmetries inherent in problems such as image captioning ( Vendrov et al . , 2016 ) , lexical and textual entailment ( Erk , 2009 ; Vilnis and McCallum , 2015 ; Lai and Hockenmaier , 2017 ; Athiwaratkun and Wilson , 2018 ) , and knowledge graph completion and reasoning ( He et al . , 2015 ; Nickel and Kiela , 2017 ; Li et al . , 2017 ) .

Models that easily encode asymmetry , and related properties such as transitivity ( the two components of commonplace relations such as partially ordered sets and lattices ) , have great utility in these applications , leaving less to be learned from the data than arbitrary relational models .

At their best , they resemble a hybrid between embedding models and structured prediction .

As noted by Vendrov et al . ( 2016 ) and Li et al . ( 2017 ) , while the models learn sets of embeddings , these parameters obey rich structural constraints .

The entire set can be thought of as one , sometimes provably consistent , structured prediction , such as an ontology in the form of a single directed acyclic graph .

While the structured prediction analogy applies best to Order Embeddings ( OE ) , which embeds consistent partial orders , other region-and density-based representations have been proposed for the express purpose of inducing a bias towards asymmetric relationships .

For example , the Gaussian Embedding ( GE ) model ( Vilnis and Mc-Callum , 2015 ) aims to represent the asymmetry and uncertainty in an object 's relations and attributes by means of uncertainty in the representation .

However , while the space of representations is a manifold of probability distributions , the model is not truly probabilistic in that it does not model asymmetries and relations in terms of prob-abilities , but in terms of asymmetric comparison functions such as the originally proposed KL divergence and the recently proposed thresholded divergences ( Athiwaratkun and Wilson , 2018 ) .

Probabilistic models are especially compelling for modeling ontologies , entailment graphs , and knowledge graphs .

Their desirable properties include an ability to remain consistent in the presence of noisy data , suitability towards semisupervised training using the expectations and uncertain labels present in these large-scale applications , the naturality of representing the inherent uncertainty of knowledge they store , and the ability to answer complex queries involving more than 2 variables .

Note that the final one requires a true joint probabilistic model with a tractable inference procedure , not something provided by e.g. matrix factorization .

We take the dual approach to density-based embeddings and model uncertainty about relationships and attributes as explicitly probabilistic , while basing the probability on a latent space of geometric objects that obey natural structural biases for modeling transitive , asymmetric relations .

The most similar work are the probabilistic order embeddings ( POE ) of Lai ( Lai and Hockenmaier , 2017 ) , which apply a probability measure to each order embedding 's forward cone ( the set of points greater than the embedding in each dimension ) , assigning a finite and normalized volume to the unbounded space .

However , POE suffers severe limitations as a probabilistic model , including an inability to model negative correlations between concepts , which motivates the construction of our box lattice model .

Our model represents objects , concepts , and events as high-dimensional products-of-intervals ( hyperrectangles or boxes ) , with an event 's unary probability coming from the box volume and joint probabilities coming from overlaps .

This contrasts with POE 's approach of defining events as the forward cones of vectors , extending to infinity , integrated under a probability measure that assigns them finite volume .

One desirable property of a structured representation for ordered data , originally noted in ( Vendrov et al . , 2016 ) is a " slackness " shared by OE , POE , and our model : when the model predicts an " edge " or lack thereof ( i.e. P ( a|b )

= 0 or 1 , or a zero constraint violation in the case of OE ) , being exposed to that fact again will not update the model .

Moreover , there are large degrees of freedom in parameter space that exhibit this slackness , giving the model the ability to embed complex structure with 0 loss when compared to models based on symmetric inner products or distances between embeddings , e.g. bilinear GLMs ( Collins et al . , 2002 ) , Trans-E ( Bordes et al . , 2013 ) , and other embedding models which must always be pushing and pulling parameters towards and away from each other .

Our experiments demonstrate the power of our approach to probabilistic ordering-biased relational modeling .

First , we investigate an instructive 2-dimensional toy dataset that both demonstrates the way the model self organizes its box event space , and enables sensible answers to queries involving arbitrary numbers of variables , despite being trained on only pairwise data .

We achieve a new state of the art in denotational probability modeling on the Flickr entailment dataset ( Lai and Hockenmaier , 2017 ) , and a matching state-of-the-art on WordNet hypernymy ( Vendrov et al . , 2016 ; Miller , 1995 ) with the concurrent work on thresholded Gaussian embedding of Athiwaratkun and Wilson ( 2018 ) , achieving our best results by training on additional co-occurrence expectations aggregated from leaf types .

We find that the strong empirical performance of probabilistic ordering models , and our box lattice model in particular , and their endowment of new forms of training and querying , make them a promising avenue for future research in representing structured knowledge .

We have only scratched the surface of possible applications .

An exciting direction is the incorporation of multi-relational data for general knowledge representation and inference .

Secondly , more complex representations , such as 2n-dimensional products of 2-dimensional convex polyhedra , would offer greater flexibility in tiling event space .

Improved inference of the latent boxes , either through better optimization or through Bayesian approaches is another natural extension .

Our greatest interest is in the application of this powerful new tool to the many areas where other structured embeddings have shown promise .

They first show that the existing probabilistic approach cannot capture negative correlation, and further propose a box lattice coupled with a probability measure.


State-of-the-art statistical machine translation ( MT ) systems have made significant progress towards producing user-acceptable translation output .

However , there is still no efficient way for MT systems to inform users which words are likely translated correctly and how confident it is about the whole sentence .

We propose a novel framework to predict wordlevel and sentence-level MT errors with a large number of novel features .

Experimental results show that the MT error prediction accuracy is increased from 69.1 to 72.2 in F-score .

The Pearson correlation between the proposed confidence measure and the human-targeted translation edit rate ( HTER ) is 0.6 .

Improvements between 0.4 and 0.9 TER reduction are obtained with the n-best list reranking task using the proposed confidence measure .

Also , we present a visualization prototype of MT errors at the word and sentence levels with the objective to improve post-editor productivity .

State-of-the-art Machine Translation ( MT ) systems are making progress to generate more usable translation outputs .

In particular , statistical machine translation systems ( Koehn et al . , 2007 ; Bach et al . , 2007 ; Shen et al . , 2008 ) have advanced to a state that the translation quality for certain language pairs ( e.g. Spanish-English , French-English , Iraqi-English ) in certain domains ( e.g. broadcasting news , force-protection , travel ) is acceptable to users .

However , a remaining open question is how to predict confidence scores for machine translated words and sentences .

An MT system typically returns the best translation candidate from its search space , but still has no reliable way to inform users which word is likely to be correctly translated and how confident it is about the whole sentence .

Such information is vital to realize the utility of machine translation in many areas .

For example , a post-editor would like to quickly identify which sentences might be incorrectly translated and in need of correction .

Other areas , such as cross-lingual question-answering , information extraction and retrieval , can also benefit from the confidence scores of MT output .

Finally , even MT systems can leverage such information to do n-best list reranking , discriminative phrase table and rule filtering , and constraint decoding ( Hildebrand and Vogel , 2008 ) .

Numerous attempts have been made to tackle the confidence estimation problem .

The work of Blatz et al . ( 2004 ) is perhaps the best known study of sentence and word level features and their impact on translation error prediction .

Along this line of research , improvements can be obtained by incorporating more features as shown in ( Quirk , 2004 ; Sanchis et al . , 2007 ; Raybaud et al . , 2009 ; Specia et al . , 2009 ) .

Soricut and Echihabi ( 2010 ) developed regression models which are used to predict the expected BLEU score of a given translation hypothesis .

Improvement also can be obtained by using target part-of-speech and null dependency link in a MaxEnt classifier ( Xiong et al . , 2010 ) .

Ueffing and Ney ( 2007 ) introduced word posterior probabilities ( WPP ) features and applied them in the n-best list reranking .

From the usability point of view , back-translation is a tool to help users to assess the accuracy level of MT output ( Bach et al . , 2007 ) .

Literally , it translates backward the MT output into the source language to see whether the output of backward translation matches the original source sentence .

However , previous studies had a few shortcomings .

First , source-side features were not extensively investigated .

Blatz et al.(2004 ) only investigated source ngram frequency statistics and source language model features , while other work mainly focused on target side features .

Second , previous work attempted to incorporate more features but faced scalability issues , i.e. , to train many features we need many training examples and to train discriminatively we need to search through all possible translations of each training example .

Another issue of previous work was that they are all trained with BLEU / TER score computing against the translation references which is different from predicting the human-targeted translation edit rate ( HTER ) which is crucial in post-editing applications ( Snover et al . , 2006 ; Papineni et al . , 2002 ) .

Finally , the backtranslation approach faces a serious issue when forward and backward translation models are symmetric .

In this case , back-translation will not be very informative to indicate forward translation quality .

In this paper , we predict error types of each word in the MT output with a confidence score , extend it to the sentence level , then apply it to n-best list reranking task to improve MT quality , and finally design a visualization prototype .

We try to answer the following questions :

• Can we use a rich feature set such as sourceside information , alignment context , and dependency structures to improve error prediction performance ?

• Can we predict more translation error types i.e substitution , insertion , deletion and shift ?

• How good do our prediction methods correlate with human correction ?

• Do confidence measures help the MT system to select a better translation ?

• How confidence score can be presented to improve end-user perception ?

In Section 2 , we describe the models and training method for the classifier .

We describe novel features including source-side , alignment context , and dependency structures in Section 3 .

Experimental results and analysis are reported in Section 4 .

Section 5 and 6 present applications of confidence scores .

In this paper we proposed a method to predict confidence scores for machine translated words and sentences based on a feature-rich classifier using linguistic and context features .

Our major contributions are three novel feature sets including source side information , alignment context , and dependency structures .

Experimental results show that by combining the source side information , alignment context , and dependency structure features with word posterior probability and target POS context ( Ueffing & Ney 2007 ; Xiong et al . , 2010 ) , the MT error prediction accuracy is increased from 69.1 to 72.2 in F-score .

Our framework is able to predict error types namely insertion , substitution and shift .

The Pearson correlation with human judgement increases from 0.52 to 0.6 .

Furthermore , we show that the proposed confidence scores can help the MT system to select better translations and as a result improvements between 0.4 and 0.9 TER reduction are obtained .

Finally , we demonstrate a prototype to visualize translation errors .

This work can be expanded in several directions .

First , we plan to apply confidence estimation to perform a second-pass constraint decoding .

After the first pass decoding , our confidence estimation model can label which word is likely to be correctly translated .

The second-pass decoding utilizes the confidence informa-tion to constrain the search space and hopefully can find a better hypothesis than in the first pass .

This idea is very similar to the multi-pass decoding strategy employed by speech recognition engines .

Moreover , we also intend to perform a user study on our visualization prototype to see if it increases the productivity of post-editors .

They propose a framework to predict word and sentence-level errors with linguistic and context features and apply it to n-best list reranking for better translations.


Figurative and metaphorical language are commonplace in discourse , and figurative expressions play an important role in communication and cognition .

However , figurative language has been a relatively under-studied area in NLP , and it remains an open question to what extent modern language models can interpret nonliteral phrases .

To address this question , we introduce Fig-QA , a Winograd-style nonliteral language understanding task consisting of correctly interpreting paired figurative phrases with divergent meanings .

We evaluate the performance of several state-of-the-art language models on this task , and find that although language models achieve performance significantly over chance , they still fall short of human performance , particularly in zero-or few-shot settings .

This suggests that further work is needed to improve the nonliteral reasoning capabilities of language models .

All our words are but crumbs that fall down from the feast of the mind ( Gibran , 1926 ) .

When humans read such a metaphorical phrase , how do they interpret it ?

Conceptual metaphors structure our everyday language and are used to map everyday physical experiences and emotions onto abstract concepts ( Lakoff and Johnson , 1981 ) .

They allow us to communicate complex ideas , to emphasize emotions , and to make humorous statements ( Fussell and Moss , 2008 ) .

However , despite relating words in a way that differs from their accepted definition , these phrases are readily interpreted by human listeners , and are common in discourse ( Shutova , 2011 ) , occurring on average every three sentences ( Mio and Katz , 1996 ; Fussell and Moss , 2008 )

The ability to interpret figurative language has been viewed as a bottleneck in natural language un-derstanding , but it has not been studied as widely as literal language ( Shutova , 2011 ; Tong et al . , 2021 ) .

Figurative language often relies on shared commonsense or cultural knowledge , and in some cases may be difficult to solve using language statistics .

This presents a challenge to language models ( LMs ) , as strong LMs trained only on text may not be able to make sense of the physical world , nor the social or cultural knowledge that language is grounded in ( Bender and Koller , 2020 ; Bisk et al . , 2020 ) .

Most previous work on figurative language focuses on metaphor detection , where a model is trained to identify the existence of metaphors in text ( Tsvetkov et al . , 2014 ; Stowe and Palmer , 2018 ; Leong et al . , 2020 ) , with datasets consisting mostly of conventionalized metaphors and idioms in wide use .

However , identifying these common metaphors that already appear often in language may be an easy task for LMs , and not fully test their ability to interpret figurative language .

The little work that exists on metaphor interpretation frames it as a task linking metaphorical phrases to literal rewordings , either through paraphrase detection ( Bizzoni and Lappin , 2018 ) or paraphrase generation ( Shutova , 2010 ; Su et al . , 2017 ; Mao et al . , 2018 ) ( details in § 7 ) Another line of work probes for metaphorical understanding in LMs , but this is similar to the metaphor detection task , in that the LM is not actually asked to choose an interpretation for the metaphor ( Pedinotti et al . , 2021 ; Aghazadeh et al . , 2022 ) .

While interesting , this work does not take into account the fact that metaphors are rich with different implications that may vary depending on the context .

In this work , we ask whether or not LMs can correctly make inferences regarding creative , relatively novel metaphors generated by humans .

This task is harder for two reasons : ( 1 ) inference is harder than identification or paraphrasing , as it requires understanding the underlying semantics , and ( 2 ) the metaphors in our dataset are novel cre-ations , and many may not appear even once in the LMs ' training data .

We propose a minimal task inspired by the Winograd schema ( Levesque et al . , 2012 ) , where LMs are tasked with choosing the entailed phrase from two opposite metaphorical phrases .

An example of a paired sentence is " Her commitment is as sturdy as ( plywood / oak ) " .

The correct answer would be either " She was ( committed / uncommitted ) " .

This can also be seen as an entailment task , where input x is the premise , and the output y is the hypothesis .

2We crowdsource a benchmark Fig-QA , consisting of 10,256 such metaphors and implications ( § 2 ) , which can be used to evaluate the nonliteral reasoning abilities of LMs or for more broad studies of figurative language in general ( we provide preliminary analyses in § 3 ) .

Through extensive experiments over strong pre-trained LMs ( § 4 ) , we find that although they can be fine-tuned to do reasonably well , their few-shot performance falls significantly short of human performance ( § 5 ) .

An in-depth analysis ( § 6 ) uncovers several insights :

( 1 ) LMs do not make use of the metaphorical context well , instead relying on the predicted probability of interpretations alone , ( 2 ) the task of associating a metaphor with an interpretation is more difficult than the reverse , ( 3 ) even strong models such as GPT-3 make inexplicable errors that are not well-aligned with human ones , indicating that further work is needed to properly model nonliteral language .

We present a Winograd-like benchmark task to test the ability of LMs to reason about figurative language , based on large-scale collection of creative metaphors written by humans .

We find a large gap between LM zero-shot and human performance on this dataset , but show that models can be fine-tuned to perform well on this particular task .

We hope that this work will encourage further study of nonliteral reasoning in LMs , especially in few-shot settings .

Given that metaphorical reasoning may play a role in problem-solving and linguistic creativity , the development of models , training methods , or datasets that enable metaphorical reasoning may improve models ' abilities to reason creatively and draw analogies between situations that may appear to be different on the surface .

One avenue we hope to investigate is multimodal metaphors , as this dataset currently includes only text-based metaphors .

Nonliteral expressions also remain understudied cross-linguistically , but further work on identifying and interpreting metaphors in other languages may also improve the abilities of multilingual models .

They introduce a Winogrand-style nonliteral language understanding task to test if language models can make correct inferences on human-generated metaphors.


In this paper , we propose a method for learning a classifier which combines outputs of more than one Japanese named entity extractors .

The proposed combination method belongs to the family of stacked generalizers , which is in principle a technique of combining outputs of several classifiers at the first stage by learning a second stage classifier to combine those outputs at the first stage .

Individual models to be combined are based on maximum entropy models , one of which always considers surrounding contexts of a fixed length , while the other considers those of variable lengths according to the number of constituent morphemes of named entities .

As an algorithm for learning the second stage classifier , we employ a decision list learning method .

Experimental evaluation shows that the proposed method achieves improvement over the best known results with Japanese named entity extractors based on maximum entropy models .

In the recent corpus-based NLP research , system combination techniques have been successfully applied to several tasks such as parts-of-speech tagging ( van Halteren et al . , 1998 ) , base noun phrase chunking ( Tjong Kim Sang , 2000 ) , and parsing ( Henderson and Brill , 1999 ; Henderson and Brill , 2000 ) .

The aim of system combination is to combine portions of the individual systems ' outputs which are partial but can be regarded as highly accurate .

The process of system combination can be decomposed into the following two sub-processes :

Collect systems which behave as differently as possible : it would help a lot if at least the collected systems tend to make errors of different types , because simple voting technique can identify correct outputs .

Previously studied techniques for collecting such systems include : i ) using several existing real systems ( van Halteren et al . , 1998 ; Brill and Wu , 1998 ; Henderson and Brill , 1999 ; Tjong Kim Sang , 2000 ) , ii ) bagging / boosting techniques ( Henderson and Brill , 1999 ; Henderson and Brill , 2000 ) , and iii ) switching the data expression and obtaining several models ( Tjong Kim Sang , 2000 ) .

2 . Combine the outputs of the several systems : previously studied techniques include : i ) voting techniques ( van Halteren et al . , 1998 ; Tjong Kim Sang , 2000 ; Henderson and Brill , 1999 ; Henderson and Brill , 2000 ) , ii ) switching among several systems according to confidence values they provide ( Henderson and Brill , 1999 ) , iii ) stacking techniques ( Wolpert , 1992 ) which train a second stage classifier for combining outputs of classifiers at the first stage ( van Halteren et al . , 1998 ; Brill and Wu , 1998 ; Tjong Kim Sang , 2000 ) .

In this paper , we propose a method for combining outputs of ( Japanese ) named entity chunkers , which belongs to the family of stacking techniques .

In the sub-process 1 , we focus on models which differ in the lengths of preceding / subsequent contexts to be incorporated in the models .

As the base model for supervised learning of Japanese named entity chunking , we employ a model based on the maximum entropy model ( Uchimoto et al . , 2000 ) , which performed the best in IREX ( Information Retrieval and Extraction Exercise ) Workshop ( IREX Committee , 1999 ) among those based on machine learning techniques .

Uchimoto et al . ( 2000 ) reported that the optimal number of preceding / subsequent contexts to be incorporated in the model is two morphemes to both left

In this paper , we train several maximum entropy models which differ in the lengths of preceding / subsequent contexts , and then combine their outputs .

As the sub-process 2 , we propose to apply a stacking technique which learns a classifier for combining outputs of several named entity chunkers .

This second stage classifier learns rules for accepting / rejecting outputs of several individual named entity chunkers .

The proposed method can be applied to the cases where the number of constituent systems is quite small ( e.g. , two ) .

Actually , in the experimental evaluation , we show that the results of combining the best performing model of Uchimoto et al . ( 2000 ) with the one which performs poorly but extracts named entities quite different from those of the best performing model can help improve the performance of the best model .

This paper proposed a method for learning a classifier to combine outputs of more than one Japanese named entity chunkers .

Experimental evaluation showed that the proposed method achieved improvement in F-measure over the best known results with an ME model ( Uchimoto et al . , 2000 ) , when a complementary model extracted named entities quite differently from the best performing model .

They propose a decision list-based classifier which learns to combine outputs from multiple maximum entropy-based Japanese named entity extraction models with different context lengths.

combine output multiple
japanese name entity


In this paper , we propose a joint architecture that captures language , rhyme and meter for sonnet modelling .

We assess the quality of generated poems using crowd and expert judgements .

The stress and rhyme models perform very well , as generated poems are largely indistinguishable from human-written poems .

Expert evaluation , however , reveals that a vanilla language model captures meter implicitly , and that machine-generated poems still underperform in terms of readability and emotion .

Our research shows the importance expert evaluation for poetry generation , and that future research should look beyond rhyme / meter and focus on poetic language .

With the recent surge of interest in deep learning , one question that is being asked across a number of fronts is : can deep learning techniques be harnessed for creative purposes ?

Creative applications where such research exists include the composition of music ( Humphrey et al . , 2013 ; Sturm et al . , 2016 ; Choi et al . , 2016 ) , the design of sculptures ( Lehman et al . , 2016 ) , and automatic choreography ( Crnkovic-Friis and Crnkovic-Friis , 2016 ) .

In this paper , we focus on a creative textual task : automatic poetry composition .

A distinguishing feature of poetry is its aesthetic forms , e.g. rhyme and rhythm / meter .

In this work , we treat the task of poem generation as a constrained language modelling task , such that lines of a given poem rhyme , and each line follows a canonical meter and has a fixed number 1 Noting that there are many notable divergences from this in the work of particular poets ( e.g. Walt Whitman ) and poetry types ( such as free verse or haiku ) .

Shall I compare thee to a summer 's day ?

Thou art more lovely and more temperate :

Rough winds do shake the darling buds of May , And summer 's lease hath all too short a date : of stresses .

Specifically , we focus on sonnets and generate quatrains in iambic pentameter ( e.g. see Figure 1 ) , based on an unsupervised model of language , rhyme and meter trained on a novel corpus of sonnets .

• our proposed stress and rhyme models work very well , generating sonnet quatrains with stress and rhyme patterns that are indistinguishable from human-written poems and rated highly by an expert ; • a vanilla language model trained over our sonnet corpus , surprisingly , captures meter implicitly at human-level performance ; • while crowd workers rate the poems generated by our best model as nearly indistinguishable from published poems by humans , an expert annotator found the machine-generated poems to lack readability and emotion , and our best model to be only comparable to a vanilla language model on these dimensions ; • most work on poetry generation focuses on meter ( Greene et al . , 2010 ; Ghazvininejad et al . , 2016 ; Hopkins and Kiela , 2017 ) ; our results suggest that future research should look beyond meter and focus on improving readability .

In this , we develop a new annotation framework for the evaluation of machine-generated poems , and release both a novel data of sonnets and the full source code associated with this research.2

We propose a joint model of language , meter and rhyme that captures language and form for modelling sonnets .

We provide quantitative analyses for each component , and assess the quality of generated poems using judgements from crowdworkers and a literature expert .

Our research reveals that vanilla LSTM language model captures meter implicitly , and our proposed rhyme model performs exceptionally well .

Machine-generated generated poems , however , still underperform in terms of readability and emotion .

They propose a joint architecture that captures language, rhyme and meter for sonnet modelling to generate quatrains coupled with a corpus to train.


This paper presents a deep semantic similarity model ( DSSM ) , a special type of deep neural networks designed for text analysis , for recommending target documents to be of interest to a user based on a source document that she is reading .

We observe , identify , and detect naturally occurring signals of interestingness in click transitions on the Web between source and target documents , which we collect from commercial Web browser logs .

The DSSM is trained on millions of Web transitions , and maps source-target document pairs to feature vectors in a latent space in such a way that the distance between source documents and their corresponding interesting targets in that space is minimized .

The effectiveness of the DSSM is demonstrated using two interestingness tasks : automatic highlighting and contextual entity search .

The results on large-scale , real-world datasets show that the semantics of documents are important for modeling interestingness and that the DSSM leads to significant quality improvement on both tasks , outperforming not only the classic document models that do not use semantics but also state-of-the-art topic models .

Tasks of predicting what interests a user based on the document she is reading are fundamental to many online recommendation systems .

A recent survey is due to Ricci et al . ( 2011 ) .

In this paper , we exploit the use of a deep semantic model for two such interestingness tasks in which document semantics play a crucial role : automatic highlighting and contextual entity search .

In this task we want a recommendation system to automatically discover the entities ( e.g. , a person , location , organi-zation etc . ) that interest a user when reading a document and to highlight the corresponding text spans , referred to as keywords afterwards .

We show in this study that document semantics are among the most important factors that influence what is perceived as interesting to the user .

For example , we observe in Web browsing logs that when a user reads an article about a movie , she is more likely to browse to an article about an actor or character than to another movie or the director .

After identifying the keywords that represent the entities of interest to the user , we also want the system to recommend new , interesting documents by searching the Web for supplementary information about these entities .

The task is challenging because the same keywords often refer to different entities , and interesting supplementary information to the highlighted entity is highly sensitive to the semantic context .

For example , " Paul Simon " can refer to many people , such as the singer and the senator .

Consider an article about the music of Paul Simon and another about his life .

Related content about his upcoming concert tour is much more interesting in the first context , while an article about his family is more interesting in the second .

At the heart of these two tasks is the notion of interestingness .

In this paper , we model and make use of this notion of interestingness with a deep semantic similarity model ( DSSM ) .

The model , extending from the deep neural networks shown recently to be highly effective for speech recognition ( Hinton et al . , 2012 ; Deng et al . , 2013 ) and computer vision ( Krizhevsky et al . , 2012 ; Markoff , 2014 ) , is semantic because it maps documents to feature vectors in a latent semantic space , also known as semantic representations .

The model is deep because it employs a neural network with several hidden layers including a special convolutional-pooling structure to identify keywords and extract hidden semantic features at different levels of abstractions , layer by layer .

The semantic representation is computed through a deep neural network after its training by backpropagation with respect to an objective tailored to the respective interestingness tasks .

We obtain naturally occurring " interest " signals by observing Web browser transitions , from a source document to a target document , in Web usage logs of a commercial browser .

Our training data is sampled from these transitions .

The use of the DSSM to model interestingness is motivated by the recent success of applying related deep neural networks to computer vision ( Krizhevshy et al . 2012 ; Markoff , 2014 ) , speech recognition ( Hinton et al . 2012 ) , text processing ( Collobert et al . 2011 ) , and Web search ( Huang et al . 2013 ) .

Among them , ( Huang et al . 2013 ) is most relevant to our work .

They also use a deep neural network to map documents to feature vectors in a latent semantic space .

However , their model is designed to represent the relevance between queries and documents , which differs from the notion of interestingness between documents studied in this paper .

It is often the case that a user is interested in a document because it provides supplementary information about the entities or concepts she encounters when reading another document although the overall contents of the second documents is not highly relevant .

For example , a user may be interested in knowing more about the history of University of Washington after reading the news about President Obama 's visit to Seattle .

To better model interestingness , we extend the model of Huang et al . ( 2013 ) in two significant aspects .

First , while Huang et al . treat a document as a bag of words for semantic mapping , the DSSM treats a document as a sequence of words and tries to discover prominent keywords .

These keywords represent the entities or concepts that might interest users , via the convolutional and max-pooling layers which are related to the deep models used for computer vision ( Krizhevsky et al . , 2013 ) and speech recognition ( Deng et al . , 2013a ) but are not used in Huang et al . 's model .

The DSSM then forms the high-level semantic representation of the whole document based on these keywords .

Second , instead of directly computing the document relevance score using cosine similarity in the learned semantic space , as in Huang et al . ( 2013 ) , we feed the features derived from the semantic representations of documents to a ranker which is trained in a supervised manner .

As a result , a document that is not highly relevant to another document a user is reading ( i.e. , the distance between their derived feature 1 We stress here that , although the click signal is available to form a dataset and a gold standard ranker ( to be described in vectors is big ) may still have a high score of interestingness because the former provides useful information about an entity mentioned in the latter .

Such information and entity are encoded , respectively , by ( some subsets of ) the semantic features in their corresponding documents .

In Sections 4 and 5 , we empirically demonstrate that the aforementioned two extensions lead to significant quality improvements for the two interestingness tasks presented in this paper .

Before giving a formal description of the DSSM in Section 3 , we formally define the interestingness function , and then introduce our data set of naturally occurring interest signals .

Modeling interestingness is fundamental to many online recommendation systems .

We obtain naturally occurring interest signals by observing Web browsing transitions where users click from one webpage to another .

We propose to model this " interestingness " with a deep semantic similarity model ( DSSM ) , based on deep neural networks with special convolutional-pooling structure , mapping source-target document pairs to feature vectors in a latent semantic space .

We train the DSSM using browsing transitions between documents .

Finally , we demonstrate the effectiveness of our model on two interestingness tasks : automatic highlighting and contextual entity search .

Our results on large-scale , real-world datasets show that the semantics of documents computed by the DSSM are important for modeling interestingness and that the new model leads to significant improvements on both tasks .

DSSM is shown to outperform not only the classic document models that do not use ( latent ) semantics but also stateof-the-art topic models that do not have the deep and convolutional architecture characterizing the DSSM .

One area of future work is to extend our method to model interestingness given an entire user session , which consists of a sequence of browsing events .

We believe that the prior browsing and interaction history recorded in the session provides additional signals for predicting interestingness .

To capture such signals , our model needs to be extended to adequately represent time series ( e.g. , causal relations and consequences of actions ) .

One potentially effective model for such a purpose is based on the architecture of recurrent neural networks ( e.g. , Mikolov et al . 2010 ; Chen and Deng , 2014 ) , which can be incorporated into the deep semantic model proposed in this paper .

They propose a CNN-based model that models documents as a sequence of words in latent space and trains it on millions of Web transitions.


Current measures for evaluating text simplification systems focus on evaluating lexical text aspects , neglecting its structural aspects .

In this paper we propose the first measure to address structural aspects of text simplification , called SAMSA .

It leverages recent advances in semantic parsing to assess simplification quality by decomposing the input based on its semantic structure and comparing it to the output .

SAMSA provides a reference-less automatic evaluation procedure , avoiding the problems that reference-based methods face due to the vast space of valid simplifications for a given sentence .

Our human evaluation experiments show both SAMSA 's substantial correlation with human judgments , as well as the deficiency of existing reference-based measures in evaluating structural simplification .

Text simplification ( TS ) addresses the translation of an input sentence into one or more simpler sentences .

It is a useful preprocessing step for several NLP tasks , such as machine translation ( Chandrasekar et al . , 1996 ; Mishra et al . , 2014 ) and relation extraction ( Niklaus et al . , 2016 ) , and has also been shown useful in the development of reading aids , e.g. , for people with dyslexia ( Rello et al . , 2013 ) or non-native speakers ( Siddharthan , 2002 ) .

The task has attracted much attention in the past decade ( Zhu et al . , 2010 ; Woodsend and Lapata , 2011 ; Wubben et al . , 2012 ; Siddharthan and Angrosh , 2014 ; Narayan and Gardent , 2014 ) , but has yet to converge on an evaluation protocol that yields comparable results across different methods and strongly correlates with human judgments .

This is in part due to the difficulty to combine the effects of different simplification operations ( e.g. , deletion , splitting and substitution ) .

al . ( 2016 ) has recently made considerable progress towards that goal , and proposed to tackle it both by using an improved reference-based measure , named SARI , and by increasing the number of references .

However , their research focused on lexical , rather than structural simplification , which provides a complementary view of TS quality as this paper will show .

This paper focuses on the evaluation of the structural aspects of the task .

We introduce the semantic measure SAMSA ( Simplification Automatic evaluation Measure through Semantic Annotation ) , the first structure-aware measure for TS in general , and the first to use semantic structure in this context in particular .

SAMSA stipulates that an optimal split of the input is one where each predicate-argument structure is assigned its own sentence , and measures to what extent this assertion holds for the input-output pair in question , by using semantic structure .

SAMSA focuses on the core semantic components of the sentence , and is tolerant towards the deletion of other units .

2For example , SAMSA will assign a high score to the output split " John got home .

for the input sentence " John got home and gave Mary a call .

" , as it splits each of its predicate-argument structures to a different sentence .

Splits that alter predicate-argument relations such as " John got home and gave .

Mary called . " are penalized by SAMSA .

SAMSA 's use of semantic structures for TS evaluation has several motivations .

First , it provides means to measure the extent to which the meaning of the source is preserved in the output .

Second , it provides means for measuring whether the input sentence was split to semantic units of the right granularity .

Third , defining a semantic measure that does not require references avoids the difficulties incurred by their non-uniqueness , and the difficulty in collecting high quality references , as reported by Xu et al . ( 2015 ) and by Narayan and Gardent ( 2014 ) with respect to the Parallel Wikipedia Corpus ( PWKP ; Zhu et al . , 2010 ) .

SAMSA is further motivated by its use of semantic annotation only on the source side , which allows to evaluate multiple systems using same source-side annotation , and avoids the need to parse system outputs , which can be garbled .

In this paper we use the UCCA scheme for defining semantic structure ( Abend and Rappoport , 2013 ) .

UCCA has been shown to be preserved remarkably well across translations ( Sulem et al . , 2015 ) and has also been successfully used for machine translation evaluation ( Birch et al . , 2016 )

We note , however , that SAMSA can be adapted to work with any semantic scheme that captures predicate-argument relations , such as AMR ( Banarescu et al . , 2013 ) or Discourse Representation Structures ( Kamp , 1981 ) , as used by Narayan and Gardent ( 2014 ) .

We experiment with SAMSA both where semantic annotation is carried out manually , and where it is carried out by a parser .

We conduct human rating experiments and compare the resulting system rankings with those predicted by SAMSA .

We find that SAMSA 's rankings obtain high correlations with human rankings , and compare favorably to existing referencebased measures for TS .

Moreover , our results show that existing measures , which mainly target lexical simplification , are ill-suited to predict human judgments where structural simplification is involved .

Finally , we apply SAMSA to the dataset of the QATS shared task on simplification evaluation ( Štajner et al . , 2016 ) .

We find that SAMSA obtains comparative correlation with human judgments on the task , despite operating in a more restricted setting , as it does not use human ratings as training data and focuses only on structural aspects of simplicity .

Section 5 details the collection of human judgments .

Our experimental setup for comparing our human and automatic rankings is given in Section 6 , and results are given in Section 7 , showing superior results for SAMSA .

A discussion on the results is presented in Section 8 .

Section 9 presents experiments with SAMSA on the QATS evaluation benchmark .

We presented the first structure-aware metric for text simplification , SAMSA , and the first evaluation experiments that directly target the structural simplification component , separately from the lexical component .

We argue that the structural and lexical dimensions of simplification are loosely related , and that TS evaluation protocols should assess both .

We empirically demonstrate that strong measures that assess lexical simplification quality ( notably SARI ) , fail to correlate with human judgments when structural simplification is performed by the evaluated systems .

Our experiments show that SAMSA correlates well with human judgments in such settings , which demonstrates its usefulness for evaluating and tuning statistical simplification systems , and shows that structural evaluation provides a complementary perspective on simplification quality .

They propose a reference-free metric that uses semantic parsers to decompose inputs and outputs before comparing to assess their semantic structures.