We present a simple preordering approach for machine translation based on a featurerich logistic regression model to predict whether two children of the same node in the source-side parse tree should be swapped or not .

Given the pair-wise children regression scores we conduct an efficient depth-first branch-and-bound search through the space of possible children permutations , avoiding using a cascade of classifiers or limiting the list of possible ordering outcomes .

We report experiments in translating English to Japanese and Korean , demonstrating superior performance as ( a ) the number of crossing links drops by more than 10 % absolute with respect to other state-of-the-art preordering approaches , ( b ) BLEU scores improve on 2.2 points over the baseline with lexicalised reordering model , and ( c ) decoding can be carried out 80 times faster .

Source-side preordering for translation is the task of rearranging the order of a given source sentence so that it best resembles the order of the target sentence .

It is a divide-and-conquer strategy aiming to decouple long-range word movement from the core translation task .

The main advantage is that translation becomes computationally cheaper as less word movement needs to be considered , which results in faster and better translations , if preordering is done well and efficiently .

Preordering also can facilitate better estimation of alignment and translation models as the parallel data becomes more monotonically-aligned , and translation gains can be obtained for various system architectures , e.g. phrase-based , hierarchical phrase-based , etc .

For these reasons , preordering has a clear research and commercial interest , as reflected by the extensive previous work on the subject ( see Section 2 ) .

From these approaches , we are particularly interested in those that ( i ) involve little or no human intervention , ( ii ) require limited computational resources at runtime , and ( iii ) make use of available linguistic analysis tools .

In this paper we propose a novel preordering approach based on a logistic regression model trained to predict whether to swap nodes in the source-side dependency tree .

For each pair of sibling nodes in the tree , the model uses a feature-rich representation that includes lexical cues to make relative reordering predictions between them .

Given these predictions , we conduct a depth-first branch-and-bound search through the space of possible permutations of all sibling nodes , using the regression scores to guide the search .

First , the search for permutations is efficient and does not require specific heuristics or hard limits for nodes with many children .

Second , the inclusion of the regression prediction directly into the search allows for finer-grained global decisions as the predictions that the model is more confident about are preferred .

Finally , the use of a single regression model to handle any number of child nodes avoids incurring sparsity issues , while allowing the integration of a vast number of features into the preordering model .

We empirically contrast our proposed method against another preordering approach based on automatically-extracted rules when translating English into Japanese and Korean .

We demonstrate a significant reduction in number of crossing links of more than 10 % absolute , as well as translation gains of over 2.2 BLEU points over the baseline .

We also show it outperforms a multi-class classification approach and analyse why this is the case .

We have presented a novel preordering approach that estimates a preference for swapping or not swapping pairs of children nodes in the sourceside dependency tree by training a feature-rich logistic regression model .

Given the pair-wise scores , we efficiently search through the space of possible children permutations using depth-first branch-and-bound search .

The approach is able to incorporate large numbers of features including lexical cues , is efficient at runtime even with a large number of children , and proves superior to other state-of-the-art preordering approaches both in terms of crossing score and translation performance .


There is an increasing interest in studying natural language and computer code together , as large corpora of programming texts become readily available on the Internet .

For example , StackOverflow currently has over 15 million programming related questions written by 8.5 million users .

Meanwhile , there is still a lack of fundamental NLP techniques for identifying code tokens or software-related named entities that appear within natural language sentences .

In this paper , we introduce a new named entity recognition ( NER ) corpus for the computer programming domain , consisting of 15,372 sentences annotated with 20 fine-grained entity types .

We trained indomain BERT representations ( BERTOverflow ) on 152 million sentences from Stack-Overflow , which lead to an absolute increase of +10 F 1 score over off-the-shelf BERT .

We also present the SoftNER model which achieves an overall 79.10 F 1 score for code and named entity recognition on StackOverflow data .

Our SoftNER model incorporates a context-independent code token classifier with corpus-level features to improve the BERTbased tagging model .

Recently there has been significant interest in modeling human language together with computer code ( Quirk et al . , 2015 ; Iyer et al . , 2016 ; Yin and Neubig , 2018 ) , as more data becomes available on websites such as StackOverflow and GitHub .

This is an ambitious yet promising direction for scaling up language understanding to richer domains .

Access to domain-specific NLP tools could help a wide range of downstream applications .

For example , extracting software knowledge bases from 1 Our code and data are available at : https:// github.com / jeniyat / StackOverflowNER/ text ( Movshovitz-Attias and Cohen , 2015 ) , developing better quality measurements of StackOverflow posts ( Ravi et al . , 2014 ) , finding similar questions ( Amirreza Shirani , 2019 ) and more .

However , there is a lack of NLP resources and techniques for identifying software-related named entities ( e.g. , variable names or application names ) within natural language texts .

In this paper , we present a comprehensive study that investigates the unique challenges of named entity recognition in the social computer programming domain .

These named entities are often ambiguous and have implicit reliance on the accompanied code snippets .

For example , the word ' list ' commonly refers to a data structure , but can also be used as a variable name ( Figure 1 ) .

In order to recognize these entities , we propose a software-related named entity recognizer ( Soft-NER ) that utilizes an attention network to combine the local sentence-level context with corpuslevel information extracted from the code snippets .

Using our newly annotated corpus of 15,372 sentences in StackOverflow , we rigorously test our proposed SoftNER model , which outperforms BiLSTM-CRF model and fine-tuned BERT model for identifying 20 types of software-related named entities .

Our key contributions are the following :

• A new StackOverflow NER corpus manually annotated with 20 types of named en-tities , including all in-line code within natural language sentences ( § 2 ) .

We demonstrate that NER in the software domain is an ideal benchmark task for testing effectiveness of contextual word representations , such as ELMo ( Peters et al . , 2018 ) and BERT ( Devlin et al . , 2019 ) , due to its inherent polysemy and salient reliance on context .

An in-domain trained neural SoftNER tagger for StackOveflow ( § 3 ) that can recognize 20 fine-grained named entity types related to software developing .

We also tested its performance on GitHub data of readme files and issue reports .

A code token recognizer ( § 3.1 ) that utilizes StackOveflow code snippets to capture the spelling patterns of code-related tokens , and consistently improves the NER tagger .

In-domain pretrained ELMo and BERT representations ( § 3.3 ) on 152 million sentences from StackOverflow that significantly outperforms off-the-shelf ELMo and leads to more than 21 points increase in F 1 score over offthe-shelf BERT .

Overall , our named entity tagger ( SoftNER ) achieves a 79.10 % F 1 score on StackOverflow and 61.08 % F 1 score on GitHub data for extracting the 20 software related named entity types .

We believe this performance is sufficiently strong to be practically useful .

We have released our data and code , including the named entity tagger , our annotated corpus , annotation guideline , a specially designed tokenizer , and pre-trained StackOverflow BERT and ELMo embeddings .

In this work , we investigated the task of named entity recognition in the social computer programming domain .

We developed a new NER corpus of 15,372 sentences from StackOverflow and 6,510 sentences from GitHub annotated with 20 fine-grained named entities .

We demonstrate that this new corpus is an ideal benchmark dataset for contextual word representations , as there are many challenging ambiguities that often require long-distance context to resolve .

We also proposed a novel attention based model , named Soft-NER , that outperforms the state-of-the-art NER models on this dataset .

Furthermore , we investigated the important sub-task of code recognition .

Our code recognition model captures additional spelling information beyond then contextual word representations and consistently helps to improve the NER performance .

We believe our corpus , StackOverflow-specific BERT embeddings and named entity tagger will be useful for various language-and-code tasks , such as code retrieval , software knowledge base extraction and automated question-answering .

We implemented a CRF baseline model using CRFsuite 8 to extract the software entities .

This model uses standard orthographic , contextual and gazetteer features .

It also includes the code markdown tags ( § 3.1.3 ) and a set of regular expression features .

The regular expressions are developed to recognize specific categories of coderelated entities .

Feature ablation experiments on this CRF model are presented in Table 9


Populating Knowledge Base ( KB ) with new knowledge facts from reliable text resources usually consists of linking name mentions to KB entities and identifying relationship between entity pairs .

However , the task often suffers from errors propagating from upstream entity linkers to downstream relation extractors .

In this paper , we propose a novel joint inference framework to allow interactions between the two subtasks and find an optimal assignment by addressing the coherence among preliminary local predictions : whether the types of entities meet the expectations of relations explicitly or implicitly , and whether the local predictions are globally compatible .

We further measure the confidence of the extracted triples by looking at the details of the complete extraction process .

Experiments show that the proposed framework can significantly reduce the error propagations thus obtain more reliable facts , and outperforms competitive baselines with state-of-the-art relation extraction models .

Recent advances in natural language processing have made it possible to construct structured KBs from online encyclopedia resources , at an unprecedented scale and much more efficiently than traditional manual edit .

However , in those KBs , entities which are popular to the community usually contain more knowledge facts , e.g. , the basketball player LeBron James , the actor Nicholas Cage , etc . , while most other entities often have fewer facts .

On the other hand , knowledge facts should be updated as the development of entities , such as changes in the cabinet , a marriage event , or an acquisition between two companies , etc .

In order to address the above issues , we could consult populating existing KBs from reliable text resources , e.g. , newswire , which usually involves enriching KBs with new entities and populating KBs with new knowledge facts , in the form of < Entity , Relation , Entity > triple .

In this paper , we will focus on the latter , identifying relationship between two existing KB entities .

This task can be intuitively considered in a pipeline paradigm , that is , name mentions in the texts are first linked to entities in the KB ( entity linking , EL ) , and then the relationship between them are identified ( relation extraction , RE ) .

It is worth mentioning that the first task EL is different from the task of named entity recognition ( NER ) in traditional information extraction ( IE ) tasks , where NER recognizes and classifies the entity mentions ( to several predefined types ) in the texts , but EL focuses on linking the mentions to their corresponding entities in the KB .

Such pipeline systems often suffer from errors propagating from upstream to downstream , since only the local best results are selected to the next step .

One idea to solve the problem is to allow interactions among the local predictions of both subtasks and jointly select an optimal assignment to eliminate possible errors in the pipeline .

Suppose we are extracting knowledge facts from two sentences in Figure 1 : in sentence

[ 1 ] , if we are more confident to extract the relation fb : org.headquarters1 , we will be then prompted to select Bryant University , which indeed favors the RE prediction that requires an organization to be its subject .

On the other side , if we are sure to link to Kobe Bryant in sentence [ 2 ] , we will probably select fb : pro athlete.teams , whose subject position expects an athlete , e.g. , an NBA player .

It is not difficult to see that the argument type expectations of relations can encourage the two subtasks interact with each other and select coherent predictions for both of them .

In KBs with well-defined schemas , such as Freebase , type requirements can be collected and utilized explicitly ( Yao et al . , 2010 ) .

However , in other KBs with less reliable or even no schemas , it is more appropriate to implicitly capture the type expectations for a given relation ( Riedel et al . , 2013 ) .

Furthermore , previous RE approaches usually process each triple individually , which ignores whether those local predictions are compatible with each other .

For example , suppose the local predictions of the two sentences above are < Kobe Bryant , fb : org.headquarters , Smithfield , Rhode Is-land > and < Kobe Bryant , fb : pro athlete.teams , Los Angeles Lakers > , respectively , which , in fact , disagree with each other with respect to the KB , since , in most cases , these two relations can not share subjects .

Now we can see that either the relation predictions or the EL results for " Bryant " are incorrect .

Those disagreements provide us an effective way to remove the possible incorrect predictions that cause the incompatibilities .

On the other hand , the automatically extracted knowledge facts inevitably contain errors , especially for those triples collected from open domain .

Extractions with confidence scores will be more than useful for users to make proper decisions according to their requirements , such as trading recall for precision , or supporting approximate queries .

In this paper , we propose a joint framework to populate an existing KB with new knowledge facts extracted from reliable text resources .

The joint framework is designed to address the error propagation issue in a pipeline system , where subtasks are optimized in isolation and locally .

We find an optimal configuration from top k results of both subtasks , which maximizes the scores of each step , fulfills the argument type expectations of relations , which can be captured explicitly or implicitly , in the KB , and avoids globally incoherent predictions .

We formulate this optimization problem in an Integer Linear Program ( ILP ) framework , and further adopt a logistic regression model to measure the reliability of the whole process , and assign confidences to all extracted triples to facilitate further applications .

The experiments on a real-world case study show that our framework can eliminate error propagations in the pipeline systems by taking relations ' argument type expectations and global compatibilities into account , thus outperforms the pipeline approaches based on state-ofthe-art relation extractors by a large margin .

Furthermore , we investigate both explicit and implicit type clues for relations , and provide suggestions about which to choose according to the characteristics of existing KBs .

Additionally , our proposed confidence estimations can help to achieve a precision of over 85 % for a considerable amount of high quality extractions .

In the rest of the paper , we first review related work and then define the knowledge base population task that we will address in this paper .

Next we detail the proposed framework and present our experiments and results .

Finally , we conclude this paper with future directions .

In this paper , we propose a joint framework for the task of populating KBs with new knowledge facts , which performs joint inference on two subtasks , maximizes their preliminary scores , fulfills the type expectations of relations and avoids global incompatibilities with respect to all local predictions to find an optimal assignment .

Experimental results show that our framework can significantly eliminate the error propagations in pipeline systems and outperforms competitive pipeline systems with state-of-the-art RE models .

Regarding the explicit argument type expectations and the implicit ones , the latter can result in a higher overall precision , while the former performs better in acquiring high quality knowledge facts with higher confidence control , indicating that if the KB has a well-defined schema we can use explicit type requirements for the KBP task , and if not , our model can still perform well by mining the implicit ones .

Our framework can also generalize well with other preliminary RE models .

Furthermore , we assign extraction confidences to all extracted facts to facilitate further applications .

By setting a suitable threshold , our framework can populate high quality reliable knowledge facts to existing KBs .

For future work , we will address the NIL issue of EL where we currently assume all entities should be linked to a KB .

It would be also interesting to jointly model the two subtasks through structured learning , instead of joint inference only .

Currently we only use the coherence of extracted triples and the KB to estimate confidences , which would be nice to directly model the issue in a joint model .


Generative Adversarial Networks ( GANs ) are a promising approach for text generation that , unlike traditional language models ( LM ) , does not suffer from the problem of " exposure bias " .

However , A major hurdle for understanding the potential of GANs for text generation is the lack of a clear evaluation metric .

In this work , we propose to approximate the distribution of text generated by a GAN , which permits evaluating them with traditional probability-based LM metrics .

We apply our approximation procedure on several GAN-based models and show that they currently perform substantially worse than stateof-the-art LMs .

Our evaluation procedure promotes better understanding of the relation between GANs and LMs , and can accelerate progress in GAN-based text generation .

Neural networks have revolutionized the field of text generation , in machine translation ( Sutskever et al . , 2014 ; Neubig , 2017 ; Luong et al . , 2015 ; Chen et al . , 2018 ) , summarization ( See et al . , 2017 ) , image captioning ( You et al . , 2016 ) and many other applications ( Goldberg , 2017 ) .

Traditionally , text generation models are trained by going over a gold sequence of symbols ( characters or words ) from left-to-right , and maximizing the probability of the next symbol given the history , namely , a language modeling ( LM ) objective .

A commonly discussed drawback of such LM-based text generation is exposure bias ( Ranzato et al . , 2015 ) : during training , the model predicts the next token conditioned on the ground truth history , while at test time prediction is based on predicted tokens , causing a train-test mismatch .

Models trained in this manner often struggle to overcome previous prediction errors .

Generative Adversarial Networks ( Goodfellow et al . , 2014 ) offer a solution for exposure bias .

The authors contributed equally Originally introduced for images , GANs leverage a discriminator , which is trained to discriminate between real images and generated images via an adversarial loss .

In such a framework , the generator is not directly exposed to the ground truth data , but instead learns to imitate it using global feedback from the discriminator .

This has led to several attempts to use GANs for text generation , with a generator using either a recurrent neural network ( RNN ) ( Yu et al . , 2017 ; Guo et al . , 2017 ; Press et al . , 2017 ; Rajeswar et al . , 2017 ) , or a Convolutional Neural Network ( CNN ) ( Gulrajani et al . , 2017 ; Rajeswar et al . , 2017 ) .

However , evaluating GANs is more difficult than evaluating LMs .

While in language modeling , evaluation is based on the log-probability of a model on held-out text , this can not be straightforwardly extended to GAN-based text generation , because the generator outputs discrete tokens , rather than a probability distribution .

Currently , there is no single evaluation metric for GAN-based text generation , and existing metrics that are based on n-gram overlap are known to lack robustness and have low correlation with semantic coherence ( Semeniuta et al . , 2018 ) .

In this paper , we propose a method for evaluating GANs with standard probability-based evaluation metrics .

We show that the expected prediction of a GAN generator can be viewed as a LM , and suggest a simple Monte-Carlo method for approximating it .

The approximated probability distribution can then be evaluated with standard LM metrics such as perplexity or Bits Per Character ( BPC ) .

To empirically establish our claim , we implement our evaluation on several RNN-based GANs : ( Press et al . , 2017 ; Yu et al . , 2017 ; Guo et al . , 2017 ) .

We find that all models have substantially lower BPC compared to state-of-the-art LMs .

By directly comparing to LMs , we put in perspective the current performance of RNN-based GANs for text generation .

Our results are also in line with recent concurrent work by Caccia et al . ( 2018 ) , who reached a similar conclusion by comparing the performance of textual GANs to that of LMs using metrics suggested for GAN evaluation .

Our code is available at : http : //github.com / GuyTevet / SeqGAN-eval and http://github.com / GuyTevet/ rnn-gan-eval .

We propose an evaluation procedure for text GANs that is based on approximating the GAN output distribution and using standard LM metrics .

We provide a bound for the number of samples required for the approximation and empirically show in practice as few as 2000 samples per time-step suffice .

We evaluate character-based GAN models using our procedure , and show their performance is substantially lower than state-of-the-art LM .

We hope our simple evaluation method leads to progress in GAN-based text generation by shedding light on the quality of such models .


Statistical phrase-based translation learns translation rules from bilingual corpora , and has traditionally only used monolingual evidence to construct features that rescore existing translation candidates .

In this work , we present a semi-supervised graph-based approach for generating new translation rules that leverages bilingual and monolingual data .

The proposed technique first constructs phrase graphs using both source and target language monolingual corpora .

Next , graph propagation identifies translations of phrases that were not observed in the bilingual corpus , assuming that similar phrases have similar translations .

We report results on a large Arabic-English system and a medium-sized Urdu-English system .

Our proposed approach significantly improves the performance of competitive phrasebased systems , leading to consistent improvements between 1 and 4 BLEU points on standard evaluation sets .

Statistical approaches to machine translation ( SMT ) use sentence-aligned , parallel corpora to learn translation rules along with their probabilities .

With large amounts of data , phrase-based translation systems ( Koehn et al . , 2003 ; Chiang , 2007 ) achieve state-of-the-art results in many typologically diverse language pairs ( Bojar et al . , 2013 ) .

However , the limiting factor in the success of these techniques is parallel data availability .

Even in resource-rich languages , learning reliable translations of multiword phrases is a challenge , and an adequate phrasal inventory is crucial ⇤

This work was done while the first author was interning at Microsoft Research for effective translation .

This problem is exacerbated in the many language pairs for which parallel resources are either limited or nonexistent .

While parallel data is generally scarce , monolingual resources exist in abundance and are being created at accelerating rates .

Can we use monolingual data to augment the phrasal translations acquired from parallel data ?

The challenge of learning translations from monolingual data is of long standing interest , and has been approached in several ways ( Rapp , 1995 ; Callison-Burch et al . , 2006 ; Haghighi et al . , 2008 ; Ravi and Knight , 2011 ) .

Our work introduces a new take on the problem using graphbased semi-supervised learning to acquire translation rules and probabilities by leveraging both monolingual and parallel data resources .

On the source side , labeled phrases ( those with known translations ) are extracted from bilingual corpora , and unlabeled phrases are extracted from monolingual corpora ; together they are embedded as nodes in a graph , with the monolingual data determining edge strengths between nodes ( § 2.2 ) .

Unlike previous work ( Irvine and Callison-Burch , 2013a ; Razmara et al . , 2013 ) , we use higher order n-grams instead of restricting to unigrams , since our approach goes beyond OOV mitigation and can enrich the entire translation model by using evidence from monolingual text .

This enhancement alone results in an improvement of almost 1.4 BLEU points .

On the target side , phrases initially consisting of translations from the parallel data are selectively expanded with generated candidates ( § 2.1 ) , and are embedded in a target graph .

We then limit the set of translation options for each unlabeled source phrase ( § 2.3 ) , and using a structured graph propagation algorithm , where translation information is propagated from labeled to unlabeled phrases proportional to both source and target phrase similarities , we estimate probability distributions over translations for Figure 1 : Example source and target graphs used in our approach .

Labeled phrases on the source side are black ( with their corresponding translations on the target side also black ) ; unlabeled and generated ( § 2.1 ) phrases on the source and target sides respectively are white .

Labeled phrases also have conditional probability distributions defined over target phrases , which are extracted from the parallel corpora .

the unlabeled source phrases ( § 2.4 ) .

The additional phrases are incorporated in the SMT system through a secondary phrase table ( § 2.5 ) .

We evaluated the proposed approach on both Arabic-English and Urdu-English under a range of scenarios ( § 3 ) , varying the amount and type of monolingual corpora used , and obtained improvements between 1 and 4 BLEU points , even when using very large language models .

In this work , we presented an approach that can expand a translation model extracted from a sentence-aligned , bilingual corpus using a large amount of unstructured , monolingual data in both source and target languages , which leads to improvements of 1.4 and 1.2 BLEU points over strong baselines on evaluation sets , and in some scenarios gains in excess of 4 BLEU points .

In the future , we plan to estimate the graph structure through other learned , distributed representations .


We present a new approach to the design of deep networks for natural language processing ( NLP ) , based on the general technique of Tensor Product Representations ( TPRs ) for encoding and processing symbol structures in distributed neural networks .

A network architecture -the Tensor Product Generation Network ( TPGN ) -is proposed which is capable in principle of carrying out TPR computation , but which uses unconstrained deep learning to design its internal representations .

Instantiated in a model for image-caption generation , TPGN outperforms LSTM baselines when evaluated on the COCO dataset .

The TPR-capable structure enables interpretation of internal representations and operations , which prove to contain considerable grammatical content .

Our caption-generation model can be interpreted as generating sequences of grammatical categories and retrieving words by their categories from a plan encoded as a distributed representation .

In this paper we introduce a new architecture for natural language processing ( NLP ) .

On what type of principles can a computational architecture be founded ?

It would seem a sound principle to require that the hypothesis space for learning which an architecture provides include network hypotheses that are independently known to be suitable for performing the target task .

Our proposed architecture makes available to deep learning network configurations that perform natural language generation by use of Tensor Product Representations ( TPRs ) ( Smolensky and Legendre , 2006 ) .

Whether learning will create TPRs is unknown in advance , but what we can say with certainty is that the hypothesis space being searched during learn-ing includes TPRs as one appropriate solution to the problem .

TPRs are a general method for generating vector-space embeddings of complex symbol structures .

Prior work has proved that TPRs enable powerful symbol processing to be carried out using neural network computation ( Smolensky , 2012 ) .

This includes generating parse trees that conform to a grammar ( Cho et al . , 2017 ) , although incorporating such capabilities into deep learning networks such as those developed here remains for future work .

The architecture presented here relies on simpler use of TPRs to generate sentences ; grammars are not explicitly encoded here .

We test the proposed architecture by applying it to image-caption generation ( on the MS-COCO dataset , ( COCO , 2017 ) ) .

The results improve upon a baseline deploying a state-of-the-art LSTM architecture ( Vinyals et al . , 2015 ) , and the TPR foundations of the architecture provide greater interpretability .

Section 3 presents the proposed architecture , the Tensor Product Generation Network ( TPGN ) .

Section 4 describes the particular model we study for image captioning , and Section 5 presents the experimental results .

Importantly , what the model has learned is interpreted in Section 5.3 .

Section 6 discusses the relation of the new model to previous work and Section 7 concludes .

Tensor Product Representation ( TPR ) ( Smolensky , 1990 ) is a general technique for constructing vector embeddings of complex symbol structures in such a way that powerful symbolic functions can be computed using hand-designed neural network computation .

Integrating TPR with deep learning is a largely open problem for which the work presented here proposes a general approach : design deep architectures that are TPRcapable -TPR computation is within the scope of the capabilities of the architecture in principle .

For natural language generation , we proposed such an architecture , the Tensor Product Generation Network ( TPGN ): it embodies the TPR operation of unbinding which is used to extract particular symbols ( e.g. , words ) from complex structures ( e.g. , sentences ) .

The architecture can be interpreted as containing a part that encodes a sentence and a part that selects one structural role at a time to extract from the sentence .

We applied the approach to image-caption generation , developing a TPGN model that was evaluated on the COCO dataset , on which it outperformed LSTM baselines on a range of standard metrics .

Unlike standard LSTMs , however , the TPGN model admits a level of interpretability : we can see which roles are being unbound by the unbinding vectors generated internally within the model .

We find such roles contain considerable grammatical information , enabling POS tag prediction for the words they generate and displaying clustering by POS .


This paper describes a simple and principled approach to automatically construct sentiment lexicons using distant supervision .

We induce the sentiment association scores for the lexicon items from a model trained on a weakly supervised corpora .

Our empirical findings show that features extracted from such a machine-learned lexicon outperform models using manual or other automatically constructed sentiment lexicons .

Finally , our system achieves the state-of-the-art in Twitter Sentiment Analysis tasks from Semeval-2013 and ranks 2nd best in Semeval-2014 according to the average rank .

One of the early and rather successful models for sentiment analysis ( Pang and Lee , 2004 ; Pang and Lee , 2008 ) relied on manually constructed lexicons that map words to their sentiment , e.g. , positive , negative or neutral .

The document-level polarity is then assigned by performing some form of averaging , e.g. , majority voting , of individual word polarities found in the document .

These systems show an acceptable level of accuracy , they are easy to build and are highly computationally efficient as the only operation required to assign a polarity label are the word lookups and averaging .

However , the information about word polarities in a document are best exploited when using machine learning models to train a sentiment classifier .

In fact , most successful sentiment classification systems rely on supervised learning .

Interestingly , a simple bag of words model using just unigrams and bigrams with an SVM has shown excellent results ( Wang and Manning , 2012 ) performing on par or beating more complicated models , e.g. , using neural networks ( Socher et al . , 2011 ) .

Regarding Twitter sentiment analysis , the top performing system ( Mohammad et al . , 2013 ) from Semeval-2013 Twittter Sentiment Analysis task ( Nakov et al . , 2013 ) follows this recipe by training an SVM on various surface form , sentiment and semantic features .

Perhaps , the most valuable finding is that sentiment lexicons appear to be the most useful source of features accounting for over 8 point gains in the F-measure on top of the standard feature sets .

Sentiment lexicons are mappings from words to scores capturing the degree of the sentiment expressed by a given word .

While several manually constructed lexicons are made available , e.g. , the MPQA ( Wilson et al . , 2005 ) , the Bing and Liu ( Hu and Liu , 2004 ) and NRC Emoticon ( Mohammad and Turney , 2013 ) lexicons , providing high quality word-sentiment associations compiled by humans , still their main drawback is low recall .

For example , the largest NRC Emoticon lexicon contains only 14k items , whereas tweets with extremely sparse surface forms are known to form very large vocabularies .

Hence , using larger lexicons with better recall has the potential of learning more accurate models .

Extracting such lexicons automatically is a challenging and interesting problem ( Lau et al . , 2011 ; Bro and Ehrig , 2013 ; Liu et al . , 2013 ; Tai and Kao , 2013 ; Yang et al . , 2014 ; Huang et al . , 2014 ) .

However , different from previous work our goal is not to extract human-interpretable lexicons but to use them as a source of features to improve the classifier accuracy .

Following this idea , the authors in ( Mohammad et al . , 2013 ) use features derived from the lexicons to build a state-of-the-art sentiment classifier for Twitter .

They construct automatic lexicons using noisy labels automatically inferred from emoticons and hashtags present in the tweets .

The wordsentiment association scores are estimated using pointwise mutual information ( PMI ) computed between a word and a tweet label .

While the idea to model statistical correlations between the words and tweet labels using PMI or any other metric is rather intuitive , we believe there is a more effective way to exploit noisy labels for estimating the word-sentiment association scores .

Our method relies on the idea of distant supervision ( Marchetti-Bowick and Chambers , 2012 ) .

We use a large distantly supervised Twitter corpus , which contains noisy opinion labels ( positive or negative ) to learn a supervised polarity classifier .

We encode tweets using words and multi-word expressions as features ( which are also entries in our lexicon ) .

The weights from the learned model are then used to define which lexicon items to keep , i.e. , items that constitute a good sentiment lexicon .

The scores for the lexicon items can be then directly used to encode new tweets or used to derive more advanced features .

Using machine learning to induce the scores for the lexicon items has an advantage of learning the scores that are directly optimized for the classification task , where lexicon items with higher discriminative power tend to receive higher weights .

To assess the effectiveness of our approach , we reimplemented the state-of-the-art system ranking 1st in Semeval-2013 Twitter Sentiment Analysis challenge and used it as our baseline .

We show that adding features from our machine-learned sentiment lexicon yields better results than any of the automatic PMI lexicons used in the baseline and all of them combined together .

Our system obtains new state-of-the-art results on the SemEval-2013 message level task with an F-score of 71.32 -a 2 % of absolute improvement over the previous best system in SemEval-2013 .

We also evaluate the utility of the ML lexicon on the five test sets from a recent Semeval-2014 task showing significant improvement over a strong baseline .

Finally , our system shows high accuracy among the 42 systems participating in the Semeval-2014 challenge ranking 2nd best according to the average rank across all test sets .

We demonstrated a simple and principled approach grounded in machine learning to construct sentiment lexicons .

We show that using off-the-shelf machine learning tools to automatically extract lexicons greatly outperforms other automatically constructed lexicons that use pointwise mutual information to estimate sentiment scores for the lexicon items .

We have shown that combining our machinelearned lexicon with the previous best system yields state-of-the-art results in Semeval-2013 gaining over 2 points in F-score and ranking our system 2nd according to the average rank over the five test sets of Semeval-2014 .

Finally , our ML-based lexicon shows excellent results when added on top of the current state-of-the-art NRC system .

While our experimental study is focused on Twitter , our method is general enough to be applied to sentiment classification tasks on other domains .

In the future , we plan to experiment with constructing ML lexicons from larger Twitter corpora also using hashtags .

Recently , deep convolutional neural networks for sentence modelling ( Kalchbrenner et al . , 2014 ; Kim , 2014 ) have shown promising results on several NLP tasks .

In particular , ( Tang et al . , 2014 ) showed that learning sentiment-specific word embeddings and using them as features can boost the accuracy of existing sentiment classifiers .

In the future work we plan to explore such approaches .


Parallel corpus is an indispensable resource for translation model training in statistical machine translation ( SMT ) .

Instead of collecting more and more parallel training corpora , this paper aims to improve SMT performance by exploiting full potential of the existing parallel corpora .

Two kinds of methods are proposed : offline data optimization and online model optimization .

The offline method adapts the training data by redistributing the weight of each training sentence pairs .

The online method adapts the translation model by redistributing the weight of each predefined submodels .

Information retrieval model is used for the weighting scheme in both methods .

Experimental results show that without using any additional resource , both methods can improve SMT performance significantly .

Statistical machine translation relies heavily on the available training data .

Typically , the more data is used to estimate the parameters of the translation model , the better it can approximate the true translation probabilities , which will obviously lead to a higher translation performance .

However , large corpora are not easily available .

The collected corpora are usually from very different areas .

For example , the parallel corpora provided by LDC come from quite different domains , such as Hongkong laws , Hangkong Hansards and Hongkong news .

This results in the problem that a translation system trained on data from a particular domain(e.g .

Hongkong Hansards ) will perform poorly when translating text from a different domain(e.g . news articles ) .

Our experiments also show that simply putting all these domain specific corpora together will not always improve translation quality .

From another aspect , larger amount of training data also requires larger computational resources .

With the increasing of training data , the improvement of translation quality will become smaller and smaller .

Therefore , while keeping collecting more and more parallel corpora , it is also important to seek effective ways of making better use of available parallel training data .

There are two cases when we train a SMT system .

In one case , we know the target test set or target test domain , for example , when building a specific domain SMT system or when participating the NIST MT evaluation1 .

In the other case , we are unaware of any information of the testing data .

This paper presents two methods to exploit full potential of the available parallel corpora in the two cases .

For the first case , we try to optimize the training data offline to make it match the test data better in domain , topic and style , thus improving the translation performance .

For the second case , we first divide the training data into several domains and train submodels for each domain .

Then , in the translation process , we try to optimize the predefined models according to the online input source sentence .

Information retrieval model is used for similar sentences retrieval in both methods .

Our preliminary experiments show that both methods can improve SMT performance without using any additional data .

The remainder of this paper is organized as follows : Section 2 describes the offline data selection and optimization method .

Section 3 describes the online model optimization method .

The evaluation and discussion are given in section 4 .

Related work is introduced before concluding .

This paper presents two new methods to improve statistical machine translation performance by making better use of the available parallel training corpora .

The offline data selection method adapts the training corpora to the test domain by retrieving similar sentence pairs and redistributing their weight in the training data .

Experimental results show that the selected small subset achieves comparable or even better performance than the baseline system with much less training data .

The optimized training data can further improve translation performance without using any additional resource .

The online model optimization method adapts the translation model to the online test sentence by redistributing the weight of each predefined submodels .

Preliminary results show the effectiveness of the method .

Our work also demonstrates that in addition to larger training data , more relevant training data is also important for SMT model training .

In future work , we will improve our methods in several aspects .

Currently , the similar sentence retrieval model and the weighting schemes are very simple .

It might work better by trying other sophisticated similarity measure models or using some optimization algorithms to determine submodel 's weights .

Introducing language model optimization into our system might further improve translation performance .


The paper describes two parsing schemes : a shallow approach based on machine learning and a cascaded finite-state parser with a hand-crafted grammar .

It discusses several ways to combine them and presents evaluation results for the two individual approaches and their combination .

An underspecification scheme for the output of the finite-state parser is introduced and shown to improve performance .

In several areas of Natural Language Processing , a combination of different approaches has been found to give the best results .

It is especially rewarding to combine deep and shallow systems , where the former guarantees interpretability and high precision and the latter provides robustness and high recall .

This paper investigates such a combination consisting of an n-gram based shallow parser and a cascaded finite-state parser 1 with hand-crafted grammar and morphological checking .

The respective strengths and weaknesses of these approaches are brought to light in an in-depth evaluation on a treebank of German newspaper texts ( Skut et al . , 1997 ) containing ca .

The evaluation format chosen ( dependency tuples ) is used as the common denominator of the systems 1

Although not everyone would agree that finite-state parsers constitute a ' deep ' approach to parsing , they still are knowledge-based , require efforts of grammar-writing , a complex linguistic lexicon , manage without training data , etc . in building a hybrid parser with improved performance .

An underspecification scheme allows the finite-state parser partially ambiguous output .

It is shown that the other parser can in most cases successfully disambiguate such information .

Section 2 discusses the evaluation format adopted ( dependency structures ) , its advantages , but also some of its controversial points .

Section 3 formulates a classification problem on the basis of the evaluation format and applies a machine learner to it .

Section 4 describes the architecture of the cascaded finite-state parser and its output in a novel underspecification format .

Section 5 explores several combination strategies and tests them on several variants of the two base components .

Section 6 provides an in-depth evaluation of the component systems and the hybrid parser .

The paper has presented two approaches to German parsing ( n-gram based machine learning and cascaded finite-state parsing ) , and evaluated them on the basis of a large amount of data .

A new representation format has been introduced that allows underspecification of select types of syntactic ambiguity ( attachment and subcategorization ) even in the absence of a full-fledged chart .

Several methods have been discussed for combining the two approaches .

It has been shown that while combination with the shallow approach can only marginally improve performance of the cascaded parser if ideal disambiguation is assumed , a quite substantial rise is registered in situations closer to the real world where POS tagging is deficient and resolution of attachment and subcategorization ambiguities less than perfect .

In ongoing work , we look at integrating a statistic context-free parser called BitPar , which was written by Helmut Schmid and achieves .816

Interestingly , the performance goes up to .9474

F-score when BitPar is combined with the FS parser ( upper bound ) and .9443 for the lower bound .

So at least for German , combining parsers seems to be a pretty good idea .

Thanks are due to Helmut Schmid and Prof. C. Rohrer for discussions , and to the reviewers for their detailed comments .


Tree substitution grammars ( TSGs ) are a compelling alternative to context-free grammars for modelling syntax .

However , many popular techniques for estimating weighted TSGs ( under the moniker of Data Oriented Parsing ) suffer from the problems of inconsistency and over-fitting .

We present a theoretically principled model which solves these problems using a Bayesian non-parametric formulation .

Our model learns compact and simple grammars , uncovering latent linguistic structures ( e.g. , verb subcategorisation ) , and in doing so far out-performs a standard PCFG .

Many successful models of syntax are based on Probabilistic Context Free Grammars ( PCFGs ) ( e.g. , Collins ( 1999 ) ) .

However , directly learning a PCFG from a treebank results in poor parsing performance , due largely to the unrealistic independence assumptions imposed by the context-free assumption .

Considerable effort is required to coax good results from a PCFG , in the form of grammar engineering , feature selection and clever smoothing ( Collins , 1999 ; Charniak , 2000 ; Charniak and Johnson , 2005 ; Johnson , 1998 ) .

This effort must be repeated when moving to different languages , grammar formalisms or treebanks .

We propose that much of this hand-coded knowledge can be obtained automatically as an emergent property of the treebanked data , thereby reducing the need for human input in crafting the grammar .

We present a model for automatically learning a Probabilistic Tree Substitution Grammar ( PTSG ) , an extension to the PCFG in which non-terminals can rewrite as entire tree fragments ( elementary trees ) , not just immediate children .

These large fragments can be used to encode non-local context , such as head-lexicalisation and verb sub-categorisation .

Since no annotated data is available providing TSG derivations we must induce the PTSG productions and their probabilities in an unsupervised way from an ordinary treebank .

This is the same problem addressed by Data Oriented Parsing ( DOP , Bod et al . ( 2003 ) ) , a method which uses as productions all subtrees of the training corpus .

However , many of the DOP estimation methods have serious shortcomings ( Johnson , 2002 ) , namely inconsistency for DOP1 ( Bod , 2003 ) and overfitting of the maximum likelihood estimate ( Prescher et al . , 2004 ) .

In this paper we develop an alternative means of learning a PTSG from a treebanked corpus , with the twin objectives of a ) finding a grammar which accurately models the data and b ) keeping the grammar as simple as possible , with few , compact , elementary trees .

This is achieved using a prior to encourage sparsity and simplicity in a Bayesian nonparametric formulation .

The framework allows us to perform inference over an infinite space of grammar productions in an elegant and efficient manner .

The net result is a grammar which only uses the increased context afforded by the TSG when necessary to model the data , and otherwise uses context-free rules .

That is , our model learns to use larger rules when the CFG 's independence assumptions do not hold .

This contrasts with DOP , which seeks to use all elementary trees from the training set .

While our model is able , in theory , to use all such trees , in practice the data does not justify such a large grammar .

Grammars that are only about twice the size of a treebank PCFG provide large gains in accuracy .

We obtain additional improvements with grammars that are somewhat larger , but still much smaller than the DOP all-subtrees grammar .

The rules in these grammars are intuitive , potentially offering insights into grammatical structure which could be used in , e.g. , the development of syntactic ontologies and guidelines for future treebanking projects .

In this work we have presented a non-parametric Bayesian model for inducing tree substitution grammars .

By incorporating a structured prior over elementary rules our model is able to reason over the infinite space of all such rules , producing compact and simple grammars .

In doing so our model learns local structures for latent linguistic phenomena , such as verb subcategorisation and lexical agreement .

Our experimental results show that the induced grammars strongly out-perform standard PCFGs , and are comparable to a state-of-the-art parser on small data samples .

While our results on the full treebank are well shy of the best available parsers , we have proposed a number of improvements to the model and the parsing algorithm that could lead to state-of-theart performance in the future .


News headline generation aims to produce a short sentence to attract readers to read the news .

One news article often contains multiple keyphrases that are of interest to different users , which can naturally have multiple reasonable headlines .

However , most existing methods focus on the single headline generation .

In this paper , we propose generating multiple headlines with keyphrases of user interests , whose main idea is to generate multiple keyphrases of interest to users for the news first , and then generate multiple keyphrase-relevant headlines .

We propose a multi-source Transformer decoder , which takes three sources as inputs : ( a ) keyphrase , ( b ) keyphrase-filtered article , and ( c ) original article to generate keyphrase-relevant , highquality , and diverse headlines .

Furthermore , we propose a simple and effective method to mine the keyphrases of interest in the news article and build a first large-scale keyphraseaware news headline corpus , which contains over 180 K aligned triples of news article , headline , keyphrase .

Extensive experimental comparisons on the real-world dataset show that the proposed method achieves state-of-theart results in terms of quality and diversity 1 .

News Headline Generation is an under-explored subtask of text summarization ( See et al . , 2017 ; Gehrmann et al . , 2018 ; Zhong et al . , 2019 ) .

Unlike text summaries that contain multiple contextrelated sentences to cover the main ideas of a document , news headlines often contain a single short sentence to encourage users to read the news .

Since one news article typically contains multiple keyphrases or topics of interest to different users , it is useful to generate multiple headlines covering different keyphrases for the news article .

Multiheadline generation aims to generate multiple independent headlines , which allows us to recommend news with different news headlines based on the interests of users .

Besides , multi-headline generation can provide multiple hints for human news editors to assist them in writing news headlines .

However , most existing methods ( Takase et al . , 2016 ; Ayana et al . , 2016 ; Murao et al . , 2019 ; Colmenares et al . , 2019 ; Zhang et al . , 2018 ) focus on single-headline generation .

The headline generation process is treated as an one-to-one mapping ( the input is an article and the output is a headline ) , which trains and tests the models without any additional guiding information or constraints .

We argue that this may lead to two problems .

Firstly , since it is reasonable to generate multiple headlines for the news , training to generate the single ground-truth might result in a lack of more detailed guidance .

Even worse , a single ground-truth without any constraint or guidance is often not enough to measure the quality of the generated headline for model testing .

For example , even if a generated headline is considered reasonable by humans , it can get a low score in ROUGE ( Lin , 2004 ) , because it might focus on the keyphrases or aspects that are not consistent with the ground-truth .

In this paper , we incorporate the keyphrase information into the headline generation as additional guidance .

Unlike one-to-one mapping employed in previous works , we treat the headline generation process as a two-to-one mapping , where the inputs are news articles and keyphrases , and the output is a headline .

We propose a keyphrase-aware news multi-headline generation method , which contains two modules : ( a ) Keyphrase Generation Model , which aims to generate multiple keyphrases of interest to users for the news article .

( b ) Keyphrase-Aware Multi-Headline Generation Model , which takes the news article and a keyphrase as input and generates a keyphrase-relevant news headline .

For training models , we build a first large-scale news keyphrase-aware headline corpus that contains 180 K aligned triples of news article , headline , keyphrase .

As in years past , a lot of the food trends of the year were based on creating perfectly photogenic dishes .

An aesthetically pleasing dish , however , does n't mean it will stand the test of time .

In fact , it 's not uncommon for food trends to be all the hype one year and die out the next .

From broccoli coffee to " bowl food , " here are 10 food trends that you likely wo n't see in 2019 .

[ 15 sentences with 307 words are abbreviated from here . ]

In 2018 , restaurants all over the US decided it was a good idea to place gold foil on everything from ice cream to chicken wings to pizza resulting in an expensive food trend .

For example , the Ainsworth in New York City sells $ 1,000 worth of gold covered chicken wings .

It seems everyone can agree that this is a food trend that might soon disappear .

In this paper , we demonstrate how to enable news headline generation systems to be aware of keyphrases such that the model can generate diverse news headlines in a controlled manner .

We also build a first large-scale keyphrase-aware news headline corpus , which is based on mining the keyphrases of users ' interests in news articles with user queries .

Moreover , we propose a keyphraseaware news multi-headline generation model that contains a multi-source Transformer decoder with three variants of attention-based fusing mechanisms .

Extensive experiments on the real-world dataset show that our approach can generate highquality , keyphrase-relevant , and diverse news headlines , which outperforms many strong baselines .


Universal schema builds a knowledge base ( KB ) of entities and relations by jointly embedding all relation types from input KBs as well as textual patterns observed in raw text .

In most previous applications of universal schema , each textual pattern is represented as a single embedding , preventing generalization to unseen patterns .

Recent work employs a neural network to capture patterns ' compositional semantics , providing generalization to all possible input text .

In response , this paper introduces significant further improvements to the coverage and flexibility of universal schema relation extraction : predictions for entities unseen in training and multilingual transfer learning to domains with no annotation .

We evaluate our model through extensive experiments on the English and Spanish TAC KBP benchmark , outperforming the top system from TAC 2013 slot-filling using no handwritten patterns or additional annotation .

We also consider a multilingual setting in which English training data entities overlap with the seed KB , but Spanish text does not .

Despite having no annotation for Spanish data , we train an accurate predictor , with additional improvements obtained by tying word embeddings across languages .

Furthermore , we find that multilingual training improves English relation extraction accuracy .

Our approach is thus suited to broad-coverage automated knowledge base construction in a variety of languages and domains .

The goal of automatic knowledge base construction ( AKBC ) is to build a structured knowledge base ( KB ) of facts using a noisy corpus of raw text evidence , and perhaps an initial seed KB to be augmented ( Carlson et al . , 2010 ; Suchanek et al . , 2007 ; Bollacker et al . , 2008 ) .

AKBC supports downstream reasoning at a high level about extracted entities and their relations , and thus has broad-reaching applications to a variety of domains .

One challenge in AKBC is aligning knowledge from a structured KB with a text corpus in order to perform supervised learning through distant supervision .

Universal schema ( Riedel et al . , 2013 ) along with its extensions ( Yao et al . , 2013 ; Gardner et al . , 2014 ; Neelakantan et al . , 2015 ; Rocktaschel et al . , 2015 ) , avoids alignment by jointly embedding KB relations , entities , and surface text patterns .

This propagates information between KB annotation and corresponding textual evidence .

The above applications of universal schema express each text relation as a distinct item to be embedded .

This harms its ability to generalize to inputs not precisely seen at training time .

Recently , Toutanova et al . ( 2015 ) addressed this issue by embedding text patterns using a deep sentence encoder , which captures the compositional semantics of textual relations and allows for prediction on inputs never seen before .

This paper further expands the coverage abilities of universal schema relation extraction by introducing techniques for forming predictions for new entities unseen in training and even for new domains with no associated annotation .

In the extreme example of domain adaptation to a completely new language , we may have limited linguistic resources or labeled data such as treebanks , and only rarely a KB with adequate coverage .

Our method performs multilingual transfer learning , providing a predictive model for a language with no coverage in an existing KB , by leveraging common representations for shared entities across text corpora .

As depicted in Figure 1 , we simply require that one language have an available KB of seed facts .

We can further improve our models by tying a small set of word embeddings across languages using only simple knowledge about word-level translations , learning to embed semantically similar textual patterns from different languages into the same latent space .

In extensive experiments on the TAC Knowledge Base Population ( KBP ) slot-filling benchmark we outperform the top 2013 system with an F1 score of 40.7 and perform relation extraction in Spanish with no labeled data or direct overlap between the Spanish training corpus and the training KB , demonstrating that our approach is wellsuited for broad-coverage AKBC in low-resource languages and domains .

Interestingly , joint training with Spanish improves English accuracy .

English Low-resource in KB not in KB Figure 1 :

Splitting the entities in a multilingual AKBC training set into parts .

We only require that entities in the two corpora overlap .

Remarkably , we can train a model for the low-resource language even if entities in the lowresource language do not occur in the KB .

By jointly embedding English and Spanish corpora along with a KB , we can train an accurate Spanish relation extraction model using no direct annotation for relations in the Spanish data .

This approach has the added benefit of providing significant accuracy improvements for the English model , outperforming the top system on the 2013 TAC KBC slot filling task , without using the hand-coded rules or additional annotations of alternative systems .

By using deep sentence encoders , we can perform prediction for arbitrary input text and for entities unseen in training .

Sentence encoders also provides opportunities to improve cross-lingual transfer learning by sharing word embeddings across languages .

In future work we will apply this model to many more languages and domains besides newswire text .

We would also like to avoid the entity detection problem by using a deep architecture to both identify entity mentions and identify relations between them .


We propose UDP , the first training-free parser for Universal Dependencies ( UD ) .

Our algorithm is based on PageRank and a small set of head attachment rules .

It features two-step decoding to guarantee that function words are attached as leaf nodes .

The parser requires no training , and it is competitive with a delexicalized transfer system .

UDP offers a linguistically sound unsupervised alternative to cross-lingual parsing for UD , which can be used as a baseline for such systems .

The parser has very few parameters and is distinctly robust to domain change across languages .

Grammar induction and unsupervised dependency parsing are active fields of research in natural language processing ( Klein and Manning , 2004 ; Gelling et al . , 2012 ) .

However , many data-driven approaches struggle with learning relations that match the conventions of the test data , e.g. , Klein and Manning reported the tendency of their DMV parser to make determiners the heads of German nouns , which would not be an error if the test data used a DP analysis ( Abney , 1987 ) .

Even supervised transfer approaches ( McDonald et al . , 2011 ) suffer from target adaptation problems when facing word order differences .

The Universal Dependencies ( UD ) project ( Nivre et al . , 2015 ; Nivre et al . , 2016 ) offers a dependency formalism that aims at providing a consistent representation across languages , while enforcing a few hard constraints .

The arrival of such treebanks , expanded and improved on a regular basis , provides a new milestone for crosslingual dependency parsing research ( McDonald et al . , 2013 ) .

Furthermore , given that UD rests on a series of simple principles like the primacy of lexical heads , cf .

Johannsen et al . ( 2015 ) for more details , we expect that such a formalism lends itself more naturally to a simple and linguistically sound rulebased approach to cross-lingual parsing .

In this paper we present such an approach .

Our system is a dependency parser that requires no training , and relies solely on explicit part-ofspeech ( POS ) constraints that UD imposes .

In particular , UD prescribes that trees are single-rooted , and that function words like adpositions , auxiliaries , and determiners are always dependents of content words , while other formalisms might treat them as heads ( De Marneffe et al . , 2014 ) .

We ascribe our work to the viewpoints of Bender ( 2009 ) about the incorporation of linguistic knowledge in language-independent systems .

We have presented UDP , an unsupervised dependency parser for Universal Dependencies ( UD ) that makes use of personalized PageRank and a small set of head-dependent rules .

The parser requires no training data and estimates adposition direction directly from the input .

We achieve competitive performance on all but two UD languages , and even beat a multi-source delexicalized parser ( MSD ) on Hindi .

We evaluated the parser on three POS setups and across domains .

Our results show that UDP is less affected by deteriorating POS tags than MSD , and is more resilient to domain changes .

Given how much of the overall dependency structure can be explained by this fairly system , we propose UDP as an additional UD parsing baseline .

The parser , the in-house annotated test sets , and the domain data splits are made freely available .

6UD is a running project , and the guidelines are bound to evolve overtime .

Indeed , the UD 2.0 guidelines have been recently released .

UDP can be augmented with edge labeling for some deterministic labels like case or det .

Some further constrains can be incorporated in UDP .

Moreover , the parser makes no special treatment of multiword expression that would require a lexicon , coordinations or proper names .

All these three kinds of structures have a flat tree where all words depend on the leftmost one .

While coordination attachment is a classical problem in parsing and out of the scope of our work , a proper name sequence can be straightforwardly identified from the partof-speech tags , and it falls thus in the area of structures predictable using simple heuristics .

Moreover , our use of PageRank could be expanded to directly score the potential dependency edges instead of words , e.g. , by means of edge reification .


We study a conversational reasoning model that strategically traverses through a largescale common fact knowledge graph ( KG ) to introduce engaging and contextually diverse entities and attributes .

For this study , we collect a new Open-ended Dialog ↔ KG parallel corpus called OpenDialKG , where each utterance from 15 K human-to-human roleplaying dialogs is manually annotated with ground-truth reference to corresponding entities and paths from a large-scale KG with 1M+ facts .

We then propose the DialKG Walker model that learns the symbolic transitions of dialog contexts as structured traversals over KG , and predicts natural entities to introduce given previous dialog contexts via a novel domain-agnostic , attention-based graph path decoder .

Automatic and human evaluations show that our model can retrieve more natural and human-like responses than the state-ofthe-art baselines or rule-based models , in both in-domain and cross-domain tasks .

The proposed model also generates a KG walk path for each entity retrieved , providing a natural way to explain conversational reasoning .

The key element of an open-ended dialog system is its ability to understand conversational contexts and to respond naturally by introducing relevant entities and attributes , which often leads to increased engagement and coherent interactions ( Chen et al . , 2018 ) .

While a large-scale knowledge graph ( KG ) includes vast knowledge of all the related entities connected via one or more factual connections from conversational contexts , the core challenge is in the domain-agnostic and scalable prediction of a small subset from those reachable entities that follows natural conceptual threads that can keep conversations engaging and meaningful .

Hence , we study a data-driven reasoning model topical jumps across open-ended multi-turn dialogs are annotated and grounded with a large-scale commonfact KG .

To generate a KG entity response at each dialog turn , the model learns walkable paths within KG that lead to engaging and natural topics or entities given dialog context , while pruning non-ideal ( albeit factually correct ) KG paths among 1M+ candidate facts .

that map dialog transitions with KG paths , aimed at identifying a subset of ideal entities to mention as a response to previous dialog contexts .

Figure 1 illustrates a motivating dialog example between two conversation participants , which spans multiple related KG entities from a starting seed entity The Catcher in the Rye .

Specifically , we observe that there exists a small subset of walkable patterns within a KG or a preferred sequence of graph traversal steps which often leads to more engaging entities or attributes than others ( e.g. Literacy Realism , Nathaniel Hawthorne , etc . vs. Catch Me

If You Can , 277 , etc . all connected via one-or multi-hop factual connections ) .

Note also that the walkable degree of each entity varies by dialog contexts and domains , thus making conventional rule-based or entity-toentity learning approaches intractable or not scalable for open-ended dialogs with 1M+ candidate facts .

Therefore , pruning the search space for entities based on dialog contexts and their relationbased walk paths is a crucial step in operating knowledge-augmented dialog systems at scale .

To this end , we propose a new model called DialKG Walker that can learn natural knowledge paths among entities mentioned over dialog contexts , and reason grounded on a large commonsense KG .

Specifically , we propose a novel graph decoder that attends on viable KG paths to predict the most relevant entities from a KG , by associating these paths with the given input contexts : dialog , sentence , and a set of starting KG entities mentioned in the previous turn .

We then build a parallel zeroshot learning model that predicts entities in the KG embeddings space , and ranks candidate entities based on decoded graph path output .

To train the DialKG Walker model with groundtruth reference to KG entities , we collect a new human-to-human multi-turn dialogs dataset ( 91 K utterances across 15 K dialog sessions ) using Par-lAI ( Miller et al . , 2017 ) , where conversation participants play a role either as a user or as an assistant , while annotating their mention of an entity in a large-scale common fact KG .

This new dataset provides a new way for researchers to study how conversational topics could jump across many different entities within multi-turn dialogs , grounded on KG paths that thread all of them .

To the best of our knowledge , our OpenDialKG is the first parallel Dialog ↔ KG corpus where each mention of a KG entity and its factual connection in an openended dialog is fully annotated , allowing for indepth study of symbolic reasoning and natural language conversations .

Note that our approaches are distinct from the previous work on dialog systems in that we completely ground dialogs in a large-scale commonfact KG , allowing for domain-agnostic conversational reasoning in open-ended conversations across various domains and tasks ( e.g. chit-chat , recommendations , etc . )

We therefore perform extensive cross-domain and transfer learning evaluations to demonstrate its flexibility .

See Section 5 for the detailed literature review .

Our contributions are as follows : we propose ( 1 ) a novel attention-based graph decoder that walks an optimal path within a large commonsense KG ( 100 K entities , 1.1 M facts ) to effectively prune unlikely candidate entities , and ( 2 ) a zeroshot learning model that leverages previous sentence , dialog , and KG contexts to re-rank candidates from pruned decoder graph output based on their relevance and path scores , which allows for generalizable and robust classification with a large number of candidate classes .

We present ( 3 ) a new parallel open-ended dialog ↔ KG corpus called OpenDialKG where each mention of an entity in dialog is manually linked with its corresponding ground-truth KG path .

We show that the proposed approaches outperform baselines in both indomain and cross-domain evaluation , demonstrating that the model learns domain-agnostic walking patterns that are generalizable for unseen domains .

We study conversational reasoning grounded on knowledge graphs , and formulate an approach in which the model learns to navigate a largescale , open-ended KG given conversational contexts .

For this study , we collect a newly annotated Dialog ↔ KG parallel corpus of 15 K humanto-human dialogs which includes ground-truth annotation of each dialog turn to its reasoning reference in a large-scale common fact KG .

Our proposed DialKG Walker model improves upon the state-of-the-art knowledge-augmented conversation models by 1 ) a novel attention-based graph decoder that penalizes decoding of unnatural paths which effectively prunes candidate entities and paths from a large search space ( 1.1 M facts ) , 2 ) a zeroshot learning model that predicts its relevance score in the KG embeddings space , combined score of which is used for candidate ranking .

The empirical results from in-domain , cross-domain , and transfer learning evaluation demonstrate the efficacy of the proposed model in domain-agnostic conversational reasoning .


Existing techniques for disambiguating named entities in text mostly focus on Wikipedia as a target catalog of entities .

Yet for many types of entities , such as restaurants and cult movies , relational databases exist that contain far more extensive information than Wikipedia .

This paper introduces a new task , called Open-Database Named-Entity Disambiguation ( Open-DB NED ) , in which a system must be able to resolve named entities to symbols in an arbitrary database , without requiring labeled data for each new database .

We introduce two techniques for Open-DB NED , one based on distant supervision and the other based on domain adaptation .

In experiments on two domains , one with poor coverage by Wikipedia and the other with near-perfect coverage , our Open-DB NED strategies outperform a state-of-the-art Wikipedia NED system by over 25 % in accuracy .

Named-entity disambiguation ( NED ) is the task of linking names mentioned in text with an established catalog of entities ( Bunescu and Pasca , 2006 ; Ratinov et al . , 2011 ) .

It is a vital first step for semantic understanding of text , such as in grounded semantic parsing ( Kwiatkowski et al . , 2011 ) , as well as for information retrieval tasks like person name search ( Chen and Martin , 2007 ; Mann and Yarowsky , 2003 ) .

NED requires a catalog of symbols , called referents , to which named-entities will be resolved .

Most NED systems today use Wikipedia as the catalog of referents , but exclusive focus on Wikipedia as a target for NED systems has significant drawbacks : despite its breadth , Wikipedia still does not contain all or even most real-world entities mentioned in text .

As one example , it has poor coverage of entities that are mostly important in a small geographical region , such as hotels and restaurants , which are widely discussed on the Web . 57 % of the named-entities in the Text Analysis Conference 's ( TAC ) 2009 entity linking task refer to an entity that does not appear in Wikipedia ( McNamee et al . , 2009 ) .

Wikipedia is clearly a highly valuable resource , but it should not be thought of as the only one .

Instead of relying solely on Wikipedia , we propose a novel approach to NED , which we refer to as Open-DB NED : the task is to resolve an entity to Wikipedia or to any relational database that meets mild conditions about the format of the data , described below .

Leveraging structured , relational data should allow systems to achieve strong accuracy , as with domain-specific or database-specific NED techniques like Hoffart et al . 's NED system for YAGO ( Hoffart et al . , 2011 ) .

And because of the availability of huge numbers of databases on the Web , many for specialized domains , a successful system for this task will cover entities that a Wikipedia NED or database-specific system can not .

We investigate two complementary learning strategies for Open-DB NED , both of which significantly relax the assumptions of traditional NED systems .

The first strategy , a distant supervision approach , uses the relational information in a given database and a large corpus of unlabeled text to learn a database-specific model .

The second strat-egy , a domain adaptation approach , assumes a single source database that has accompanying labeled data .

Classifiers in this setting must learn a model that transfers from the source database to any new database , without requiring new training data for the new database .

Experiments show that both strategies outperform a state-of-the-art Wikipedia NED system by wide margins without requiring any labeled data from the test domain , highlighting the significant advantage of having domain-specific relational data .

The next section contrasts Open-DB NED with previous work .

Sections 4 and 5 present our distant supervision strategy and domain-adaptation strategy , respectively .

Section 6 introduces a technique that is a hybrid of the two learning strategies .

Section 7 describes our experiments , and Section 8 concludes .

This paper introduces the task of Open-DB Named Entity Disambiguation , and presents two distinct strategies for solving this task .

Experiments indicate that a mixture of the two strategies significantly outperforms a state-of-the-art Wikipedia NED system , on a dataset where Wikipedia has good coverage and on another dataset where Wikipedia has poor coverage .

The results indicate that there is a significant benefit to leveraging other sources of knowledge in addition to Wikipedia , and that it is possible to leverage this knowledge without requiring labeled data for each new source .

The initial success of these Open-DB NED approaches indicates that this task is a promising area for future research , including exciting extensions that link large numbers of domainspecific databases to text .


Solving algebraic word problems has recently emerged as an important natural language processing task .

To solve algebraic word problems , recent studies suggested neural models that generate solution equations by using ' Op ( operator / operand ) ' tokens as a unit of input / output .

However , such a neural model suffered two issues : expression fragmentation and operand-context separation .

To address each of these two issues , we propose a pure neural model , Expression-Pointer Transformer ( EPT ) , which uses ( 1 ) ' Expression ' token and ( 2 ) operand-context pointers when generating solution equations .

The performance of the EPT model is tested on three datasets : ALG514 , DRAW-1 K , and MAWPS .

Compared to the state-of-the-art ( SoTA ) models , the EPT model achieved a comparable performance accuracy in each of the three datasets ; 81.3 % on ALG514 , 59.5 % on DRAW-1 K , and 84.5 % on MAWPS .

The contribution of this paper is two-fold ; ( 1 ) We propose a pure neural model , EPT , which can address the expression fragmentation and the operandcontext separation .

( 2 ) The fully automatic EPT model , which does not use hand-crafted features , yields comparable performance to existing models using hand-crafted features , and achieves better performance than existing pure neural models by at most 40 % .

Solving algebraic word problems has recently become an important research task in that automatically generating solution equations requires understanding natural language .

Table 1 shows a sample algebraic word problem , along with corresponding solution equations that are used to generate answers for the problem .

To solve such problems with deep learning technology , researchers recently suggested neural models that generate solution equations automatically ( Huang Problem One number is eight more than twice another and their sum is 20 .

+ x 1 = 20 Answers ( 16 , 4 ) et al . , 2018 ; Amini et al . , 2019 ; Chiang and Chen , 2019 ; Wang et al . , 2019 ) .

However , suggested neural models showed a fairly large performance gap compared to existing state-of-the-art models based on hand-crafted features in popular algebraic word problem datasets , such as ALG514 ( 44.5 % for pure neural model vs. 83.0 % for using hand-crafted features ) ( Huang et al . , 2018 ; Upadhyay and Chang , 2016 ) .

To address the large performance gap in this study , we propose a larger unit of input / output ( I / O ) token called " Expressions " for a pure neural model .

Figure 1 illustrates conventionally used " Op ( operator / operands ) "

versus our newly proposed " Expression " token .

To improve the performance of pure neural models that can solve algebraic word problems , we identified two issues that can be addressed using Expression tokens , which are shown in Figure 1 : ( 1 ) expression fragmentation and ( 2 ) operandcontext separation .

First , the expression fragmentation issue is a segmentation of an expression tree , which represents a computational structure of equations that are used to generate a solution .

This issue arises when Op , rather than the whole expression tree , is used as an input / output unit of a problem-solving model .

For example , as shown in Figure 1 ( a ) , using Op tokens as an input to a problem-solving model disassembles a tree structure into operators ( " × " ) and operands ( " x 1 " and " 2 " ) .

Meanwhile , we propose using the Figure 1 : Illustration using the word problem in Table 1 for the ( a ) expression fragmentation issue , ( b ) operandcontext separation issue , and ( c ) our solution for these two issues .

" Expression " ( ×(x 1 , 2 ) ) token , which can explicitly capture a tree structure as a whole , as shown in Figure 1 ( c ) .

The second issue of operand-context separation is the disconnection between an operand and a number that is associated with the operand .

This issue arises when a problem-solving model substitutes a number stated in an algebraic word problem into an abstract symbol for generalization .

As shown in Figure 1 ( b ) , when using an Op token , the number 8 is changed into an abstract symbol ' N 1 ' .

Meanwhile , when using an Expression token , the number 8 is not transformed into a symbol .

Rather a pointer is made to the location where the number 8 occurred in an algebraic word problem .

Therefore , using such an " operand-context pointer " enables a model to access contextual information about the number directly , as shown in Figure 1 ( c ) ; thus , the operand-context separation issue can be addressed .

In this paper , we propose a pure neural model called Expression-Pointer Transformer ( EPT ) to address the two issues above .

The contribution of this paper is two-fold ;

We propose a pure neural model , Expression-Pointer Transformer ( EPT ) , which can address the expression fragmentation and operandcontext separation issues .

The EPT model is the first pure neural model that showed comparable accuracy to the existing state-of-the-art models , which used handcrafted features .

Compared to the state-ofthe-art pure neural models , the EPT achieves better performance by about 40 % .

In the rest of the paper , we introduce existing approaches to solve algebraic word problems in Section 2 .

Next , Section 3 introduces our proposed model , EPT , and Section 4 reports the experimental settings .

Then in Section 5 , results of two studies are presented .

Section 5.1 presents a performance comparison between EPT and existing SoTA models .

Section 5.2 presents an ablation study examining the effects of Expression tokens and applying operand-context pointers .

Finally , in Section 6 , a conclusion is presented with possible future directions for our work .

In this study , we proposed a neural algebraic word problem solver , Expression-Pointer Transformer ( EPT ) , and examined its characteristics .

We designed EPT to address two issues : expression fragmentation and operand-context separation .

The EPT resolves the expression fragmentation issue by generating ' Expression ' tokens , which simultaneously generate an operator and required operands .

In addition , the EPT resolves the operand-context separation issue by applying operand-context pointers .

Our work is meaningful in that we demonstrated a possibility for alleviating the costly procedure of devising hand-crafted features in the domain of solving algebraic word problems .

As future work , we plan to generalize the EPT to other datasets , including non-English word problems or non-algebraic domains in math , to extend our model .


In this paper I will report the result of a quantitative analysis of the dynamics of the constituent elements of Japanese terminology .

In Japanese technical terms , the linguistic contribution of morphemes greatly differ according to their types of origin .

To analyse this aspect , a quantitative method is applied , which can properly characterise the dynamic nature of morphemes in terminology on the basis of a small sample .

In computational linguistics , the interest in terminological applications such as automatic term extraction is growing , and many studies use the quantitative information ( cf . Kageura & Umino , 1996 ) .

However , the basic quantitative nature of terminological structure , which is essential for terminological theory and applications , has not yet been exploited .

The static quantitative descriptions are not sufficient , as there are terms which do not appear in the sample .

So it is crucial to establish some models , by which the terminological structure beyond the sample size can be properly described .

In Japanese terminology , the roles of morphemes are different according to their types of origin , i.e. the morphemes borrowed mainly from Western languages ( borrowed morphemes ) and the native morphemes including Chineseorigined morphemes which are the majority .

There are some quantitative studies ( Ishii , 1987 ; Nomura & Ishii , 1989 ) , but they only treat the static nature of the sample .

Located in the intersection of these two backgrounds , the aim of the present study is twofold , i.e. ( 1 ) to introduce a quantitative framework in which the dynamic nature of terminology can be described , and to examine its theoretical validity , and ( 2 ) to describe the quantitative dynamics of morphemes as a ' mass ' in Japanese terminology , with reference to the types of origin .

Using the LNRE models , we have succesfully analysed the dynamic nature of the morphemes in Japanese terminology .

As the majority of the terminological data is located in the LNRE zone , it is important to use the statistical framework which allows for the LNRE characteristics .

The LNRE models give the suitable means .

We are currently extending our research to integrating the quantitative nature of morphological distributions to the qualitative mode ] of term formation , by taking into account the po-sitional and combinatorial nature of morphemes and the distributions of term length .


We study the problem of leveraging the syntactic structure of text to enhance pre-trained models such as BERT and RoBERTa .

Existing methods utilize syntax of text either in the pre-training stage or in the fine-tuning stage , so that they suffer from discrepancy between the two stages .

Such a problem would lead to the necessity of having human-annotated syntactic information , which limits the application of existing methods to broader scenarios .

To address this , we present a model that utilizes the syntax of text in both pre-training and fine-tuning stages .

Our model is based on Transformer with a syntax-aware attention layer that considers the dependency tree of the text .

We further introduce a new pre-training task of predicting the syntactic distance among tokens in the dependency tree .

We evaluate the model on three downstream tasks , including relation classification , entity typing , and question answering .

Results show that our model achieves state-of-the-art performance on six public benchmark datasets .

First , we demonstrate that infusing automatically produced syntax of text improves pre-trained models .

Second , global syntactic distances among tokens bring larger performance gains compared to local head relations between contiguous tokens .

Work is done during internship at Microsoft .

For questions , please contact D. Tang and Z. Xu .

Pre-trained models such as BERT ( Devlin et al . , 2019 ) , GPT ( Radford et al . , 2018 ) , and RoBERTa ( Liu et al . , 2019 ) have advanced the state-of-the-art performances of various natural language processing tasks .

The successful recipe is that a model is first pre-trained on a huge volume of unsupervised data with self-supervised objectives , and then is fine-tuned on supervised data with the same data scheme .

Dominant pre-trained models represent a text as a sequence of tokens2 .

The merits are that such basic text representations are available from vast amounts of unsupervised data , and that models pre-trained and fine-tuned with the same paradigm usually achieve good accuracy in practice ( Guu et al . , 2020 ) .

However , an evident limitation of these methods is that richer syntactic structure of text is ignored .

In this paper , we seek to enhance pre-trained models with syntax of text .

Related studies attempt to inject syntax information either only in the finetuning stage ( Nguyen et al . , 2020 ; Sachan et al . , 2020 ) , or only in the pre-training stage ( Wang et al . , 2020 ) , which results in discrepancies .

When only fusing syntax information in the fine-tuning phase , Sachan et al . ( 2020 ) finds that there is no performance boost unless high quality human-annotated dependency parses are available .

However , this requirement would limit the application of the model to broader scenarios where human-annotated dependency information is not available .

To address this , we conduct a large-scale study on injecting automatically produced syntax of text in both the pre-training and fine-tuning stages .

We construct a pre-training dataset by applying an offthe-shelf dependency parser ( Qi et al . , 2020 ) to one billion sentences from common crawl news .

With these data , we introduce a syntax-aware pretraining task , called dependency distance prediction , which predicts the syntactic distance between tokens in the dependency structure .

Compared with the pre-training task of dependency head prediction ( Wang et al . , 2020 ) that only captures local syntactic relations among words , dependency distance prediction leverages global syntax of the text .

In addition , we developed a syntax-aware attention layer , which can be conveniently integrated into Transformer ( Vaswani et al . , 2017 ) to allow tokens to selectively attend to contextual tokens based on their syntactic distance in the dependency structure .

We conduct experiments on entity typing , question answering and relation classification on six benchmark datasets .

Experimental results show that our method achieves state-of-the-art performance on all six datasets .

Further analysis shows that our model can indicate the importance of syntactic information on downstream tasks , and that the newly introduced dependency distance prediction task could capture the global syntax of the text , performs better than dependency head prediction .

In addition , compared with experimental results of injecting syntax information in either the pre-training or fine-tuning stage , injecting syntax information in both stages achieves the best performance .

In summary , the contribution of this paper is threefold .

( 1 ) We demonstrate that infusing automatically produced dependency structures into the pre-trained model shows superior performance over downstream tasks .

( 2 ) We propose a syntax-aware attention layer and a pre-training task for infusing syntactic information into the pre-trained model .

( 3 ) We find that the newly introduced dependency distance prediction task performs better than the dependency head prediction task .

In this paper , we present SEPREM that leverage syntax information to enhance pre-trained models .

To inject syntactic information , we introduce a syntax-aware attention layer and a newly designed pre-training task are proposed .

Experimental results show that our method achieves state-of-theart performance over six datasets .

Further analysis shows that the proposed dependency distance prediction task performs better than dependency head prediction task .