Accurate scoring of syntactic structures such as head-modifier arcs in dependency parsing typically requires rich , highdimensional feature representations .

A small subset of such features is often selected manually .

This is problematic when features lack clear linguistic meaning as in embeddings or when the information is blended across features .

In this paper , we use tensors to map high-dimensional feature vectors into low dimensional representations .

We explicitly maintain the parameters as a low-rank tensor to obtain low dimensional representations of words in their syntactic roles , and to leverage modularity in the tensor for easy training with online algorithms .

Our parser consistently outperforms the Turbo and MST parsers across 14 different languages .

We also obtain the best published UAS results on 5 languages .

Finding an expressive representation of input sentences is crucial for accurate parsing .

Syntactic relations manifest themselves in a broad range of surface indicators , ranging from morphological to lexical , including positional and part-of-speech ( POS ) tagging features .

Traditionally , parsing research has focused on modeling the direct connection between the features and the predicted syntactic relations such as head-modifier ( arc ) relations in dependency parsing .

Even in the case of firstorder parsers , this results in a high-dimensional vector representation of each arc .

Discrete features , and their cross products , can be further complemented with auxiliary information about words participating in an arc , such as continuous vector representations of words .

The exploding dimensionality of rich feature vectors must then be balanced with the difficulty of effectively learning the associated parameters from limited training data .

A predominant way to counter the high dimensionality of features is to manually design or select a meaningful set of feature templates , which are used to generate different types of features ( Mc-Donald et al . , 2005a ; Koo and Collins , 2010 ; Martins et al . , 2013 ) .

Direct manual selection may be problematic for two reasons .

First , features may lack clear linguistic interpretation as in distributional features or continuous vector embeddings of words .

Second , designing a small subset of templates ( and features ) is challenging when the relevant linguistic information is distributed across the features .

For instance , morphological properties are closely tied to part-of-speech tags , which in turn relate to positional features .

Therefore , we may suffer a performance loss if we select only a small subset of the features .

On the other hand , by including all the rich features , we face over-fitting problems .

We depart from this view and leverage highdimensional feature vectors by mapping them into low dimensional representations .

We begin by representing high-dimensional feature vectors as multi-way cross-products of smaller feature vectors that represent words and their syntactic relations ( arcs ) .

The associated parameters are viewed as a tensor ( multi-way array ) of low rank , and optimized for parsing performance .

By explicitly representing the tensor in a low-rank form , we have direct control over the effective dimensionality of the set of parameters .

We obtain role-dependent low-dimensional representations for words ( head , modifier ) that are specifically tailored for parsing accuracy , and use standard online algorithms for optimizing the low-rank tensor components .

The overall approach has clear linguistic and computational advantages :

• Our low dimensional embeddings are tailored to the syntactic context of words ( head , modifier ) .

This low dimensional syntactic abstraction can be thought of as a proxy to manually constructed POS tags .

• By automatically selecting a small number of dimensions useful for parsing , we can leverage a wide array of ( correlated ) features .

Unlike parsers such as MST , we can easily benefit from auxiliary information ( e.g. , word vectors ) appended as features .

We implement the low-rank factorization model in the context of first-and third-order dependency parsing .

The model was evaluated on 14 languages , using dependency data from CoNLL 2008 and CoNLL 2006 .

We compare our results against the MST ( McDonald et al . , 2005a ) and Turbo ( Martins et al . , 2013 ) parsers .

The low-rank parser achieves average performance of 89.08 % across 14 languages , compared to 88.73 % for the Turbo parser , and 87.19 % for MST .

The power of the low-rank model becomes evident in the absence of any part-of-speech tags .

For instance , on the English dataset , the low-rank model trained without POS tags achieves 90.49 % on first-order parsing , while the baseline gets 86.70 % if trained under the same conditions , and 90.58 % if trained with 12 core POS tags .

Finally , we demonstrate that the model can successfully leverage word vector representations , in contrast to the baselines .

Accurate scoring of syntactic structures such as head-modifier arcs in dependency parsing typically requires rich , high-dimensional feature representations .

We introduce a low-rank factorization method that enables to map high dimensional feature vectors into low dimensional representations .

Our method maintains the parameters as a low-rank tensor to obtain low dimensional representations of words in their syntactic roles , and to leverage modularity in the tensor for easy training with online algorithms .

We implement the approach on first-order to third-order dependency parsing .

Our parser outperforms the Turbo and MST parsers across 14 languages .

Future work involves extending the tensor component to capture higher-order structures .

In particular , we would consider second-order structures such as grandparent-head-modifier by increasing the dimensionality of the tensor .

This tensor will accordingly be a four or five-way array .

The online update algorithm remains applicable since each dimension is optimized in an alternating fashion .

To obtain a rich representation for parsing, manually designed templates have been used which can cause a lack of clear linguistic interpretation or design difficulties.


This article describes a linguistically informed method for integrating phrasal verbs into statistical machine translation ( SMT ) systems .

In a case study involving English to Bulgarian SMT , we show that our method does not only improve translation quality but also outperforms similar methods previously applied to the same task .

We attribute this to the fact that , in contrast to previous work on the subject , we employ detailed linguistic information .

We found out that features which describe phrasal verbs as idiomatic or compositional contribute most to the better translation quality achieved by our method .

Phrasal verbs are a type of multiword expressions ( MWEs ) and as such , their meaning is not derivable , or is only partially derivable , from the semantics of their lexemes .

This , together with the high frequency of MWEs in every day communication ( see Jackendoff ( 1997 ) ) , calls for a special treatment of such expressions in natural language processing ( NLP ) applications .

Here , we concentrate on statistical machine translation ( SMT ) where the word-to-word translation of MWEs often results in wrong translations ( Piao et al . , 2005 ) .

Previous work has shown that the application of dedicated methods to identify MWEs and then integrate them in some way into the SMT process often improves translation quality .

Generally , automatically extracted lexicons of MWEs are employed in the identification step .

Further , various integration strategies have been proposed .

The so called static strategy suggests training the SMT system on corpora in which each MWE is treated as a single unit , e.g. call off .

This improves SMT indirectly by improving the alignment between source and target sentences in the training data .

Various versions of this strategy are applied in Lambert and Banchs ( 2005 ) , Carpuat and Diab ( 2010 ) , and Simova and Kordoni ( 2013 ) .

In all cases there is some improvement in translation quality , caused mainly by the better treatment of separable PVs , such as in turn the light on .

Another strategy , which is referred to as dynamic , is to modify directly the SMT system .

Ren et al . ( 2009 ) , for example , treat bilingual MWEs pairs as parallel sentences which are then added to training data and subsequently aligned with GIZA++ ( Och and Ney , 2003 ) .

Other approaches perform feature mining and modify directly the automatically extracted translation table .

Ren et al . ( 2009 ) and Simova and Kordoni ( 2013 ) employ Moses1 to build and train phrase-based SMT systems and then , in addition to the standard phrasal translational probabilities , they add a binary feature which indicates whether an MWE is present in a given source phrase or not .

Carpuat and Diab ( 2010 ) employ the same approach but the additional feature indicates the number of MWEs in each phrase .

All studies report improvements over a baseline system with no MWE knowledge but these improvements are comparable to those achieved by static methods .

In this article , we further improve the dynamic strategy by adding features which , unlike all previous work , also encode some of the linguistic properties of MWEs .

Since it is their peculiar linguistic nature that makes those expressions problematic for SMT , it is our thesis that providing more linguistic information to the translation process will improve it .

In particular , we concentrate on a specific type of MWEs , namely phrasal verbs ( PVs ) .

We add 4 binary features to the translation table which indicate not only the presence of a PV but also its transitivity , separability , and idiomaticity .

We found that PVs are very suitable for this study since we can easily extract the necessary informa-tion from various language resources .

To prove our claim , we perform a case study with an English to Bulgarian SMT system .

Bulgarian lacks PVs in the same form they appear in English .

It is often the case that an English PV is translated to a single Bulgarian verb .

Such manyto-one mappings cause the so called translation asymmetries which make the translation of PVs very problematic .

We perform automated and manual evaluations with a number of feature combinations which show that the addition of all 4 features proposed above improves translation quality significantly .

Moreover , our method outperforms static and dynamic methods previously applied to the same test data .

A notable increase in performance is observed for separable PVs where the verb and the particle(s ) were not adjacent in the input English sentence as well as for idiomatic PVs .

This clearly demonstrates the importance of linguistic information for the proper treatment of PVs in SMT .

We would like to point out that we view the work presented here as a preliminary study towards a more general linguistically informed method for handling similar types of translation asymmetries .

The experiments with a single phenomenon , namely PVs , serve as a case study the purpose of which is to demonstrate the validity of our approach and the crucial role of properly integrated linguistic information into SMT .

Our work , however , can be immediately extended to other phenomena , such as collocations and noun compounds .

The remainder of the paper is organised as follows .

Section 2 describes the asymmetries caused by PVs in English to Bulgarian translation .

Section 3 provides details about the resources involved in the experiments .

Section 4 describes our method and the experimental setup .

Section 5 presents the results and discusses the improvements in translation quality achieved by the method .

In this article , we showed that the addition of linguistically informative features to a phrase-based SMT model improves the translation quality of a particular type of MWEs , namely phrasal verbs .

In a case study involving SMT from English to Bulgarian , we showed that adding features which encode not only the presence of a PV in a given phrase but also its transitiveness , separability , and idiomaticity led to better translation quality compared to previous work which employs both static and dynamic strategies .

In future research , we will extend our method to other language pairs which exhibit the same type of translation asymmetries when it comes to PVs .

Such language pairs include , among others , English-Spanish and English-Portuguese .

Further , we will apply our linguistically informed method to other phenomena which cause similar issues for SMT .

Immediate candidate phenomena include other types of MWEs , collocations , and noun compounds .

When it comes to MWEs , we will pay special attention to the compositionality aspect since it seems to have contributed most to the good performance achieve by our method in the study presented here .

Although multiword expressions frequently appear in daily communications, statistical machine translation models often produce wrong translations calling for special treatments of such expressions.

statistical machine translation
treatment


Natural-language prompts have recently been used to coax pretrained language models into performing other AI tasks , using a fill-in-theblank paradigm ( Petroni et al . , 2019 ) or a few-shot extrapolation paradigm ( Brown et al . , 2020 ) .

For example , language models retain factual knowledge from their training corpora that can be extracted by asking them to " fill in the blank " in a sentential prompt .

However , where does this prompt come from ?

We explore the idea of learning prompts by gradient descent-either fine-tuning prompts taken from previous work , or starting from random initialization .

Our prompts consist of " soft words , " i.e. , continuous vectors that are not necessarily word type embeddings from the language model .

Furthermore , for each task , we optimize a mixture of prompts , learning which prompts are most effective and how to ensemble them .

Across multiple English LMs and tasks , our approach hugely outperforms previous methods , showing that the implicit factual knowledge in language models was previously underestimated .

Moreover , this knowledge is cheap to elicit : random initialization is nearly as good as informed initialization .

Pretrained language models , such as ELMo ( Peters et al . , 2018 ) , BERT ( Devlin et al . , 2019 ) , and BART ( Lewis et al . , 2020a ) , have proved to provide useful representations for other NLP tasks .

Recently , Petroni et al . ( 2019 ) and Jiang et al . ( 2020 ) demonstrated that language models ( LMs ) also contain factual and commonsense knowledge that can be elicited with a prompt .

For example , to query the date-of- , " where we have filled the first blank with " Mozart , " and ask a cloze language model to fill in the second blank .

The prompts used by Petroni et al . ( 2019 ) are manually created , while Jiang et al . ( 2020 ) use mining and paraphrasing based methods to automatically augment the prompt sets .

Finding out what young children know is difficult because they can be very sensitive to the form of the question ( Donaldson , 1978 ) .

Opinion polling is also sensitive to question design ( Broughton , 1995 ) .

We observe that when we are querying an LM rather than a human , we have the opportunity to tune prompts using gradient descent-the workhorse of modern NLP-so that they better elicit the desired type of knowledge .

A neural LM sees the prompt as a sequence of continuous word vectors ( Baroni et al . , 2014 ) .

We tune in this continuous space , relaxing the constraint that the vectors be the embeddings of actual English words .

Allowing " soft prompts " consisting of " soft words " is not only convenient for optimization , but is also more expressive .

Soft prompts can emphasize particular words ( by lengthening their vectors ) or particular dimensions of those words .

They can also adjust words that are misleading , ambiguous , or overly specific .

Consider the following prompt for the relation date-of-death :

This prompt may work for the male singer Cab Calloway , but if we want it to also work for the female painter Mary Cassatt , it might help to soften " performed " and " his " so that they do not insist on the wrong occupation and gender , and perhaps to soften " until " into a weaker connective ( as Cassatt was in fact too blind to paint in her final years ) .

Another way to bridge between these cases is to have one prompt using " performed " and another using " painted . "

In general , there may be many varied lexical patterns that signal a particular relation , and having more patterns will get better coverage ( Hearst , 1992 ; Riloff and Jones , 1999 ) .

We therefore propose to learn a mixture of soft prompts .

We test the idea on several cloze language models , training prompts to complete factual and com-mon sense relations from 3 datasets .

Comparing on held-out examples , our method dramatically outperforms previous work , even when initialized randomly .

So when regarded as approximate knowledge bases , language models know more than we realized .

We just had to find the right ways to ask .

Well-crafted natural language prompts are a powerful way to extract information from pretrained language models .

In the case of cloze prompts used to query BERT and BART models for single-word answers , we have demonstrated startlingly large and consistent improvements from rapidly learning prompts that work-even though the resulting " soft prompts " are no longer natural language .

Our code and data are available at https:// github.com / hiaoxui / soft-prompts .

How about few-shot prediction with pretrained generative LMs ?

Here , Lewis et al . ( 2020b ) show how to assemble a natural language prompt for input x from relevant input-output pairs ( x i , y i ) selected by a trained retrieval model .

Allowing fine-tuned soft string pairs is an intriguing future possibility for improving such methods without needing to fine-tune the entire language model .

When prompting language models as a knowledge base, existing methods use actual sentences as queries while models may be sensitive to the form of questions.


A pun is a form of wordplay for an intended humorous or rhetorical effect , where a word suggests two or more meanings by exploiting polysemy ( homographic pun ) or phonological similarity to another word ( heterographic pun ) .

This paper presents an approach that addresses pun detection and pun location jointly from a sequence labeling perspective .

We employ a new tagging scheme such that the model is capable of performing such a joint task , where useful structural information can be properly captured .

We show that our proposed model is effective in handling both homographic and heterographic puns .

Empirical results on the benchmark datasets demonstrate that our approach can achieve new state-ofthe-art results .

There exists a class of language construction known as pun in natural language texts and utterances , where a certain word or other lexical items are used to exploit two or more separate meanings .

It has been shown that understanding of puns is an important research question with various real-world applications , such as human-computer interaction ( Morkes et al . , 1999 ; Hempelmann , 2008 ) and machine translation ( Schröter , 2005 ) .

Recently , many researchers show their interests in studying puns , like detecting pun sentences ( Vadehra , 2017 ) , locating puns in the text ( Cai et al . , 2018 ) , interpreting pun sentences ( Sevgili et al . , 2017 ) and generating sentences containing puns ( Ritchie , 2005 ; Hong and Ong , 2009 ; Yu et al . , 2018 ) .

A pun is a wordplay in which a certain word suggests two or more meanings by exploiting polysemy , homonymy , or phonological similarity to another sign , for an intended humorous or rhetorical effect .

Puns can be generally categorized into two groups , namely hetero-graphic puns ( where the pun and its latent target are phonologically similar ) and homographic puns ( where the two meanings of the pun reflect its two distinct senses ) ( Miller et al . , 2017 ) .

( 1 ) When the church bought gas for their annual barbecue , proceeds went from the sacred to the propane .

The first punning joke exploits the sound similarity between the word " propane " and the latent target " profane " , which can be categorized into the group of heterographic puns .

Another categorization of English puns is homographic pun , exemplified by the second instance leveraging distinct senses of the word " gut " .

Pun detection is the task of detecting whether there is a pun residing in the given text .

The goal of pun location is to find the exact word appearing in the text that implies more than one meanings .

Most previous work addresses such two tasks separately and develop separate systems ( Pramanick and Das , 2017 ; Sevgili et al . , 2017 ) .

Typically , a system for pun detection is built to make a binary prediction on whether a sentence contains a pun or not , where all instances ( with or without puns ) are taken into account during training .

For the task of pun location , a separate system is used to make a single prediction as to which word in the given sentence in the text that trigger more than one semantic interpretations of the text , where the training data involves only sentences that contain a pun .

Therefore , if one is interested in solving both problems at the same time , a pipeline approach that performs pun detection followed by pun location can be used .

Compared to the pipeline methods , joint learning has been shown effective ( Katiyar and Cardie , 2016 ; Peng et al . , 2018 ) since it is able to re-duce error propagation and allows information exchange between tasks which is potentially beneficial to all the tasks .

In this work , we demonstrate that the detection and location of puns can be jointly addressed by a single model .

The pun detection and location tasks can be combined as a sequence labeling problem , which allows us to jointly detect and locate a pun in a sentence by assigning each word a tag .

Since each context contains a maximum of one pun ( Miller et al . , 2017 ) , we design a novel tagging scheme to capture this structural constraint .

Statistics on the corpora also show that a pun tends to appear in the second half of a context .

To capture such a structural property , we also incorporate word position knowledge into our structured prediction model .

Experiments on the benchmark datasets show that detection and location tasks can reinforce each other , leading to new state-of-the-art performance on these two tasks .

To the best of our knowledge , this is the first work that performs joint detection and location of English puns by using a sequence labeling approach.1 2 Approach

In this paper , we propose to perform pun detection and location tasks in a joint manner from a sequence labeling perspective .

We observe that each text in our corpora contains a maximum of one pun .

Hence , we design a novel tagging scheme to incorporate such a constraint .

Such a scheme guarantees that there is a maximum of one word that will be tagged as a pun during the testing phase .

We also found the interesting structural property such as the fact that most puns tend to appear at the second half of the sentences can be helpful for such a task , but was not explored in previous works .

Furthermore , unlike many previous approaches , our approach , though simple , is generally applicable to both heterographic and homographic puns .

Empirical results on the benchmark datasets prove the effectiveness of the proposed approach that the two tasks of pun detection and location can be addressed by a single model from a sequence labeling perspective .

Future research includes the investigations on how to make use of richer semantic and linguistic information for detection and location of puns .

Research on puns for other languages such as Chinese is still under-explored , which could also be an interesting direction for our future studies .

Existing pun detection works take a pipeline approach that is composed of detection and location while joint learning approaches perform well on other NLP tasks.