{
  "name": "768.pdf",
  "metadata": {
    "source": "docling-granite",
    "title": "Detecting Lexical Entailment in Context",
    "authors": [],
    "emails": [],
    "sections": [
      {
        "heading": "Anonymous ACL submission",
        "text": ""
      },
      {
        "heading": "1 Introduction",
        "text": "Many NLP applications require detecting relations between word meanings beyond synonymy and paraphrasing. For instance, given 'Carlsen plays chess. ' and the question 'Which game does Carlsen play?' , successfully answering the question requires knowing that chess is a kind of game , or more generally, that chess entails game .\n\nWhile prior work has defined lexical entailment as a relation between word types (Turney and Mohammad, 2013), we argue entailment relations are better defined when illustrating word meaning with an example in context. Ignoring context is problematic since entailment might hold between some senses of the words, but not others. Consider the word game in two distinct contexts:\n\nThe championship game was played in NYC.\n\nThe hunters were interested in the big game .\n\nGiven the sentence, Carlsen is the world chess champion , chess = ⇒ game as used in the first context, while chess = ⇒ game in the second context.\n\n̸\n\nIn this paper, we investigate how to represent and compare the meaning of words in context for lexical entailment. Since distributional representations for word types have proved useful to detect lexical entailment out of context in supervised settings (Baroni et al., 2012; Roller et al., 2014; Turney and Mohammad, 2013), we propose to transform context-agnostic word type representations into contextualized representations that highlight salient properties of the context (Section 3), and use these contextualized representations with a range of semantic similarity features (Section 4) to successfully detect entailment.\n\nAs we will see, these context representations significantly improve performance over contextagnostic baselines not only in English, but also between English and French words (Section 7) on two novel datasets (Section 5). We also show that our features are sensitive to word sense changes indicated by context, and adequately capture the direction of entailment relation (Section 8). Moreover, we establish a new state-of-the-art on an existing dataset that captures a broader range of semantic relations in context (Shwartz and Dagan, 2015), and show that the proposed features, induced solely from large amounts of raw text, yield systems that perform as well, or better than existing systems that require additional human annotation."
      },
      {
        "heading": "2 Defining Lexical Entailment in Context",
        "text": "We frame the task of lexical entailment in context as a binary classification task on examples consisting of a 4-tuple ( w l , w r , c l , c r ) , where w l and w r are two words, and c l and c r are sentences which illustrate each word usage. The example is treated as positive if w l = ⇒ w r , given the meaning of each word exemplified by the contexts, and negative otherwise. Table 1 provides examples.\n\nWords ( w l , w r ) Exemplars ( c l , c r ) Does w l = ⇒ w r ? staff , stick c l = He walked with the help of a wooden staff . c r = The kid had a candied apple on a stick . Yes staff , body c l = The hospital has an excellent nursing staff . c r = The whole body filed out of the auditorium. Yes staff , stick c l = The hospital has an excellent nursing staff . c r = The kid had a candied apple on a stick . No\n\nTable 1: Examples of the context-aware lexical entailment task\n\nWe say that entailment holds if the meaning of w l in the context of c l is more specific than the meaning of w r in the context of c r . The nature of entailment relations captured out-of-context can be broader depending on the test beds considered 1 . Zhitomirsky-Geffet and Dagan (2009) formalize lexical entailment as a substitutional relationship, which encompasses synonymy, hypernymy, some meronymy relations, and also cause-effect relations. However, we limit entailment to the specificity relation in this work to better understand the impact of context.\n\nNote that lexical entailment in context is not textual entailment. Recognizing textual entailment (Dagan et al., 2013) and natural language inference (Bowman et al., 2015) involves detecting entailment relations between sentences, while lexical entailment is a relation between words."
      },
      {
        "heading": "3 Representing Words and their Contexts for Entailment",
        "text": "How can we construct vector representations of the meaning of target words w l and w r in their respective exemplar contexts c l and c r ? We start from existing representations for word types which have proven useful for detecting lexical entailment and other semantic relations out of context (Baroni et al., 2012; Kruszewski and Baroni, 2015; Vylomova et al., 2016; Turney and Mohammad, 2013).\n\nGiven an example ( w l , w r , c l , c r ) , let ⃗ w l and ⃗ w r refer to the context-agnostic representations of w l and w r , and let C l and C r represent the matrices obtained by row-wise stacking of the context- agnostic representations of words in c l and c r respectively.\n\n1 We refer the reader to Turney and Mohammad (2013) and Shwartz et al (2017) for comprehensive surveys of supervised and unsupervised methods for the out-of-context task.\n\nFigure 1: Constructing word-in-context representations for 'bank', in the context 'the river bank'. ⊙ indicates element-wise multiplication."
      },
      {
        "heading": "3.1 Contextualized Word Representations",
        "text": "Following Thater et al. (2011); Erk and Pad´ o (2008), our first approach is to apply a filter to word type representations to highlight the salient dimensions of the exemplar context, emphasizing relevant dimensions of and downplaying unimportant ones. However, while prior work represents context by averaging word vectors, we propose richer representations that better capture the salient geometrical properties of the exemplar context that might get lost by averaging (Figure 1).\n\nFirst, we construct fixed length representations for the contexts c l and c r by running convolutional filters over C l and C r . Specifically, we calculate the column-wise maximum, minimum and the mean over the matrices C l and C r , as done by (Tang et al., 2014) for supervised sentiment classification. This yields three d -dimensional vectors for c l ( ⃗ c l,max , ⃗ c l,min , ⃗ c l,mean ), and three d -dimensional vectors for c r ( ⃗ c r,max , ⃗ c r,min , ⃗ c r,mean ). Computing the maximum and minimum across all vector dimensions captures the exterior surface of the 'instance manifold'\n\n(the volume in embedding space within which all words in the instance reside), while the mean summarizes the density per-dimension within the manifold (Hovy, 2015).\n\nSecond, we transform initial context-agnostic representations for target word types by taking an element-wise product of the word type vectors ( w ⃗ ∗ ) with vectors representing salient dimensions of the exemplar context ( ⃗ c ∗ ,max , ⃗ c ∗ ,min , ⃗ c ∗ ,mean ) where ∗ ∈ { l, r } . This yields three d -dimensional vectors for w l ( ⃗ w l,max , ⃗ w l,min , ⃗ w l,mean ), and three for w r ( ⃗ w r,max , ⃗ w r,min , ⃗ w r,mean ). We refer to our final word-in-context representations for w l and w r as ⃗ w l,mask and ⃗ w r,mask respectively, where ⃗ w l,mask is the concatenation of ⃗ w l,max , ⃗ w l,min , ⃗ w l,mean , and ⃗ w r,mask is also similarly constructed."
      },
      {
        "heading": "3.2 A Shared Vector Space for Words and Contexts: Context2vec",
        "text": "An alternative approach to contextualizing word representations is to directly compare the representations of words with representations of contexts. In a single language, this can be done using Context2Vec (Melamud et al., 2016), a neural model that, given a target word and its sentential context, embeds both the word and the context in the same low-dimensional space, with the objective of having the context predict the target word via a log linear model. This model approaches the state-of-the-art on lexical substitution, sentence completion, and supervised word sense disambiguation."
      },
      {
        "heading": "4 Comparing Words and Contexts for Entailment",
        "text": "Given words and context representations described above, how can we predict entailment?"
      },
      {
        "heading": "4.1 Supervised Logistic Regression model",
        "text": "Prior work on lexical entailment out of context suggests that the entailment relationship between words is a learnable function of the concatenation of their individual representations (Baroni et al., 2012; Turney and Mohammad, 2013). After concatenation, they are used as features for a logistic regression (Roller et al., 2014) or an SVM classifier (Baroni et al., 2012; Turney and Mohammad, 2013). We follow the same practice with our context-aware word representations. Intuitively, the classifier learns to weight the importance of each dimension of a word representation to detect entailment. By learning different weights for the same features describing w l or w r , the classifier can detect asymmetric behavior too, as we will see in Section 8.\n\nEquivalence is the same as Entailment is more specific than Negation is the exact opposite of Alternation is mutually exclusive with Other-related is related in some other way to Independent is not related to\n\nTable 2: Semantic relations in CONTEXT-PPDB. Negation and alternation are conflated into a single relation while creating the dataset."
      },
      {
        "heading": "4.2 Similarity Features",
        "text": "We hypothesize that entailment relations hold between related words and introduce similarity features to capture this non-directional relation between words and contexts.\n\nGiven a pair of vectors ( ⃗ l, ⃗ r ), we use three similarity measures: the cosine similarity cosine ( ⃗ l, ⃗ r ) , the dot product ⃗ l · ⃗ r , and the euclidean distance ‖ ⃗ l -⃗ r ‖ . The cosine similarity captures the difference in the two vectors in terms of the angle between then, the euclidean distance measures the difference in magnitude, and the dot product captures both magnitude and angle.\n\nWe apply these measures to three types of representations. Our first set of similarities is intended to directly capture the similarities between contextualized word representations (Section 3.1). We calculate pairwise similarities between ( ⃗ w l,max , ⃗ w r,max ) , ( ⃗ w l,min , ⃗ w r,min ) , ( ⃗ w l,mean , ⃗ w r,mean ) , as well as between the context-agnostic representations ( ⃗ w l , ⃗ w r ) . Second, we add similarities based on Context2Vec representations, i.e. between each pair of ⃗ w l,c 2 v , ⃗ w r,c 2 v , ⃗ c l,c 2 v , and ⃗ c r,c 2 v . Finally, following Shwartz and Dagan (2015), we capture the similarity of the most relevant word to w l in c r , that to w r in c l , as well as between c l and c r . We therefore add the following three similarities, as well as their pairwise products:\n\nmax _ w ∈ c _ r vec w _ l · vec w , max _ w ∈ c _ l vec w _ r · vec w , max _ w ∈ c _ l , w ^ prime ∈ c _ r vec w · vec w ^ prime\n\nFigure 2: Sample dataset creation process based on two synsets of the word study . The green/solid lines indicate positive examples, while the red/dashed lines indicate negative examples"
      },
      {
        "heading": "5 Evaluation Framework",
        "text": "We evaluate our models of lexical entailment in context on three complementary datasets."
      },
      {
        "heading": "5.1 CONTEXT-PPDB",
        "text": "The first dataset, CONTEXT-PPDB, is a finegrained lexical inference dataset created using 375 word pairs from a subset of the English Paraphrase Database (Ganitkevitch et al., 2013; Pavlick et al., 2015b). These word pairs are semiautomatically labeled with semantic relations outof-context. Shwartz and Dagan (2015) augmented them with examples of word usage in context, and re-annotating word pairs given the extra contextual information. The final dataset consists of 3750 words/contexts tuples with a corresponding label from Table 2, one of which is entailment."
      },
      {
        "heading": "5.2 CONTEXT-WN",
        "text": "In addition to CONTEXT-PPDB, we would like a test set for controlled analysis of lexical entailment in context. We would like to directly assess the sensitivity of our models to contexts that signal different word senses, as well as quantify the extent to which our models detect asymmetric entailment relations rather than semantic similarity.\n\nBased on the above criteria, we introduce a large-scale dataset automatically extracted from WordNet (Fellbaum, 1998a). We call this dataset CONTEXT-WN. WordNet groups synonyms into synsets . Most synsets are further accompanied by one or more short sentences illustrating the use of the members of the synset. The idea behind CONTEXT-WN is to use these example sentences as context for the words, and hypernymy relations to draw examples of lexical entailment relations. The entire process starts from a seed list of words W and proceeds as follows (see Figure 2 for an illustration) :\n\nFor each word type w ∈ W with multiple synsets, obtain all synsets S w .\n\nFor each synset i ∈ S w , pick a hypernym synset s i h , with a corresponding word form w i h . Also obtain c i and c i h which are example sentences corresponding to w i and w i h respectively - ( w i , w i h , c i , c i h ) serves as a positive example.\n\nPermute the positive examples to get negative examples. From ( w i , w i h , c i , c i h ) and ( w j , w j h , c j , c j h ), generate negative examples ( w i , w j h , c i , c j h ) and ( w j , w i h , c j , c i h ).\n\nFlip the positive examples to generate more negative examples. From ( w i , w i h , c i , c i h ) generate the negative example ( w i h , w i , c i h , c i ).\n\nWe run this process using the 9000 most frequent words from Wikipedia as W (after filtering the top 1000 as stopwords). This yields a total of 5239 positive examples. We sample an equal number of negative examples from Step 3, with another 5239 examples being generated in Step 4. The final dataset consists of 10478 negative examples.\n\nCONTEXT-WN satisfies our desiderata. The dataset has a very specific focus since we only pick hypernym-hyponym pairs. The negative examples generated in Steps 3 and 4 require discriminating between different word senses and entailment directions. Finally, with over 15000 examples distributed over 6000 word pairs, the dataset is almost five times as large as CONTEXT-PPDB.\n\nWe use a 70/5/25 train/dev/test split as in CONTEXT-PPDB. We also ensure that each set contains different word pairs, to avoid memorization and overfitting (Levy et al., 2015)."
      },
      {
        "heading": "5.3 Crosslingual dataset",
        "text": "Our final dataset takes a cross-lingual view of lexical entailment in context. To create this dataset, we follow the same methodology that was used to create CONTEXT-PPDB, replacing word pairs from PPDB with a set of 100 English-French word pairs labeled as positive out-of-context in the cross-lingual lexical entailment dataset of Vyas and Carpuat (2016). Each word pair ( w l , w r ) consists of an English word and a French word. To add context, we extract 10 sentence pairs ( c l , c r ) per word pair from the Wikipedia in the corresponding language. These thousand examples are then given to annotators on a crowdsourcing platform 2 . For each example ( w l , w r , c l , c r ) , the annotators were asked to judge whether w l = ⇒ w r given the contexts c l and c r . Each example was annotated by 5 workers 3 .\n\nFor each example, we assign the majority judgment as the correct label. 4 or more annotators agree on the label on 540 examples, and we retain only these examples as our test set. This yields a dataset of 374 positive examples and 166 negative examples."
      },
      {
        "heading": "6 Experimental Set-up",
        "text": ""
      },
      {
        "heading": "6.1 Monolingual",
        "text": "All experiments on CONTEXT-PPDB and CONTEXT-WN are with the default train/dev/test splits. We use 50 dimensional off-the-shelf GloVe embeddings (Pennington et al., 2014) to create ⃗ w l , ⃗ w r , C l , and C r . To obtain the Context2Vec representations ⃗ w l,c 2 v , ⃗ w l,c 2 v , ⃗ c l,c 2 v , and ⃗ c r,c 2 v , we use an existing model trained on the ukWaC corpus (Ferraresi et al., 2006) .\n\nWe use Logistic Regression as our classifier for both tasks, to allow for direct comparisons with previous work. We do not tune parameters of the classifier, except for adding class weights in the CONTEXT-WN experiments to account for the unbalanced data. As evaluation metric, we use weighted F1 score to compare against previous results, as well as to account for label imbalance in CONTEXT-WN."
      },
      {
        "heading": "6.2 Crosslingual",
        "text": "We evaluate our models on cross-lingual test set in a transfer setting by training on the training set of CONTEXT-PPDB. To represent ⃗ w l , ⃗ w r , we experiment with two different pre-trained, 200dimensional, bilingual embeddings - BiVec (Luong et al., 2015) and BiCVM (Hermann and Blunsom, 2014), both of which have been shown to be suitable for cross-lingual semantic tasks such as dictionary induction and document classification (Upadhyay et al., 2016). The classifier is again a Logistic Regression classifier with class weights.\n\n2 www.crowdflower.com\n\n3 We will release further details of our annotation task including guidelines and the specific task after the review period."
      },
      {
        "heading": "7 Results",
        "text": ""
      },
      {
        "heading": "7.1 CONTEXT-PPDB",
        "text": "Representation-based features only A context-agnostic baseline (Baroni et al., 2012) based on the concatenation of the word type features for ( w l , w r ) reaches F1 scores of 53.5 using Glove and 54 with Context2Vec (Table 4). The contextualized word representations and similarity features yield the best result and improves performance by 14 points. The similarity features are also effective when used in combination with the Glove or Context2Vec representations, yielding improvements of 12 to 13 points on the baseline. We also note that using ⃗ c l,c 2 v and ⃗ c r,c 2 v to add contextual information performs very poorly (F = ~60), regardless of the word type feature, indicating the superiority of our masked representations.\n\nAdding PPDB-specific features CONTEXTPPDB comes with rich information about word pairs drawn from PPDB (Pavlick et al., 2015a,b). These features include scores for likelihood of context-agnostic entailment labels, distributional similarities, and probabilities of the word pair being paraphrases, among other scores. Shwartz and Dagan (2015) establish a strong baseline of 67 F1 using these features and the most salient word/context similarities described in Section 4.2. Augmenting this system with our proposed context-aware similarities, we observe an improvement of ~2 F1 points. The contextualized representations further gives a ~3 point boost. Overall, our new context aware features help in improving upon the previous state-of-the-art by 4.8 F1 points."
      },
      {
        "heading": "7.2 CONTEXT-WN",
        "text": "We repeat all experiments on CONTEXT-WN (Table 4), except for those that require PPDB-specific features. The trends on CONTEXT-WN are similar to those on CONTEXT-PPDB. Using similarity features with the context-agnostic baseline, improves performance by ~4 F1 points. Adding the context-aware representations, further gives a tiny boost, enabling us to score 5 F1 points more than the baseline. Again, we observe that using ⃗ c l,c 2 v and ⃗ c r,c 2 v does worse than the alternatives.\n\nWord Type Features..7 Context-aware Features.Similarities.All Context-aware Features.Representations.[ ⃗ w l,mask ; ⃗ w r,mask ] Scores.P.0.674 Scores.R.0.694 Scores.F.0.677 [ ⃗ w l ; ⃗ w r ] [ ⃗ w l ; ⃗ w r ] [ ⃗ w l ; ⃗ w r ] [ ⃗ w l ; ⃗ w r ] [ ⃗ w l ; ⃗ w r ] ⃗ w l,c 2 v ; ⃗ w r,c 2 v ] ⃗ w l,c 2 v ; ⃗ w r,c 2 v ] ⃗ w l,c 2 v ; ⃗ w r,c 2 v ] ⃗ w l,c 2 v ; ⃗ w r,c 2 v ] 7 7 All All All 7 7 All All 7 [ ⃗ w l,mask ; ⃗ w r,mask ] 7 [ ⃗ w l,mask ; ⃗ w r,mask ] [ ⃗ c l,c 2 v ; ⃗ c r,c 2 v ] 7 [ ⃗ w l,mask ; ⃗ w r,mask ] 7 [ ⃗ w l,mask ; ⃗ w r,mask ] 0.531 0.539 0.535 0.516 0.642 0.637 0.512 0.521 0.642 0.642 0.638 0.637 0.605 0.604 0.603 [ [ [ 0.533 0.569 0.540 0.518 0.545 0.526 0.655 0.674 0.659 [ 0.652 0.670 0.656 [ ⃗ w l,c 2 v ; ⃗ w r,c 2 v ] All [ ⃗ c l,c 2 v ; ⃗ c r,c 2 v ] 0.601 0.601 0.600 PPDB ( w l ,w r ) Most salient only 7 0.677 0.685 0.670 PPDB ( w l ,w r ) All 7 0.695 0.701 0.692 PPDB ( w l ,w r ) All [ ⃗ w l,mask ; ⃗ w r,mask ] 0.721 0.720 0.718\n\nTable 3: Experimental results on CONTEXT-PPDB. The evaluation metric is weighted F score.\n\nThe absolute value of gains on CONTEXT-WN is smaller than on CONTEXT-PPDB, as can be expected, given that CONTEXT-WN was designed to be challenging (Section 5.2)."
      },
      {
        "heading": "7.3 Cross-lingual dataset",
        "text": "Results on the cross-lingual dataset are similar to monolingual results (Table 5). Context-aware features outperform context-agnostic baselines - similarities boost the score by 3 and 6 points in the BiVec and BiCVM settings respectively. The contextualized word representations, when used instead of the context-agnostic representations, give a boost of almost 3 points, allowing us to score 6 points and 10 points above the context-agnostic baseline in the BiVec and BiCVM cases, respectively. The strength of these results attest to the generalizable nature of our features, which helps capture correspondences beyond a single language."
      },
      {
        "heading": "8 Analysis",
        "text": "The experiments in Section 7 attest to the overall strength of our features. In this section, we aim to further test the assumptions underlying our proposed context representations."
      },
      {
        "heading": "8.1 Sensitivity to context",
        "text": "How sensitive are our models to changes in contexts? To answer this, we focus on the subset of CONTEXT-WN that comprises of positive examples, and the equivalent number of negative examples from Step 3 of the dataset creation. These examples are created by permuting the contexts of the positive examples, and thus, can directly help in answering our question. We analyze the predictions made on this subset by our model using a metric we call 'Macro-F1', defined as the weighted F1 calculated over each ( w l , w r ) word pair, and then averaged over all word pairs.\n\nModels using context-aware features do consistently better on Macro-F1 than those without (Table 4). Interestingly though, the model that uses [ ⃗ c l,c 2 v ; ⃗ c r,c 2 v ] does the best on this metric, indicating that while directly using the Context2Vec representations might not be ideal, these representation do capture some useful contextual knowledge."
      },
      {
        "heading": "8.2 Sensitivity to Entailment Direction",
        "text": "̸\n\nTo quantitatively evaluate our claim of our features learning to discriminate directionality 4, we report results on the subset of CONTEXT-WN that consists of all positive examples, and the equivalent number of flipped negative examples generated in Step 4. We measure directionality by looking at the number of example pairs ( ( w l , w r , c l , c r ) , ( w r , w l , c r , c l ) ) where both examples are correctly labeled, i.e. the former is labeled as = ⇒ and the latter as = ⇒ .\n\nContext agnostic baselines detect direction significantly above chance (Table 4). Adding our context features improves by ~3, with the overall best performing model again scoring highest on this metric. It is reassuring to observe that improved detection of context with our features does not come at the cost of detecting entailment direction. Unlike in the previous section, the model that uses [ ⃗ c l,c 2 v ; ⃗ c r,c 2 v ] is consistently poor, scoring lower than the corresponding context-agnostic baseline.\n\nWord Type Feat.s..Random.7 Context-aware Features.Similarities.Random.All Context-aware Features.Representations.Random.[ ⃗ w l,mask ; ⃗ w r,mask ] Scores.P.0.5.0.700 Scores.R.0.5.0.661 Scores.F.0.5.0.670 Scores.Macro F1.0.5.0.677 Scores.Pairwise Acc..0.25.0.623 [ ⃗ w l ; ⃗ w r ] [ ⃗ w l ; ⃗ w r ] [ ⃗ w l ; ⃗ w r ] [ ⃗ w l ; ⃗ w r ] [ ⃗ w l ; ⃗ w r ] [ ⃗ w l,c 2 v ; ⃗ w r,c 2 v ] [ ⃗ w l,c 2 v ; ⃗ w r,c 2 v ] [ ⃗ w l,c 2 v ; ⃗ w r,c 2 v ] [ ⃗ w l,c 2 v ; ⃗ w r,c 2 v ] [ ⃗ w l,c 2 v ; ⃗ w r,c 2 v ] 7 7 All All All 7 7 All All All 7 [ ⃗ w l,mask ; ⃗ w r,mask ] 7 [ ⃗ w l,mask ; ⃗ w r,mask ] [ ⃗ c l,c 2 v ; ⃗ c r,c 2 v ] 7 [ ⃗ w l,mask ; ⃗ w r,mask ] 7 [ ⃗ w l,mask ; ⃗ w r,mask ] [ ⃗ c l,c 2 v ; ⃗ c r,c 2 v ] 0.653 0.656 0.700 0.704 0.692 0.695 0.694 0.728 0.724 0.713 0.615 0.618 0.663 0.666 0.658 0.655 0.655 0.689 0.685 0.678 0.625 0.628 0.672 0.675 0.667 0.664 0.665 0.697 0.694 0.687 0.594 0.608 0.651 0.673 0.682 0.663 0.668 0.704 0.704 0.708 0.582 0.592 0.597 0.617 0.592 0.667 0.660 0.700 0.691 0.638\n\nTable 4: Results on CONTEXT-WN. Macro-F1 and Pairwise accuracy, are intended to capture contextawareness and directionality-discrimination abilities of our features, repsectively.\n\nWord Type Features..Random Context-aware Features.Similarities.Random Context-aware Features.Representations.Random Scores.P.0.570 Scores.R.0.481 Scores.F.0.501 [ ⃗ w l,bivec ; ⃗ w r,bivec ] [ ⃗ w l,bivec ; ⃗ w r,bivec ] [ ⃗ w l,bivec ; ⃗ w r,bivec ] [ ⃗ w l,bivec ; ⃗ w r,bivec ] 7 7 7 All All All 7 7 All All 7 [ ⃗ w l,mask ; ⃗ w r,mask ] 7 [ ⃗ w l,mask ; ⃗ w r,mask ] [ ⃗ w l,mask ; ⃗ w r,mask ] 7 [ ⃗ w l,mask ; ⃗ w r,mask ] 7 [ ⃗ w l,mask ; ⃗ w r,mask ] [ 0.618 0.618 0.632 0.620 0.650 0.651 0.655 0.681 0.682 0.537 0.537 0.576 0.554 0.607 0.517 0.528 0.574 0.560 0.560 0.595 0.575 0.622 0.530 0.543 0.590 [ ⃗ w l,bicvm ; ⃗ w r,bicvm ] [ ⃗ w l,bicvm ; ⃗ w r,bicvm ] [ ⃗ w l,bicvm ; ⃗ w r,bicvm ] [ ⃗ w l,bicvm ; ⃗ w r,bicvm ] 0.566 0.581 7 All ⃗ w l,mask ; ⃗ w r,mask ] 0.676 0.614 0.629\n\nTable 5: Results on cross-lingual dataset with training on CONTEXT-PPDB using bilingual embeddings."
      },
      {
        "heading": "8.3 Contextualized Masks",
        "text": "We also hypothesized that masked contextualized representations based on the full volume of the context using min and max operations (Section 3.1) better capture salient context dimensions than the more usual vector averaging approach. We test this hypothesis empirically by replacing masked word-in-context representations ⃗ w l,mask and ⃗ w r,mask by two other ways to capture context. In the first method, we use the mean of the contexts ( ⃗ c l,mean , ⃗ c r,mean ) . In the second method, we use ( ⃗ w l,mean , ⃗ w r,mean ) , i.e. the masked representations calculated by using only the mean. On both CONTEXT-PPDB and CONTEXT-WN, our preferred method outperforms the two alternatives by 2-3 F1 points. In fact, on CONTEXT-WN we can see that our method also captures directionality best."
      },
      {
        "heading": "9 Related Work",
        "text": "WordNet and lexical entailment The is-a hierarchy of WordNet (Fellbaum, 1998b) is a prominent source of information for unsupervised detection of hypernymy and entailment (Harabagiu and Moldovan, 1998; Shwartz et al., 2015), as well as a source of various datasets (Baroni and Lenci, 2011; Baroni et al., 2012). The dataset we introduce in this work is inspired by the latter line of work, but instead of just extracting word pairs we also obtain exemplar contexts from WordNet.\n\nDataset. Context aware features.Similarities Context aware features.Representations Scores.P Scores.R Scores.F Scores.Macro F1 Scores.Pairwise Acc. CONTEXT-PPDB All All All [ ⃗ w l,mean ; ⃗ w r,mean ] [ ⃗ c l,mean ; ⃗ c r,mean ] [ ⃗ w l,mask ; ⃗ w r,mask ] 0.646 0.655 0.674 0.672 0.678 0.694 0.643 0.652 0.677 NA NA CONTEXT-WN All All All [ ⃗ w l,mean ; ⃗ w r,mean ] [ ⃗ c l,mean ; ⃗ c r,mean ] [ ⃗ w l,mask ; ⃗ w r,mask ] 0.674 0.688 0.700 0.635 0.648 0.661 0.645 0.657 0.670 0.641 0.665 0.677 0.577 0.608 0.623\n\nTable 6: Impact of masks on CONTEXT-PPDB and CONTEXT-WN.\n\nModeling word meaning in context Several approaches have been proposed to model the meaning of a word in a given context to capture semantic equivalence in tasks such as lexical substitution, word sense disambiguation or paraphrase ranking (but not entailment).\n\nOne line of work (Dinu and Lapata, 2010; Reisinger and Mooney, 2010) treats each word as a set of latent word senses. These models methods start with token representations for individual occurrences of a word and then choose a set of token vectors based on the current context. An alternate set of models (Erk and Pad´ o, 2008; Thater et al., 2011; Dinu et al., 2012) avoids defining a fixed set of word senses, and instead contextualizes word type vectors as we do in this paper. These models share the idea of using an element-wise multiplication to apply a context mask to word type representations. The nature of the context representation varies: Erk and Pado (2008) use inverse selectional preferences; Thater et al (2010) combine a first order representation for the context with a second order representation for the target, Thater et al. (2011) rely on syntactic dependencies to define context. Apidianaki (2016) shows that bag-ofword context representation within a small context window works as well as syntactic definitions of context for ranking paraphrases in context.\n\nThe use of convolution is motivated by success of similar models on sentence classification tasks. Tang et al (2014) uses convolution over embedding matrices for unigrams, bigrams, and trigrams, while Hovy (2015) uses just unigrams. However, all these works only use convolved representations to predict properties of the sentence (e.g., sentiment). We use them, instead, to contextualize our target word representations.\n\nIn-context lexical semantic tasks Besides entailment, other lexical semantic tasks studied in the presence of context include lexical substitution (McCarthy and Navigli, 2007), cross-lingual lexical substitution (Mihalcea et al., 2010) and paraphrase ranking (Apidianaki, 2016). The last work is also notable because of their successful use of models of word-meaning in context from Thater et al (2011), which is closely related to our methods."
      },
      {
        "heading": "10 Conclusion",
        "text": "We proposed to address lexical entailment in context, providing exemplar sentences to ground the meaning of words being considered for entailment. We show that contextualized word representations constructed by transforming contextagnostic representations, combined with wordcontext similarity features, lead to large improvements over context-agnostic model, not only in English, but also between English and French words on two novel datasets. We also improve the state-of-the-art on a related task of detecting semantic relations in context. Our features are sensitive to changes in entailment based on context, and also capture the directionality of entailment.\n\nIn future work, we aim to further improve performance on both monolingual and cross-lingual datasets. We anticipate that richer features, capturing second-order comparisons used to detect lexical contrast (Mohammad et al., 2013) and entailment out of context (Turney and Mohammad, 2013), might be useful for this purpose, perhaps in combination with non-linear classifiers. Given the usefulness of similarities, we also plan to investigate asymmetric scoring functions(Kotlerman et al., 2010; Shwartz et al., 2017)."
      }
    ],
    "references": [
      {
        "title": "Vector-space models for PPDB paraphrase ranking in context",
        "author": [
          "Marianna Apidianaki."
        ],
        "venue": "In Proceedings of EMNLP 2016 . Austin, TX, USA, pages 2028-2034.",
        "citeRegEx": "Apidianaki.,? 2016",
        "shortCiteRegEx": "Apidianaki.",
        "year": 2016
      },
      {
        "title": "Linear Algebraic Structure of Word Meanings , with Applications to Polysemy",
        "author": [
          "Sanjeev Arora",
          "Yuanzhi Li",
          "Yingyu Liang",
          "Tengyu Ma",
          "Andrej Risteski."
        ],
        "venue": "arXiv pages 1-25. http://arxiv.org/pdf/1601.03764v1.pdf.",
        "citeRegEx": "Arora et al\\.,? 2016",
        "shortCiteRegEx": "Arora et al\\.",
        "year": 2016
      },
      {
        "title": "Entailment above the word level in distributional semantics",
        "author": [
          "Marco Baroni",
          "Raffaella Bernardi",
          "Ngoc-Quynh Do",
          "Chung-chieh Shan."
        ],
        "venue": "In Proceedings of EACL 2012 . pages 23-32. http://dl.acm.org/citation.cfm?id=2380822.",
        "citeRegEx": "Baroni et al\\.,? 2012",
        "shortCiteRegEx": "Baroni et al\\.",
        "year": 2012
      },
      {
        "title": "How we BLESSed distributional semantic evaluation",
        "author": [
          "Marco Baroni",
          "Alessandro Lenci."
        ],
        "venue": "In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics . pages 1-10. http://dl.acm.org/citation.cfm?id=2140490.2140491.",
        "citeRegEx": "Baroni and Lenci.,? 2011",
        "shortCiteRegEx": "Baroni and Lenci.",
        "year": 2011
      },
      {
        "title": "A large annotated corpus for learning natural language inference",
        "author": [
          "Samuel R. Bowman",
          "Gabor Angeli",
          "Christopher Potts",
          "Christopher D. Manning."
        ],
        "venue": "In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics.",
        "citeRegEx": "Bowman et al\\.,? 2015",
        "shortCiteRegEx": "Bowman et al\\.",
        "year": 2015
      },
      {
        "title": "Recognizing Textual Entailment: Models and Applications",
        "author": [
          "Ido Dagan",
          "Dan Roth",
          "Mark Sammons",
          "Fabio Massimo Zanzotto."
        ],
        "venue": ". Morgan & Claypool Publishers.",
        "citeRegEx": "Dagan et al\\.,? 2013",
        "shortCiteRegEx": "Dagan et al\\.",
        "year": 2013
      },
      {
        "title": "Measuring Distributional Similarity in Context",
        "author": [
          "Georgiana Dinu",
          "Mirella Lapata."
        ],
        "venue": "In Proceedings of EMNLP 2010 . Cambridge, MA, USA, pages 1162-1172. http://eprints.pascalnetwork.org/archive/00008156/.",
        "citeRegEx": "Dinu and Lapata.,? 2010",
        "shortCiteRegEx": "Dinu and Lapata.",
        "year": 2010
      },
      {
        "title": "A comparison of models of word meaning in context",
        "author": [
          "Georgiana Dinu",
          "Stefan Thater",
          "S¨ oren Laue."
        ],
        "venue": "In Proceedings of NAACL-HLT 2012 . pages 611-615.",
        "citeRegEx": "Dinu et al\\.,? 2012",
        "shortCiteRegEx": "Dinu et al\\.",
        "year": 2012
      },
      {
        "title": "A structured vector space model for word meaning in context",
        "author": [
          "Katrin Erk",
          "Sebastian Pad´ o."
        ],
        "venue": "In Proceedings of EMNLP 2010 . Honolulu, HA, USA, October, pages 897-906. https://doi.org/10.3115/1613715.1613831.",
        "citeRegEx": "Erk and o.,? 2008",
        "shortCiteRegEx": "Erk and o.",
        "year": 2008
      },
      {
        "title": "WordNet",
        "author": [
          "Christiane Fellbaum."
        ],
        "venue": "Wiley Online Library.",
        "citeRegEx": "Fellbaum.,? 1998",
        "shortCiteRegEx": "Fellbaum.",
        "year": 1998
      },
      {
        "title": "WordNet: An Electronic Lexical Database",
        "author": [
          "Christiane Fellbaum",
          "editor."
        ],
        "venue": "MIT Press.",
        "citeRegEx": "Fellbaum and editor.,? 1998",
        "shortCiteRegEx": "Fellbaum and editor.",
        "year": 1998
      },
      {
        "title": "Introducing and evaluating ukWaC , a very large web-derived corpus of English",
        "author": [
          "Adriano Ferraresi",
          "Eros Zanchetta",
          "Marco Baroni",
          "Silvia Bernardini."
        ],
        "venue": "In Proceedings of the 4th Web as Corpus Workshop .",
        "citeRegEx": "Ferraresi et al\\.,? 2006",
        "shortCiteRegEx": "Ferraresi et al\\.",
        "year": 2006
      },
      {
        "title": "PPDB : The Paraphrase Database",
        "author": [
          "Juri Ganitkevitch",
          "Benjamin Van Durme",
          "Chris Callison-Burch."
        ],
        "venue": "Proceedings of NAACL-HLT 2013 (June):758--764. http://cs.jhu.edu/ ccb/publications/ppdb.pdf.",
        "citeRegEx": "Ganitkevitch et al\\.,? 2013",
        "shortCiteRegEx": "Ganitkevitch et al\\.",
        "year": 2013
      },
      {
        "title": "Knowledge processing on an extended wordnet",
        "author": [
          "Sanda Harabagiu",
          "Dan Moldovan."
        ],
        "venue": "WordNet: An electronic lexical database 305:381-405.",
        "citeRegEx": "Harabagiu and Moldovan.,? 1998",
        "shortCiteRegEx": "Harabagiu and Moldovan.",
        "year": 1998
      },
      {
        "title": "Multilingual Models for Compositional Distributed Semantics",
        "author": [
          "Karl Moritz Hermann",
          "Phil Blunsom."
        ],
        "venue": "In Proceedings of ACL 2014 . Baltimore, MD, USA, pages 58-68.",
        "citeRegEx": "Hermann and Blunsom.,? 2014",
        "shortCiteRegEx": "Hermann and Blunsom.",
        "year": 2014
      },
      {
        "title": "Demographic Factors Improve Classification Performance",
        "author": [
          "Dirk Hovy."
        ],
        "venue": "In Proceedings of ACLIJCNLP 2015 . Beijing, China, pages 752-762.",
        "citeRegEx": "Hovy.,? 2015",
        "shortCiteRegEx": "Hovy.",
        "year": 2015
      },
      {
        "title": "Directional Distributional Similarity for Lexical Inference",
        "author": [
          "Lili Kotlerman",
          "Ido Dagan",
          "Idan Szpektor",
          "Maayan Zhitomirsky-Geffet."
        ],
        "venue": "Natural Language Engineering 16(4):359-389. https://doi.org/10.1017/S1351324910000124.",
        "citeRegEx": "Kotlerman et al\\.,? 2010",
        "shortCiteRegEx": "Kotlerman et al\\.",
        "year": 2010
      },
      {
        "title": "Deriving Boolean structures from distributional vectors",
        "author": [
          "German Kruszewski",
          "Marco Baroni."
        ],
        "venue": "Transactions of ACL 3:375-388.",
        "citeRegEx": "Kruszewski and Baroni.,? 2015",
        "shortCiteRegEx": "Kruszewski and Baroni.",
        "year": 2015
      },
      {
        "title": "Do Supervised Distributional Methods Really Learn Lexical Inference Relations? In NAACL HLT 2015",
        "author": [
          "Omer Levy",
          "Steffen Remus",
          "Chris Biemann",
          "Ido Dagan."
        ],
        "venue": "pages 970-976.",
        "citeRegEx": "Levy et al\\.,? 2015",
        "shortCiteRegEx": "Levy et al\\.",
        "year": 2015
      },
      {
        "title": "Bilingual Word Representations with Monolingual Quality in Mind",
        "author": [
          "Minh-Thang Luong",
          "Hieu Pham",
          "Christopher D. Manning."
        ],
        "venue": "In Proceedings Workshop on Vector Modeling for NLP, NAACL 2015 . Denver, CO, USA, pages 151-159.",
        "citeRegEx": "Luong et al\\.,? 2015",
        "shortCiteRegEx": "Luong et al\\.",
        "year": 2015
      },
      {
        "title": "SemEval2007 Task 10: English Lexical Substitution Task",
        "author": [
          "Diana McCarthy",
          "Roberto Navigli."
        ],
        "venue": "In Proceedings of SEMEVAL 2007 . pages 48-53. https://doi.org/10.1007/s10579-009-9084-1.",
        "citeRegEx": "McCarthy and Navigli.,? 2007",
        "shortCiteRegEx": "McCarthy and Navigli.",
        "year": 2007
      },
      {
        "title": "context2vec: Learning Generic Context Embedding with Bidirectional LSTM",
        "author": [
          "Oren Melamud",
          "Jacob Goldberger",
          "Ido Dagan."
        ],
        "venue": "In Proceedings of CoNLL 2016 . Berlin, Germany, pages 51-61.",
        "citeRegEx": "Melamud et al\\.,? 2016",
        "shortCiteRegEx": "Melamud et al\\.",
        "year": 2016
      },
      {
        "title": "SemEval-2010 Task 2: Cross-Lingual Lexical Substitution",
        "author": [
          "Rada Mihalcea",
          "Ravi Sinha",
          "Diana McCarthy."
        ],
        "venue": "In Proceedings of SemEval 2010 (ACL 2010) . Uppsala, Sweden, July, pages 9-14.",
        "citeRegEx": "Mihalcea et al\\.,? 2010",
        "shortCiteRegEx": "Mihalcea et al\\.",
        "year": 2010
      },
      {
        "title": "Computing Lexical Contrast",
        "author": [
          "Saif M. Mohammad",
          "Bonnie J. Dorr",
          "Graeme Hirst",
          "Peter D. Turney."
        ],
        "venue": "Computational Linguistics 39(3):555-590.",
        "citeRegEx": "Mohammad et al\\.,? 2013",
        "shortCiteRegEx": "Mohammad et al\\.",
        "year": 2013
      },
      {
        "title": "Adding Semantics to Data-Driven Paraphrasing",
        "author": [
          "Ellie Pavlick",
          "Johan Bos",
          "Malvina Nissim",
          "Charley Beller",
          "Benjamin Van Durme."
        ],
        "venue": "In Proceedings of ACL 2015 . Beijing, China, pages 15121522.",
        "citeRegEx": "Pavlick et al\\.,? 2015",
        "shortCiteRegEx": "Pavlick et al\\.",
        "year": 2015
      },
      {
        "title": "PPDB 2.0: Better paraphrase ranking, finegrained entailment relations, word embeddings, and style classification",
        "author": [
          "Ellie Pavlick",
          "Pushpendre Rastogi",
          "Juri Ganitkevitch",
          "Benjamin Van Durme",
          "Chris Callison-Burch."
        ],
        "venue": "Proceedings of ACL-IJCNLP 2015 pages 425-430.",
        "citeRegEx": "Pavlick et al\\.,? 2015",
        "shortCiteRegEx": "Pavlick et al\\.",
        "year": 2015
      },
      {
        "title": "GloVe: Global Vectors for Word Representation",
        "author": [
          "Jeffrey Pennington",
          "Richard Socher",
          "Christopher D Manning."
        ],
        "venue": "In Proceedings of EMNLP 2014 . Doha, Qatar, pages 1532-1543. https://doi.org/10.3115/v1/D14-1162.",
        "citeRegEx": "Pennington et al\\.,? 2014",
        "shortCiteRegEx": "Pennington et al\\.",
        "year": 2014
      },
      {
        "title": "Multi-Prototype Vector-Space Models of Word Meaning",
        "author": [
          "Joseph Reisinger",
          "Raymond J Mooney."
        ],
        "venue": "In Proceedings of NAACL 2010 . Los Angeles, CA, June, pages 109-117.",
        "citeRegEx": "Reisinger and Mooney.,? 2010",
        "shortCiteRegEx": "Reisinger and Mooney.",
        "year": 2010
      },
      {
        "title": "Inclusive yet Selective: Supervised Distributional Hypernymy Detection",
        "author": [
          "Stephen Roller",
          "Katrin Erk",
          "Gemma Boleda."
        ],
        "venue": "Proceedings of COLING 2014 pages 1025-1036.",
        "citeRegEx": "Roller et al\\.,? 2014",
        "shortCiteRegEx": "Roller et al\\.",
        "year": 2014
      },
      {
        "title": "Adding Context to Semantic Data-Driven Paraphrasing",
        "author": [
          "Vered Shwartz",
          "Ido Dagan."
        ],
        "venue": "In Proceedings of *SEM 2016 . Berlin, Germany, 2015, pages 108-113.",
        "citeRegEx": "Shwartz and Dagan.,? 2015",
        "shortCiteRegEx": "Shwartz and Dagan.",
        "year": 2015
      },
      {
        "title": "Learning to Exploit Structured Resources for Lexical Inference",
        "author": [
          "Vered Shwartz",
          "Omer Levy",
          "Ido Dagan",
          "Jacob Goldberger."
        ],
        "venue": "In Proceedings of CoNLL 2015 . Beijing, China, pages 175-184. http://www.aclweb.org/anthology/K15-1018.",
        "citeRegEx": "Shwartz et al\\.,? 2015",
        "shortCiteRegEx": "Shwartz et al\\.",
        "year": 2015
      },
      {
        "title": "Hypernyms under Siege: Linguistically-motivated Artillery for Hypernymy Detection",
        "author": [
          "Vered Shwartz",
          "Enrico Santus",
          "Dominik Schlechtweg."
        ],
        "venue": "In Proceedings of EACL 2017 . Valencia, Spain.",
        "citeRegEx": "Shwartz et al\\.,? 2017",
        "shortCiteRegEx": "Shwartz et al\\.",
        "year": 2017
      },
      {
        "title": "Learning SentimentSpecific Word Embedding",
        "author": [
          "Duyu Tang",
          "Furu Wei",
          "Nan Yang",
          "Ming Zhou",
          "Ting Liu",
          "Bing Qin."
        ],
        "venue": "In Proceedings of ACL 2014 . Baltimore, MD, USA, pages 1555-1565. https://doi.org/10.3115/1220575.1220648.",
        "citeRegEx": "Tang et al\\.,? 2014",
        "shortCiteRegEx": "Tang et al\\.",
        "year": 2014
      },
      {
        "title": "Contextualizing Semantic Representations Using Syntactically Enriched Vector Models",
        "author": [
          "Stefan Thater",
          "Hagen Fuerstenau",
          "Manfred Pinkal."
        ],
        "venue": "In Proceedings of ACL 2010 . Uppsala, Sweden, July, pages 948-957. http://eprints.pascalnetwork.org/archive/00008090/.",
        "citeRegEx": "Thater et al\\.,? 2010",
        "shortCiteRegEx": "Thater et al\\.",
        "year": 2010
      },
      {
        "title": "Word Meaning in Context : A Simple and Effective Vector Model",
        "author": [
          "Stefan Thater",
          "Hagen F¨ urstenau",
          "Manfred Pinkal."
        ],
        "venue": "In Proceedings of IJCNLP 2011 . Chiang Mai, Thailand, pages 1134-1143.",
        "citeRegEx": "Thater et al\\.,? 2011",
        "shortCiteRegEx": "Thater et al\\.",
        "year": 2011
      },
      {
        "title": "Experiments with three approaches to recognizing lexical entailment",
        "author": [
          "Peter Turney",
          "Saif Mohammad."
        ],
        "venue": "Natural Language Engineering 1(1):1-42. https://doi.org/10.1017/S1351324913000387.",
        "citeRegEx": "Turney and Mohammad.,? 2013",
        "shortCiteRegEx": "Turney and Mohammad.",
        "year": 2013
      },
      {
        "title": "Cross-lingual Models of Word Embeddings: An Empirical Comparison",
        "author": [
          "Shyam Upadhyay",
          "Manaal Faruqui",
          "Chris Dyer",
          "Dan Roth."
        ],
        "venue": "In Proceedings of ACL . Berlin, Germany. http://arxiv.org/abs/1604.00425.",
        "citeRegEx": "Upadhyay et al\\.,? 2016",
        "shortCiteRegEx": "Upadhyay et al\\.",
        "year": 2016
      },
      {
        "title": "Sparse Bilingual Word Representations for Cross-lingual Lexical Entailment",
        "author": [
          "Yogarshi Vyas",
          "Marine Carpuat."
        ],
        "venue": "Proceedings of NAACL-HLT 2016 pages 1187-1197.",
        "citeRegEx": "Vyas and Carpuat.,? 2016",
        "shortCiteRegEx": "Vyas and Carpuat.",
        "year": 2016
      },
      {
        "title": "Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learning",
        "author": [
          "Ekaterina Vylomova",
          "Laura Rimell",
          "Trevor Cohn",
          "Timothy Baldwin."
        ],
        "venue": "In Proceedings of ACL 2016 . Berlin, Germany, pages 1671-1682.",
        "citeRegEx": "Vylomova et al\\.,? 2016",
        "shortCiteRegEx": "Vylomova et al\\.",
        "year": 2016
      },
      {
        "title": "Bootstrapping Distributional Feature Vector Quality",
        "author": [
          "Maayan Zhitomirsky-Geffet",
          "Ido Dagan."
        ],
        "venue": "Computational Linguistics (November 2008).",
        "citeRegEx": "Zhitomirsky-Geffet and Dagan.,? 2009",
        "shortCiteRegEx": "Zhitomirsky-Geffet and Dagan.",
        "year": 2009
      }
    ],
    "referenceMentions": [
      {
        "referenceID": 29,
        "context": "We show that contextualized word representations constructed from existing word embeddings, and word-context similarity features lead to significant improvements over context-agnostic models on two novel entailment test sets, and also improve the state-of-the-art on the related task of detecting semantic relations in context (Shwartz and Dagan, 2015).",
        "startOffset": 328,
        "endOffset": 351
      },
      {
        "referenceID": 35,
        "context": "While prior work has defined lexical entailment as a relation between word types (Turney and Mohammad, 2013), we argue entailment relations are better defined when illustrating word meaning with an example in context.",
        "startOffset": 82,
        "endOffset": 107
      },
      {
        "referenceID": 2,
        "context": "Since distributional representations for word types have proved useful to detect lexical entailment out of context in supervised settings (Baroni et al., 2012; Roller et al., 2014; Turney and Mohammad, 2013), we propose to transform context-agnostic word type representations into contextualized representations that highlight salient properties of the context (Section 3), and use these contextualized representations with a range of semantic similarity features (Section 4) to successfully detect entailment.",
        "startOffset": 139,
        "endOffset": 158
      },
      {
        "referenceID": 28,
        "context": "Since distributional representations for word types have proved useful to detect lexical entailment out of context in supervised settings (Baroni et al., 2012; Roller et al., 2014; Turney and Mohammad, 2013), we propose to transform context-agnostic word type representations into contextualized representations that highlight salient properties of the context (Section 3), and use these contextualized representations with a range of semantic similarity features (Section 4) to successfully detect entailment.",
        "startOffset": 160,
        "endOffset": 179
      },
      {
        "referenceID": 35,
        "context": "Since distributional representations for word types have proved useful to detect lexical entailment out of context in supervised settings (Baroni et al., 2012; Roller et al., 2014; Turney and Mohammad, 2013), we propose to transform context-agnostic word type representations into contextualized representations that highlight salient properties of the context (Section 3), and use these contextualized representations with a range of semantic similarity features (Section 4) to successfully detect entailment.",
        "startOffset": 181,
        "endOffset": 206
      },
      {
        "referenceID": 29,
        "context": "Moreover, we establish a new state-of-the-art on an existing dataset that captures a broader range of semantic relations in context (Shwartz and Dagan, 2015), and show that the proposed features, induced solely from large amounts of raw text, yield systems that perform as well, or better than existing systems that require additional human annotation.",
        "startOffset": 133,
        "endOffset": 156
      },
      {
        "referenceID": 5,
        "context": "Recognizing textual entailment (Dagan et al., 2013) and natural language inference (Bowman et al., 2015) involves detecting entailment relations between sentences, while lexical entailment is a relation between words.",
        "startOffset": 32,
        "endOffset": 50
      },
      {
        "referenceID": 4,
        "context": "Recognizing textual entailment (Dagan et al., 2013) and natural language inference (Bowman et al., 2015) involves detecting entailment relations between sentences, while lexical entailment is a relation between words.",
        "startOffset": 84,
        "endOffset": 103
      },
      {
        "referenceID": 2,
        "context": "We start from existing representations for word types which have proven useful for detecting lexical entailment and other semantic relations out of context (Baroni et al., 2012; Kruszewski and Baroni, 2015; Vylomova et al., 2016; Turney and Mohammad, 2013).",
        "startOffset": 157,
        "endOffset": 176
      },
      {
        "referenceID": 17,
        "context": "We start from existing representations for word types which have proven useful for detecting lexical entailment and other semantic relations out of context (Baroni et al., 2012; Kruszewski and Baroni, 2015; Vylomova et al., 2016; Turney and Mohammad, 2013).",
        "startOffset": 178,
        "endOffset": 205
      },
      {
        "referenceID": 38,
        "context": "We start from existing representations for word types which have proven useful for detecting lexical entailment and other semantic relations out of context (Baroni et al., 2012; Kruszewski and Baroni, 2015; Vylomova et al., 2016; Turney and Mohammad, 2013).",
        "startOffset": 207,
        "endOffset": 228
      },
      {
        "referenceID": 35,
        "context": "We start from existing representations for word types which have proven useful for detecting lexical entailment and other semantic relations out of context (Baroni et al., 2012; Kruszewski and Baroni, 2015; Vylomova et al., 2016; Turney and Mohammad, 2013).",
        "startOffset": 230,
        "endOffset": 255
      },
      {
        "referenceID": 32,
        "context": "Specifically, we calculate the column-wise maximum, minimum and the mean over the matrices C l and C r , as done by (Tang et al., 2014) for supervised sentiment classification.",
        "startOffset": 117,
        "endOffset": 134
      },
      {
        "referenceID": 15,
        "context": "Computing the maximum and minimum across all vector dimensions captures the exterior surface of the 'instance manifold'\n\n(the volume in embedding space within which all words in the instance reside), while the mean summarizes the density per-dimension within the manifold (Hovy, 2015).",
        "startOffset": 273,
        "endOffset": 283
      },
      {
        "referenceID": 21,
        "context": "In a single language, this can be done using Context2Vec (Melamud et al., 2016), a neural model that, given a target word and its sentential context, embeds both the word and the context in the same low-dimensional space, with the objective of having the context predict the target word via a log linear model.",
        "startOffset": 58,
        "endOffset": 78
      },
      {
        "referenceID": 2,
        "context": "Prior work on lexical entailment out of context suggests that the entailment relationship between words is a learnable function of the concatenation of their individual representations (Baroni et al., 2012; Turney and Mohammad, 2013).",
        "startOffset": 186,
        "endOffset": 205
      },
      {
        "referenceID": 35,
        "context": "Prior work on lexical entailment out of context suggests that the entailment relationship between words is a learnable function of the concatenation of their individual representations (Baroni et al., 2012; Turney and Mohammad, 2013).",
        "startOffset": 207,
        "endOffset": 232
      },
      {
        "referenceID": 28,
        "context": "After concatenation, they are used as features for a logistic regression (Roller et al., 2014) or an SVM classifier (Baroni et al., 2012; Turney and Mohammad, 2013).",
        "startOffset": 74,
        "endOffset": 93
      },
      {
        "referenceID": 2,
        "context": "After concatenation, they are used as features for a logistic regression (Roller et al., 2014) or an SVM classifier (Baroni et al., 2012; Turney and Mohammad, 2013).",
        "startOffset": 117,
        "endOffset": 136
      },
      {
        "referenceID": 35,
        "context": "After concatenation, they are used as features for a logistic regression (Roller et al., 2014) or an SVM classifier (Baroni et al., 2012; Turney and Mohammad, 2013).",
        "startOffset": 138,
        "endOffset": 163
      },
      {
        "referenceID": 12,
        "context": "The first dataset, CONTEXT-PPDB, is a finegrained lexical inference dataset created using 375 word pairs from a subset of the English Paraphrase Database (Ganitkevitch et al., 2013; Pavlick et al., 2015b).",
        "startOffset": 155,
        "endOffset": 180
      },
      {
        "referenceID": 24,
        "context": "The first dataset, CONTEXT-PPDB, is a finegrained lexical inference dataset created using 375 word pairs from a subset of the English Paraphrase Database (Ganitkevitch et al., 2013; Pavlick et al., 2015b).",
        "startOffset": 182,
        "endOffset": 202
      },
      {
        "referenceID": 25,
        "context": "The first dataset, CONTEXT-PPDB, is a finegrained lexical inference dataset created using 375 word pairs from a subset of the English Paraphrase Database (Ganitkevitch et al., 2013; Pavlick et al., 2015b).",
        "startOffset": 182,
        "endOffset": 202
      },
      {
        "referenceID": 9,
        "context": "Based on the above criteria, we introduce a large-scale dataset automatically extracted from WordNet (Fellbaum, 1998a).",
        "startOffset": 102,
        "endOffset": 116
      },
      {
        "referenceID": 18,
        "context": "We also ensure that each set contains different word pairs, to avoid memorization and overfitting (Levy et al., 2015).",
        "startOffset": 99,
        "endOffset": 116
      },
      {
        "referenceID": 26,
        "context": "We use 50 dimensional off-the-shelf GloVe embeddings (Pennington et al., 2014) to create ⃗ w l , ⃗ w r , C l , and C r .",
        "startOffset": 54,
        "endOffset": 77
      },
      {
        "referenceID": 11,
        "context": "To obtain the Context2Vec representations ⃗ w l,c 2 v , ⃗ w l,c 2 v , ⃗ c l,c 2 v , and ⃗ c r,c 2 v , we use an existing model trained on the ukWaC corpus (Ferraresi et al., 2006) .",
        "startOffset": 156,
        "endOffset": 178
      },
      {
        "referenceID": 19,
        "context": "To represent ⃗ w l , ⃗ w r , we experiment with two different pre-trained, 200dimensional, bilingual embeddings - BiVec (Luong et al., 2015) and BiCVM (Hermann and Blunsom, 2014), both of which have been shown to be suitable for cross-lingual semantic tasks such as dictionary induction and document classification (Upadhyay et al., 2016).",
        "startOffset": 121,
        "endOffset": 139
      },
      {
        "referenceID": 14,
        "context": "To represent ⃗ w l , ⃗ w r , we experiment with two different pre-trained, 200dimensional, bilingual embeddings - BiVec (Luong et al., 2015) and BiCVM (Hermann and Blunsom, 2014), both of which have been shown to be suitable for cross-lingual semantic tasks such as dictionary induction and document classification (Upadhyay et al., 2016).",
        "startOffset": 152,
        "endOffset": 177
      },
      {
        "referenceID": 36,
        "context": "To represent ⃗ w l , ⃗ w r , we experiment with two different pre-trained, 200dimensional, bilingual embeddings - BiVec (Luong et al., 2015) and BiCVM (Hermann and Blunsom, 2014), both of which have been shown to be suitable for cross-lingual semantic tasks such as dictionary induction and document classification (Upadhyay et al., 2016).",
        "startOffset": 316,
        "endOffset": 337
      },
      {
        "referenceID": 2,
        "context": "Representation-based features only A context-agnostic baseline (Baroni et al., 2012) based on the concatenation of the word type features for ( w l , w r ) reaches F1 scores of 53.5 using Glove and 54 with Context2Vec (Table 4).",
        "startOffset": 64,
        "endOffset": 83
      },
      {
        "referenceID": 24,
        "context": "Adding PPDB-specific features CONTEXTPPDB comes with rich information about word pairs drawn from PPDB (Pavlick et al., 2015a,b).",
        "startOffset": 104,
        "endOffset": 124
      },
      {
        "referenceID": 25,
        "context": "Adding PPDB-specific features CONTEXTPPDB comes with rich information about word pairs drawn from PPDB (Pavlick et al., 2015a,b).",
        "startOffset": 104,
        "endOffset": 124
      },
      {
        "referenceID": 9,
        "context": "WordNet and lexical entailment The is-a hierarchy of WordNet (Fellbaum, 1998b) is a prominent source of information for unsupervised detection of hypernymy and entailment (Harabagiu and Moldovan, 1998; Shwartz et al., 2015), as well as a source of various datasets (Baroni and Lenci, 2011; Baroni et al., 2012).",
        "startOffset": 62,
        "endOffset": 76
      },
      {
        "referenceID": 13,
        "context": "WordNet and lexical entailment The is-a hierarchy of WordNet (Fellbaum, 1998b) is a prominent source of information for unsupervised detection of hypernymy and entailment (Harabagiu and Moldovan, 1998; Shwartz et al., 2015), as well as a source of various datasets (Baroni and Lenci, 2011; Baroni et al., 2012).",
        "startOffset": 172,
        "endOffset": 200
      },
      {
        "referenceID": 30,
        "context": "WordNet and lexical entailment The is-a hierarchy of WordNet (Fellbaum, 1998b) is a prominent source of information for unsupervised detection of hypernymy and entailment (Harabagiu and Moldovan, 1998; Shwartz et al., 2015), as well as a source of various datasets (Baroni and Lenci, 2011; Baroni et al., 2012).",
        "startOffset": 202,
        "endOffset": 222
      },
      {
        "referenceID": 3,
        "context": "WordNet and lexical entailment The is-a hierarchy of WordNet (Fellbaum, 1998b) is a prominent source of information for unsupervised detection of hypernymy and entailment (Harabagiu and Moldovan, 1998; Shwartz et al., 2015), as well as a source of various datasets (Baroni and Lenci, 2011; Baroni et al., 2012).",
        "startOffset": 266,
        "endOffset": 288
      },
      {
        "referenceID": 2,
        "context": "WordNet and lexical entailment The is-a hierarchy of WordNet (Fellbaum, 1998b) is a prominent source of information for unsupervised detection of hypernymy and entailment (Harabagiu and Moldovan, 1998; Shwartz et al., 2015), as well as a source of various datasets (Baroni and Lenci, 2011; Baroni et al., 2012).",
        "startOffset": 290,
        "endOffset": 309
      },
      {
        "referenceID": 6,
        "context": "One line of work (Dinu and Lapata, 2010; Reisinger and Mooney, 2010) treats each word as a set of latent word senses.",
        "startOffset": 18,
        "endOffset": 39
      },
      {
        "referenceID": 27,
        "context": "One line of work (Dinu and Lapata, 2010; Reisinger and Mooney, 2010) treats each word as a set of latent word senses.",
        "startOffset": 41,
        "endOffset": 67
      },
      {
        "referenceID": 34,
        "context": "An alternate set of models (Erk and Pad´ o, 2008; Thater et al., 2011; Dinu et al., 2012) avoids defining a fixed set of word senses, and instead contextualizes word type vectors as we do in this paper.",
        "startOffset": 50,
        "endOffset": 69
      },
      {
        "referenceID": 7,
        "context": "An alternate set of models (Erk and Pad´ o, 2008; Thater et al., 2011; Dinu et al., 2012) avoids defining a fixed set of word senses, and instead contextualizes word type vectors as we do in this paper.",
        "startOffset": 71,
        "endOffset": 88
      },
      {
        "referenceID": 20,
        "context": "In-context lexical semantic tasks Besides entailment, other lexical semantic tasks studied in the presence of context include lexical substitution (McCarthy and Navigli, 2007), cross-lingual lexical substitution (Mihalcea et al., 2010) and paraphrase ranking (Apidianaki, 2016).",
        "startOffset": 148,
        "endOffset": 174
      },
      {
        "referenceID": 22,
        "context": "In-context lexical semantic tasks Besides entailment, other lexical semantic tasks studied in the presence of context include lexical substitution (McCarthy and Navigli, 2007), cross-lingual lexical substitution (Mihalcea et al., 2010) and paraphrase ranking (Apidianaki, 2016).",
        "startOffset": 213,
        "endOffset": 234
      },
      {
        "referenceID": 0,
        "context": "In-context lexical semantic tasks Besides entailment, other lexical semantic tasks studied in the presence of context include lexical substitution (McCarthy and Navigli, 2007), cross-lingual lexical substitution (Mihalcea et al., 2010) and paraphrase ranking (Apidianaki, 2016).",
        "startOffset": 260,
        "endOffset": 276
      },
      {
        "referenceID": 23,
        "context": "We anticipate that richer features, capturing second-order comparisons used to detect lexical contrast (Mohammad et al., 2013) and entailment out of context (Turney and Mohammad, 2013), might be useful for this purpose, perhaps in combination with non-linear classifiers.",
        "startOffset": 104,
        "endOffset": 125
      },
      {
        "referenceID": 35,
        "context": "We anticipate that richer features, capturing second-order comparisons used to detect lexical contrast (Mohammad et al., 2013) and entailment out of context (Turney and Mohammad, 2013), might be useful for this purpose, perhaps in combination with non-linear classifiers.",
        "startOffset": 158,
        "endOffset": 183
      },
      {
        "referenceID": 16,
        "context": "Given the usefulness of similarities, we also plan to investigate asymmetric scoring functions(Kotlerman et al., 2010; Shwartz et al., 2017).",
        "startOffset": 95,
        "endOffset": 117
      },
      {
        "referenceID": 31,
        "context": "Given the usefulness of similarities, we also plan to investigate asymmetric scoring functions(Kotlerman et al., 2010; Shwartz et al., 2017).",
        "startOffset": 119,
        "endOffset": 139
      }
    ],
    "year": null,
    "abstractText": "Detecting entailment between words is a key task for several NLP applications. Previous work has largely focused on entailment between words out of context. We propose, instead, to address lexical entailment in context, providing exemplar sentences to ground the meaning of words considered in the entailment relation. We show that contextualized word representations constructed from existing word embeddings, and word-context similarity features lead to significant improvements over context-agnostic models on two novel entailment test sets, and also improve the state-of-the-art on the related task of detecting semantic relations in context (Shwartz and Dagan, 2015).",
    "creator": "Docling-granite"
  }
}