Handling terminology is an important matter in a translation workflow .

However , current Machine Translation ( MT ) systems do not yet propose anything proactive upon tools which assist in managing terminological databases .

In this work , we investigate several enhancements to analogical learning and test our implementation on translating medical terms .

We show that the analogical engine works equally well when translating from and into a morphologically rich language , or when dealing with language pairs written in different scripts .

Combining it with a phrasebased statistical engine leads to significant improvements .

If machine translation is to meet commercial needs , it must offer a sensible approach to translating terms .

Currently , MT systems offer at best database management tools which allow a human ( typically a translator , a terminologist or even the vendor of the system ) to specify bilingual terminological entries .

More advanced tools are meant to identify inconsistencies in terminological translations and might prove useful in controlledlanguage situations ( Itagaki et al . , 2007 ) .

One approach to translate terms consists in using a domain-specific parallel corpus with standard alignment techniques ( Brown et al . , 1993 ) to mine new translations .

Massive amounts of parallel data are certainly available in several pairs of languages for domains such as parliament debates or the like .

However , having at our disposal a domain-specific ( e.g. computer science ) bitext with an adequate coverage is another issue .

One might argue that domain-specific comparable ( or perhaps unrelated ) corpora are easier to acquire , in which case context-vector techniques ( Rapp , 1995 ; Fung and McKeown , 1997 ) can be used to identify the translation of terms .

We certainly agree with that point of view to a certain extent , but as discussed by Morin et al . ( 2007 ) , for many specific domains and pairs of languages , such resources simply do not exist .

Furthermore , the task of translation identification is more difficult and error-prone .

Analogical learning has recently regained some interest in the NLP community .

Lepage and Denoual ( 2005 ) proposed a machine translation system entirely based on the concept of formal analogy , that is , analogy on forms .

Stroppa and Yvon ( 2005 ) applied analogical learning to several morphological tasks also involving analogies on words .

Langlais and Patry ( 2007 ) applied it to the task of translating unknown words in several European languages , an idea investigated as well by Denoual ( 2007 ) for a Japanese to English translation task .

In this study , we improve the state-of-the-art of analogical learning by ( i ) proposing a simple yet effective implementation of an analogical solver ; ( ii ) proposing an efficient solution to the search issue embedded in analogical learning , ( iii ) investigating whether a classifier can be trained to recognize bad candidates produced by analogical learning .

We evaluate our analogical engine on the task of translating terms of the medical domain ; a domain well-known for its tendency to create new words , many of which being complex lexical constructions .

Our experiments involve five language pairs , including languages with very different morphological systems .

In the remainder of this paper , we first present in Section 2 the principle of analogical learning .

Practical issues in analogical learning are discussed in Section 3 along with our solutions .

In Section 4 , we report on experiments we conducted with our analogical device .

We conclude this study and discuss future work in Section 5 .

In this study , we proposed solutions to practical issues involved in analogical learning .

A simple yet effective implementation of a solver is described .

A search strategy is proposed which outperforms the one described in ( Langlais and Patry , 2007 ) .

Also , we showed that a classifier trained to select good candidate translations outperforms the most-frequently-generated heuristic used in several works on analogical learning .

Our analogical device was used to translate medical terms in different language pairs .

The approach rates comparably across the 10 translation directions we considered .

In particular , we do not see a drop in performance when translating into a morphology rich language ( such as Finnish ) , or when translating into languages with different scripts .

Averaged over all translation directions , the best variant could translate in first position 21 % of the terms with a precision of 57 % , while at best , one could translate 30 % of the terms with a perfect precision .

We show that the analogical translations are of better quality than those produced by a phrase-based engine trained at the character level , albeit with much lower recall .

A straightforward combination of both approaches led an improvement of 5.3 BLEU points over the SMT alone .

Better SMT performance could be obtained with a system based on morphemes , see for instance ( Toutanova et al . , 2008 ) .

However , since lists of morphemes specific to the medical domain do not exist for all the languages pairs we considered here , unsupervised methods for acquiring morphemes would be necessary , which is left as a future work .

In any case , this comparison is meaningful , since both the SMT and the analogical device work at the character level .

First , we will test our approach on terminologies from different domains , varying the size of the training material .

Second , analyzing the segmentation induced by analogical learning would be interesting .

Third , we need to address the problem of combining the translations produced by analogy into a front-end statistical translation engine .

Last , there is no reason to constrain ourselves to translating terminology only .

We targeted this task in the first place , because terminology typically plugs translation systems , but we think that analogical learning could be useful for translating infrequent entities .


Reasoning about implied relationships ( e.g. paraphrastic , common sense , encyclopedic ) between pairs of words is crucial for many cross-sentence inference problems .

This paper proposes new methods for learning and using embeddings of word pairs that implicitly represent background knowledge about such relationships .

Our pairwise embeddings are computed as a compositional function on word representations , which is learned by maximizing the pointwise mutual information ( PMI ) with the contexts in which the two words cooccur .

We add these representations to the cross-sentence attention layer of existing inference models ( e.g. BiDAF for QA , ESIM for NLI ) , instead of extending or replacing existing word embeddings .

Experiments show a gain of 2.7 % on the recently released SQuAD 2.0 and 1.3 % on MultiNLI .

Our representations also aid in better generalization with gains of around 6 - 7 % on adversarial SQuAD datasets , and 8.8 % on the adversarial entailment test set by Glockner et al . ( 2018 ) .

Reasoning about relationships between pairs of words is crucial for cross sentence inference problems such as question answering ( QA ) and natural language inference ( NLI ) .

In NLI , for example , given the premise " golf is prohibitively expensive " , inferring that the hypothesis " golf is a cheap pastime " is a contradiction requires one to know that expensive and cheap are antonyms .

Recent work ( Glockner et al . , 2018 ) has shown that current models , which rely heavily on unsupervised single-word embeddings , struggle to learn such relationships .

In this paper , we show that they can be learned with word pair vectors ( pair2vec 1 ) , which are trained unsupervised , and which significantly improve performance when added to existing cross-sentence attention mechanisms .

Unlike single-word representations , which typically model the co-occurrence of a target word x with its context c , our word-pair representations are learned by modeling the three-way cooccurrence between words ( x , y ) and the context c that ties them together , as seen in Table 1 .

While similar training signals have been used to learn models for ontology construction ( Hearst , 1992 ;

Snow et al . , 2005 ; Turney , 2005 ; Shwartz et al . , 2016 ) and knowledge base completion ( Riedel et al . , 2013 ) , this paper shows , for the first time , that large scale learning of pairwise embeddings can be used to directly improve the performance of neural cross-sentence inference models .

More specifically , we train a feedforward network R(x , y ) that learns representations for the individual words x and y , as well as how to compose them into a single vector .

Training is done by maximizing a generalized notion of the pointwise mutual information ( PMI ) among x , y , and their context c using a variant of negative sampling ( Mikolov et al . , 2013a ) .

Making R(x , y ) a compositional function on individual words alleviates the sparsity that necessarily comes with embedding pairs of words , even at a very large scale .

We show that our embeddings can be added to existing cross-sentence inference models , such as BiDAF++ ( Seo et al . , 2017 ; Clark and Gardner , 2018 ) for QA and ESIM ( Chen et al . , 2017 ) for NLI .

Instead of changing the word embeddings that are fed into the encoder , we add the pretrained pair representations to higher layers in the network where cross sentence attention mechanisms are used .

This allows the model to use the background knowledge that the pair embeddings implicitly encode to reason about the likely relationships between the pairs of words it aligns .

Experiments show that simply adding our wordpair embeddings to existing high-performing models , which already use ELMo ( Peters et al . , 2018 ) , results in sizable gains .

We show 2.72 F1 points over the BiDAF++ model ( Clark and Gardner , 2018 ) on SQuAD 2.0 ( Rajpurkar et al . , 2018 ) , as well as a 1.3 point gain over ESIM ( Chen et al . , 2017 ) on MultiNLI ( Williams et al . , 2018 ) .

Additionally , our approach generalizes well to adversarial examples , with a 6 - 7 % F1 increase on adversarial SQuAD ( Jia and Liang , 2017 ) and a 8.8 % gain on the Glockner et al . ( 2018 ) NLI benchmark .

An analysis of pair2vec on word analogies suggests that it complements the information in single-word representations , especially for encyclopedic and lexicographic relations .

We presented new methods for training and using word pair embeddings that implicitly represent background knowledge .

Our pair embeddings are computed as a compositional function of the individual word representations , which is learned by maximizing a variant of the PMI with the contexts in which the the two words co-occur .

Experiments on cross-sentence inference benchmarks demonstrated that adding these representations to existing models results in sizable improvements for both in-domain and adversarial settings .

Published concurrently with this paper , BERT ( Devlin et al . , 2018 ) , which uses a masked language model objective , has reported dramatic gains on multiple semantic benchmarks including question-answering , natural language inference , and named entity recognition .

Potential avenues for future work include multitasking BERT with pair2vec in order to more directly incorporate reasoning about word pair relations into the BERT objective .

Leveraging knowledge bases in LSTMs for improving machine reading .

In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( Volume 1 : Long Papers ) , pages 1436 - 1446 .

Association for Computational Linguistics .


It is widely recognized that the proliferation of annotation schemes runs counter to the need to re-use language resources , and that standards for linguistic annotation are becoming increasingly mandatory .

To answer this need , we have developed a representation framework comprised of an abstract model for a variety of different annotation types ( e.g. , morpho-syntactic tagging , syntactic annotation , co-reference annotation , etc . ) , which can be instantiated in different ways depending on the annotator s approach and goals .

In this paper we provide an overview of our representation framework and demonstrate its applicability to syntactic annotation .

We show how the framework can contribute to comparative evaluation and merging of parser output and diverse syntactic annotation schemes .

In particular , there is a need for a general framework for linguistic annotation that is flexible and extensible enough to accommodate different annotation types and different theoretical and practical approaches , while at the same time enabling their representation in a pivot format that can serve as the basis for comparative evaluation of parser output , such as PARSEVAL ( Harrison , et al . , 1991 ) , as well as the development of reusable editing and processing tools .

We have implemented both the abstract model and various instantiations using XML schemas ( Thompson , et al . , 2000 ) , the Resource Definition Framework ( RDF ) ( Lassila and Swick , 2000 ) and RDF schemas ( Brickley and Guha , 2000 ) , which enable description and definition of abstract data models together with means to interpret , via the model , information encoded according to different conventions .

The results have been incorporated into XCES ( Ide , et al . , 2000a ) , part of the EAGLES Guidelines developed by the Expert Advisory Group on Language Engineering Standards ( EAGLES ) 1 .

The XCES provides a ready-made , standard encoding format together with a data architecture designed specifically for linguistically annotated corpora .

The framework has been applied to the representation of terminology ( Terminological Markup Framework2 , ISO project n.16642 ) and computational lexicons ( Ide , et al . , 2000b ) , thus demonstrating its general applicability for a variety of linguistic annotation types .

We also show how the framework can contribute to comparison and merging of diverse syntactic annotation schemes .

The XCES framework for linguistic annotation is built around some relatively straightforward ideas : separation of information conveyed by means of structure and information conveyed directly by specification of content categories ; development of an abstract format that puts a layer of abstraction between site-specific annotation schemes and standard specifications ; and creation of a Data Category Registry to provide a reference set of annotation categories .

The emergence of XML and related standards such as RDF provides the enabling technology .

We are , therefore , at a point where the creation and use of annotated data and concerns about the way it is represented can be treated separately that is , researchers can focus on the question of what to encode , independent of the question of how to encode it .

The end result should be greater coherence , consistency , and ease of use and access for annotated data .


Visual question answering aims to answer the natural language question about a given image .

Existing graph-based methods only focus on the relations between objects in an image and neglect the importance of the syntactic dependency relations between words in a question .

To simultaneously capture the relations between objects in an image and the syntactic dependency relations between words in a question , we propose a novel dual channel graph convolutional network ( DC-GCN ) for better combining visual and textual advantages .

The DC-GCN model consists of three parts : an I-GCN module to capture the relations between objects in an image , a Q-GCN module to capture the syntactic dependency relations between words in a question , and an attention alignment module to align image representations and question representations .

Experimental results show that our model achieves comparable performance with the state-of-theart approaches .

As a form of visual Turing test , visual question answering ( VQA ) has drawn much attention .

The goal of VQA ( Antol et al . , 2015 ; Goyal et al . , 2017 ) is to answer a natural language question related to the contents of a given image .

Attention mechanisms are served as the backbone of the previous mainstream approaches ( Lu et al . , 2016 ; Yang et al . , 2016 ; Yu et al . , 2017 ) , however , they tend to catch only the most discriminative information , ignoring other rich complementary clues ( Liu et al . , 2019 ) .

Recent VQA studies have been exploring higher level semantic representation of images , notably using graph-based structures for better image understanding , such as scene graph generation ( Xu et al . , 2017 ; Yang et al . , 2018 ) , visual relationship detection ( Yao et al . , 2018 ) , object counting ( Zhang et al . , 2018a ) , and relation reasoning ( Cao et al . , 2018 ; Li et al . , 2019 ; Cadene et al . , 2019a ) .

Representing images as graphs allows one to explicitly model interactions between two objects in an image , so as to seamlessly transfer information between graph nodes ( e.g. , objects in an image ) .

Very recent research methods ( Li et al . , 2019 ;

Cadene et al . , 2019a ; Yu et al . , 2019 ) have achieved remarkable performances , but there is still a big gap between them and human .

As shown in Figure 1 ( a ) , given an image of a group of persons and the corresponding question , a VQA system needs to not only recognize the objects in an image ( e.g. , batter , umpire and catcher ) , but also grasp the textual information in the question " what color is the umpire 's shirt " .

However , even many competitive VQA models struggle to process them accurately , and as a result predict the incorrect answer ( black ) rather than the correct answer ( blue ) , including the state-of-the-art methods .

Although the relations between two objects in an image have been considered , the attention-based VQA models lack building blocks to explicitly capture the syntactic dependency relations between words in a question .

As shown in Figure 1 ( c ) , these dependency relations can reflect which object is being asked ( e.g. , the word umpire 's modifies the word shirt ) and which aspect of the object is being asked ( e.g. , the word color is the direct object of the word is ) .

If a VQA model only knows the word shirt rather than the relation between words umpire 's and shirt in a question , it is difficult to distinguish which object is being asked .

In fact , we do need the modified relations to discriminate the correct object from multiple similar objects .

Therefore , we consider that it is necessary to explore the relations between words at linguistic level in addition to constructing the relations between objects at visual level .

Motivated by this , we propose a dual channel graph convolutional network ( DC-GCN ) to simultaneously capture the relations between objects in an image and the syntactic dependency relations between words in a question .

Our proposed DC-GCN model consists of an Image-GCN ( I-GCN ) module , a Question GCN ( Q-GCN ) module , and an attention alignment module .

The I-GCN module captures the relations between objects in an image , the Q-GCN module captures the syntactic dependency relations between words in a question , and the attention alignment module is used to align two representations of image and question .

The contributions of this work are summarized as follows :

1 ) We propose a dual channel graph convolutional network ( DC-GCN ) to simultaneously capture the visual and textual relations , and design the attention alignment module to align the multimodal representations , thus reducing the semantic gaps between vision and language .

2 ) We explore how to construct the syntactic dependency relations between words at linguistic level via graph convolutional networks as well as the relations between objects at visual level .

3 ) We conduct extensive experiments and ablation studies on VQA-v2 and VQA-CP-v2 datasets to examine the effectiveness of our DC-GCN model .

Experimental results show that the DC-GCN model achieves competitive performance with the state-of-the-art approaches .

In this paper , we propose a dual channel graph convolutional network to explore the relations between objects in an image and the syntactic dependency relations between words in a question .

Furthermore , we explicitly construct the relations between words by dependency tree and align the image and question representations by an attention alignment module to reduce the gaps between vision and language .

Extensive experiments on the VQA-v2 and VQA-CP-v2 datasets demonstrate that our model achieves comparable performance with the stateof-the-art approaches .

We will explore more complicated object relation modeling in future work .


This paper investigates model merging , a technique for deriving Markov models from text or speech corpora .

Models are derived by starting with a large and specific model and by successively combining states to build smaller and more general models .

We present methods to reduce the time complexity of the algorithm and report on experiments on deriving language models for a speech recognition task .

The experiments show the advantage of model merging over the standard bigram approach .

The merged model assigns a lower perplexity to the test set and uses considerably fewer states .

Hidden Markov Models are commonly used for statistical language models , e.g. in part-of-speech tagging and speech recognition ( Rabiner , 1989 ) .

The models need a large set of parameters which are induced from a ( text- ) corpus .

The parameters should be optimal in the sense that the resulting models assign high probabilities to seen training data as well as new data that arises in an application .

There are several methods to estimate model parameters .

The first one is to use each word ( type ) as a state and estimate the transition probabilities between two or three words by using the relative frequencies of a corpus .

This method is commonly used in speech recognition and known as word-bigram or word-trigram model .

The relative frequencies have to be smoothed to handle the sparse data problem and to avoid zero probabilities .

The second method is a variation of the first method .

Words are automatically grouped , e.g. by similarity of distribution in the corpus ( Pereira et al . , 1993 ) .

The relative frequencies of pairs or triples of groups ( categories , clusters ) are used as model parameters , each group is represented by a state in the model .

The second method has the advantage of drastically reducing the number of model parameters and thereby reducing the sparse data problem ; there is more data per group than per word , thus estimates are more precise .

The third method uses manually defined categories .

They are linguistically motivated and usually called parts-of-speech .

An important difference to the second method with automatically derived categories is that with the manual definition a word can belong to more than one category .

A corpus is ( manually ) tagged with the categories and transition probabilities between two or three categories are estimated from their relative frequencies .

This method is commonly used for part-of-speech tagging ( Church , 1988 ) .

The fourth method is a variation of the third method and is also used for part-of-speech tagging .

This method does not need a pre-annotated corpus for parameter estimation .

Instead it uses a lexicon stating the possible parts-of-speech for each word , a raw text corpus , and an initial bias for the transition and output probabilities .

The parameters are estimated by using the Baum-Welch algorithm ( Baum et al . , 1970 ) .

The accuracy of the derived model depends heavily on the initial bias , but with a good choice results are comparable to those of method three ( Cutting et al . , 1992 ) .

This paper investigates a fifth method for estimating natural language models , combining the advantages of the methods mentioned above .

It is suitable for both speech recognition and partof-speech tagging , has the advantage of automatically deriving word categories from a corpus and is capable of recognizing the fact that a word belongs to more than one category .

Unlike other techniques it not only induces transition and output probabilities , but also the model topology , i.e. , the number of states , and for each state the outputs that have a non-zero probability .

The method is called model merging and was introduced by ( Omohundro , 1992 ) .

The rest of the paper is structured as follows .

We first give a short introduction to Markov mo-dels and present the model merging technique .

Then , techniques for reducing the time complexity are presented and we report two experiments using these techniques .

We investigated model merging , a technique to induce Markov models from corpora ..

The original procedure is improved by introducing constraints and a different initial model .

The procedures are shown to be applicable to a transliterated speech corpus .

The derived models assign lower perplexities to test data than the standard bigram model derived from the same training corpus .

Additionally , the merged model was much smaller than the bigram model .

The experiments revealed a feature of model merging that allows for improvement of the method 's time complexity .

There is a large initial part of merges that do not change the model 's perplexity w.r.t , the test part , and that do not influence the final optimal model .

The time needed to derive a model is drastically reduced by abbreviating these initial merges .

Instead of starting with the trivial model , one can start with a smaller , easy-to-produce model , but one has to ensure that its size is still larger than the optimal model .


This paper presents a novel deterministic algorithm for implicit Semantic Role Labeling .

The system exploits a very simple but relevant discursive property , the argument coherence over different instances of a predicate .

The algorithm solves the implicit arguments sequentially , exploiting not only explicit but also the implicit arguments previously solved .

In addition , we empirically demonstrate that the algorithm obtains very competitive and robust performances with respect to supervised approaches that require large amounts of costly training data .

Traditionally , Semantic Role Labeling ( SRL ) systems have focused in searching the fillers of those explicit roles appearing within sentence boundaries ( Gildea and Jurafsky , 2000 , 2002 ; Carreras and Màrquez , 2005 ; Surdeanu et al . , 2008 ; Hajič et al . , 2009 ) .

These systems limited their searchspace to the elements that share a syntactical relation with the predicate .

However , when the participants of a predicate are implicit this approach obtains incomplete predicative structures with null arguments .

The following example includes the gold-standard annotations for a traditional SRL process :

[ arg0 The network ] had been expected to have The previous analysis includes annotations for the nominal predicate loss based on the NomBank structure ( Meyers et al . , 2004 ) .

In this case the annotator identifies , in the first sentence , the arguments arg0 , the entity losing something , arg1 , the thing lost , and arg3 , the source of that loss .

However , in the second sentence there is another instance of the same predicate , loss , but in this case no argument has been associated with it .

Traditional SRL systems facing this type of examples are not able to fill the arguments of a predicate because their fillers are not in the same sentence of the predicate .

Moreover , these systems also let unfilled arguments occurring in the same sentence , like in the following example :

( 2 ) Quest Medical Inc said it adopted [ arg1 a shareholders ' rights ]

[ np plan ] in which rights to purchase shares of common stock will be distributed as a dividend to shareholders of record as of Oct 23 .

For the predicate plan in the previous sentence , a traditional SRL process only returns the filler for the argument arg1 , the theme of the plan .

However , in both examples , a reader could easily infer the missing arguments from the surrounding context of the predicate , and determine that in ( 1 ) both instances of the predicate share the same arguments and in ( 2 ) the missing argument corresponds to the subject of the verb that dominates the predicate ,

Obviously , this additional annotations could contribute positively to its semantic analysis .

In fact , Gerber and Chai ( 2010 ) pointed out that implicit arguments can increase the coverage of argument structures in NomBank by 71 % .

However , current automatic systems require large amounts of manually annotated training data for each predicate .

The effort required for this manual annotation explains the absence of generally applicable tools .

This problem has become a main concern for many NLP tasks .

This fact explains a new trend to develop accurate unsupervised systems that exploit simple but robust linguistic principles ( Raghunathan et al . , 2010 ) .

In this work , we study the coherence of the predicate and argument realization in discourse .

In particular , we have followed a similar approach to the one proposed by Dahl et al . ( 1987 ) who filled the arguments of anaphoric mentions of nominal predicates using previous mentions of the same predicate .

We present an extension of this idea assuming that in a coherent document the different ocurrences of a predicate , including both verbal and nominal forms , tend to be mentions of the same event , and thus , they share the same argument fillers .

Following this approach , we have developed a deterministic algorithm that obtains competitive results with respect to supervised methods .

That is , our system can be applied to any predicate without training data .

The main contributions of this work are the following :

• We empirically prove that there exists a strong discourse relationship between the implicit and explicit argument fillers of the same predicates .

• We propose a deterministic approach that exploits this discoursive property in order to obtain the fillers of implicit arguments .

• We adapt to the implicit SRL problem a classic algorithm for pronoun resolution .

• We develop a robust algorithm , ImpAr , that obtains very competitive results with respect to existing supervised systems .

We release an open source prototype implementing this algorithm 1 .

Section 3 presents in detail the data used in our experiments .

Section 4 describes our algorithm for implicit argument resolution .

Section 5 presents some experiments we have carried out to test the algorithm .

Section 6 discusses the results obtained .

Finally , section 7 offers some concluding remarks and presents some future research lines .

In this work we have presented a robust deterministic approach for implicit Semantic Role Labeling .

The method exploits a very simple but relevant discoursive coherence property that holds over explicit and implicit arguments of closely related nominal and verbal predicates .

This property states that if several instances of the same predicate appear in a well-written discourse , it is very likely that they maintain the same argument fillers .

We have shown the importance of this phenomenon for recovering the implicit information about semantic roles .

To our knowledge , this is the first empirical study that proves this phenomenon .

Based on these observations , we have developed a new deterministic algorithm , ImpAr , that obtains very competitive and robust performances with respect to supervised approaches .

That is , it can be applied where there is no available manual annotations to train .

The code of this algorithm is publicly available and can be applied to any document .

As input it only needs the document with explicit semantic role labeling and Super-Sense annotations .

These annotations can be easily obtained from plain text using available tools7 , what makes this algorithm the first effective tool available for implicit SRL .

As it can be easily seen , ImpAr has a large margin for improvement .

For instance , providing more accurate spans for the fillers .

We also plan to test alternative approaches to solve the arguments without explicit antecedents .

For instance , our system can also profit from additional annotations like coreference , that has proved its utility in previous works .

Finally , we also plan to study our approach on different languages and datasets ( for instance , the SemEval-2010 dataset ) .


This paper describes an experimental comparison of seven different learning algorithms on the problem of learning to disambiguate the meaning of a word from context .

The algorithms tested include statistical , neural-network , decision-tree , rule-based , and case-based classification techniques .

The specific problem tested involves disambiguating six senses of the word " line " using the words in the current and proceeding sentence as context .

The statistical and neural-network methods perform the best on this particular problem and we discuss a potential reason for this observed difference .

We also discuss the role of bias in machine learning and its importance in explaining performance differences observed on specific problems .

Recent research in empirical ( corpus-based ) natural language processing has explored a number of different methods for learning from data .

Three general approaches are statistical , neural-network , and symbolic machine learning and numerous specific methods have been developed under each of these paradigms ( Wermter , Riloff , & Scheler , 1996 ; Charniak , 1993 ; Reilly & Sharkey , 1992 ) .

An important question is whether some methods perform significantly better than others on particular types of problems .

Unfortunately , there have been very few direct comparisons of alternative methods on identical test data .

A somewhat indirect comparison of applying stochastic context-free grammars ( Periera & Shabes , 1992 ) , a transformation-based method ( Brill , 1993 ) , and inductive logic programming ( Zelle & Mooney , 1994 ) to parsing the ATIS ( Airline Travel Information Service ) corpus from the Penn Treebank ( Marcus , Santorini , & Marcinkiewicz , 1993 ) indicates fairly similar performance for these three very different methods .

Also , comparisons of Bayesian , information-retrieval , neural-network , and case-based methods on word-sense disambiguation have also demonstrated similar performance ( Leacock , Towell , & Voorhees , 1993b ; Lehman , 1994 ) .

However , in a comparison of neural-network and decision-tree methods on learning to generate the past tense of an English verb , decision trees performed significantly better ( Ling & Marinov , 1993 ; Ling , 1994 ) .

Subsequent experiments on this problem have demonstrated that an inductive logic programming method produces even better results than decision trees ( Mooney & Califf , 1995 ) .

In this paper , we present direct comparisons of a fairly wide range of general learning algorithms on the problem of discriminating six senses of the word " line " from context , using data assembled by Leacock et al . ( 1993b ) .

We compare a naive Bayesian classifier ( Duda & Hart , 1973 ) , a perceptron ( Rosenblatt , 1962 ) , a decision-tree learner ( Quinlan , 1993 ) , a k nearest-neighbor classifier ( Cover & Hart , 1967 ) , logic-based DNF ( disjunctive normal form ) and CNF ( conjunctive normal form ) learners ( Mooney , 1995 ) and a decisionlist learner ( Rivest , 1987 ) .

Tests on all methods used identical training and test sets , and ten separate random trials were run in order to measure average performance and allow statistical testing of the significance of any observed differences .

On this particular task , we found that the Bayesian and perceptron methods perform significantly better than the remaining methods and discuss a potential reason for this observed difference .

We also discuss the role of bias in machine learning and its importance in explaining the observed differences in the performance of alternative methods on specific problems .

This paper has presented fairly comprehensive experiments comparing seven quite different empirical methods on learning to disambiguate words in context .

Methods that employ a weighted combination of a large set of features , such as simple Bayesian and neural-network methods , were shown to perform better than alternative methods such as decision-tree , rule-based , and instancebased techniques on the problem of disambiguating the word " line " into one of six possible senses given the words that appear in the current and previous sentence as context .

Although different learning algorithms can frequently perform quite similarly , they all have specific biases in their representation of concepts and therefore can illustrate both strengths and weaknesses in particular applications .

Only rigorous experimental comparisons together with a qualitative analysis and explanation of their results can help determine the appropriate methods for particular problems in natural language processing .


The development of over-parameterized pretrained language models has made a significant contribution toward the success of natural language processing .

While over-parameterization of these models is the key to their generalization power , it makes them unsuitable for deployment on low-capacity devices .

We push the limits of state-of-the-art Transformer-based pre-trained language model compression using Kronecker decomposition .

We present our KroneckerBERT , a compressed version of the BERT BASE model obtained by compressing the embedding layer and the linear mappings in the multi-head attention , and the feed-forward network modules in the Transformer layers .

Our KroneckerBERT is trained via a very efficient two-stage knowledge distillation scheme using far fewer data samples than state-of-the-art models like MobileBERT and TinyBERT .

We evaluate the performance of KroneckerBERT on well-known NLP benchmarks .

We show that our KroneckerBERT with compression factors of 7.7× and 21× outperforms state-of-theart compression methods on the GLUE and SQuAD benchmarks .

In particular , using only 13 % of the teacher model parameters , it retain more than 99 % of the accuracy on the majority of GLUE tasks .

In recent years , the emergence of Pre-trained Language Models ( PLMs ) has led to a significant breakthrough in Natural Language Processing ( NLP ) .

The introduction of Transformers and unsupervised pre-training on enormous unlabeled data are the two main factors that contribute to this success .

Transformer-based models ( Devlin et al . , 2018 ; Radford et al . , 2019 ; Yang et al . , 2019 ; Shoeybi et al . , 2019 ) are powerful yet highly overparameterized .

The enormous size of these models does not meet the constraints imposed by edge devices on memory , latency , and energy consumption .

Therefore there has been a growing interest in developing new methodologies and frameworks for the compression of these large PLMs .

Similar to other deep learning models , the main directions for the compression of these models include low-bit quantization ( Gong et al . , 2014 ; Prato et al . , 2019 ) , network pruning ( Han et al . , 2015 ) , matrix decomposition ( Yu et al . , 2017 ; Lioutas et al . , 2020 ) and Knowledge distillation ( KD ) ( Hinton et al . , 2015 ) .

These methods are either used in isolation or in combination to improve compression-performance trade-off .

Recent works have been relatively successful in compressing Transformer-based PLMs to a certain degree ( Sanh et al . , 2019 ; Sun et al . , 2019 ; Jiao et al . , 2019 ; Sun et al . , 2020 ; Xu et al . , 2020 ; Wang et al . , 2020 ; Kim et al . , 2021 ) ; however , moderate and extreme compression of these models ( compression factors > 5 and 10 resepctively ) is still quite challenging .

In particular , several works ( Mao et al . , 2020 ; Zhao et al . , 2019a Zhao et al . , , 2021 ) ) that have tried to go beyond the compression factor of 10 , have done so at the expense of a significant drop in performance .

Following the classical assumption that matrices often follow a low-rank structure , low-rank decomposition methods have been used for compression of weight matrices in deep learning models ( Yu et al . , 2017 ; Swaminathan et al . , 2020 ; Winata et al . , 2019 ) and especially Transformer-based models ( Noach and Goldberg , 2020 ; Mao et al . , 2020 ) .

However , low-rank decomposition methods only exploit redundancies of the weight matrix in the horizontal and vertical dimensions and thus limit the flexibility of the compressed model .

Kronecker decomposition on the other hand exploits redun- dancies in predefined patches and hence allows for more flexibility in their representation .

Recent works prove Kronecker product to be more effective in retaining accuracy after compression than SVD ( Thakker et al . , 2019 ) .

This work proposes a novel framework that uses Kronecker decomposition for compression of Transformer-based PLMs and provides a very promising compression-performance trade-off for medium and high compression levels , with 13 % and 5 % of the original model parameters respectively .

We use Kronecker decomposition for the compression of both Transformer layers and the embedding layer .

For Transformer layers , the compression is achieved by representing every weight matrix both in the multi-head attention ( MHA ) and the feed-forward neural network ( FFN ) as a Kronecker product of two smaller matrices .

We also propose a Kronecker decomposition for compression of the embedding layer .

Previous works have tried different techniques to reduce the enormous memory consumption of this layer ( Khrulkov et al . , 2019 ; Li et al . , 2018 ) .

Our Kronecker decomposition method can substantially reduce the amount of required memory while maintaining low computation .

Using Kronecker decomposition for large compression factors leads to a reduction in the model expressiveness .

This is due to the nature of the Kronecker product and the fact that elements in this representation are tied together .

To address this issue , we propose to distill knowledge from the intermediate layers of the original uncompressed network to the Kronecker network during training .

Training of the state-of-the art BERT compression models ( Zhao et al . , 2019a , b ; Sun et al . , 2020

al . , , 2019 ) ) involve an extensive training which requires vast computational resources .

For example in ( Sun et al . , 2020 ) , first a specially designed teacher , i.e IB-BERT LARGE is trained from scratch on the en-tire English wikipedia and Book Corpus .

The student is then pretrained on the same corpus via KD while undergoing an additional progressive KD phase .

Another example is TinyBERT ( Jiao et al . , 2019 ) which requires pretraining on the entire English Wikipedia and also uses extensive data augmentation ( 20× ) for fine-tuning on the downstream tasks .

We show that our Kronecker BERT can out perform state-of-the-art with significantly less training requirements .

More precisely , our Kronecker-BERT model undergoes a very light pretraining on only 10 % of the English Wikipedia for 3 epochs followed by finetuning on the original downstream data .

Note that , while our evaluations in this work are limited to BERT , this proposed compression method can be directly used to compress other Transformer-based NLP models .

The main contributions of this paper are as follows :

• Compression of the embedding layer using the Kronecker decomposition with very low computational overhead .

We introduced a novel method for compressing Transformer-based language models that uses Kronecker decomposition for the compression of the embedding layer and the linear mappings within the Transformer blocks .

The proposed framework was used to compress the BERT BASE model .

We used a very light two-stage KD method to train the compressed model .

We show that the proposed framework can significantly reduce the size and the number of computations while outperforming stateof-the-art .

The proposed method can be directly applied for compression of other Transformer-based language models .

The combination of the proposed method with other compression techniques such layer truncation , pruning and quantization can be an interesting direction for future work .


Recent causal probing literature reveals when language models and syntactic probes use similar representations .

Such techniques may yield " false negative " causality results : models may use representations of syntax , but probes may have learned to use redundant encodings of the same syntactic information .

We demonstrate that models do encode syntactic information redundantly and introduce a new probe design that guides probes to consider all syntactic information present in embeddings .

Using these probes , we find evidence for the use of syntax in models where prior methods did not , allowing us to boost model performance by injecting syntactic information into representations .

Here , we identify a limitation of prior causal probing art in which redundant information in embeddings could lead to probes and models using different representations of the same information , which in turn could lead to uninformative causal analysis results .

We propose a new probe architecture that addresses this limitation by encouraging probes to use all sources of information in embeddings .

Recent large neural models like BERT and GPT-3 exhibit impressive performance on a large variety of linguistic tasks , from sentiment analysis to question-answering ( Devlin et al . , 2019 ; Brown et al . , 2020 ) .

Given the models ' impressive performance , but also their complexity , researchers have developed tools to understand what patterns models have learned .

In probing literature , researchers develop " probes : " models designed to extract information from the representations of trained models ( Linzen et al . , 2016 ; Conneau et al . , 2018 ; Hall Maudslay et al . , 2020 ) .

For example , Hewitt and Manning ( 2019 ) demonstrated that one can train accurate linear classifiers to predict syntactic structure from BERT or ELMO embeddings .

These probes reveal what information is present in model embeddings but not how or if models use that information ( Belinkov , 2021 ) .

To address this gap , new research in causal analysis seeks to understand how aspects of models ' representations affect their behavior ( Elazar et al . , 2020 ; Ravfogel et al . , 2020 ; Giulianelli et al . , 2018 ; Tucker et al . , 2021 ; Feder et al . , 2021 ) .

Typically , these techniques create counterfactual representations that differ from the original according to some Figure 1 : In a 2D embedding space , a model might redundantly encode syntactic representations of a sentence like " the girl saw the boy with the telescope . "

Redundant encodings could cause misalignment between the model 's decision boundary ( blue ) and a probe 's ( red ) .

We introduce dropout probes ( green ) to use all informative dimensions .

property ( e.g. , syntactic interpretation of the sentence ) .

Researchers then compare outputs when using original and counterfactual embeddings to assess whether a property encoded in the representation is causally related to model behavior .

Unfortunately , negative results -wherein researchers report that models do not appear to use a property causally -are difficult to interpret .

Such failures can be attributed to a model truly not using the property ( true negatives ) , or to a failure of the technique ( false negatives ) .

For example , as depicted in Figure 1 , if a language model encodes syntactic information redundantly ( here illustrated in two-dimensions ) , the model and probe may differentiate among parses along orthogonal dimensions .

When creating counterfactual representations with such probes , researchers could incorrectly conclude that the model does not use syntactic information .

In this work , we present new evidence for the causal use of syntactic representations on task performance in BERT , using newly-designed probes that take into account the potential redundancy in a model 's internal representation .

First , we find evidence for representational redundancy in BERTbased models .

Based on these findings , we propose a new probe design that encourages the probe to use all relevant representations of syntax in model embeddings .

These probes are then used to assess if language models use representations of syntax causally , and , unlike prior art , we find that some fine-tuned models do exhibit signatures of causal use of syntactic information .

Lastly , having found that these models causally use representations of syntax , we used our probes to boost a questionanswering model 's performance by " injecting " syntactic information at test time.1 2 Related Work

In this work , we designed and evaluated " dropout probes , " a new neural probing architecture for generating useful causal analysis of trained language models .

Our technical contribution -adding a dropout layer before probes -was inspired by a theory of redundant syntactic encodings in models .

Our results fit within three categories : we showed that 1 ) models encoded syntactic information redundantly , 2 ) dropout probes , unlike standard probes , revealed that QA models used syntactic representations causally , and 3 ) by injecting syntactic information at test time in syntacticallychallenging domains , we could increase model performance without retraining .

Despite our step towards better understanding of pretrained models , future work remains .

Natural extensions include studying pretrained models beyond those considered in this work , further research into redundancy in embeddings , more investigation into inserting symbolic knowledge into neural representations , and new methods for training models to respond appropriately to interventions .


Online discussion forums are a popular platform for people to voice their opinions on any subject matter and to discuss or debate any issue of interest .

In forums where users discuss social , political , or religious issues , there are often heated debates among users or participants .

Existing research has studied mining of user stances or camps on certain issues , opposing perspectives , and contention points .

In this paper , we focus on identifying the nature of interactions among user pairs .

The central questions are : How does each pair of users interact with each other ?

Does the pair of users mostly agree or disagree ?

What is the lexicon that people often use to express agreement and disagreement ?

We present a topic model based approach to answer these questions .

Since agreement and disagreement expressions are usually multiword phrases , we propose to employ a ranking method to identify highly relevant phrases prior to topic modeling .

After modeling , we use the modeling results to classify the nature of interaction of each user pair .

Our evaluation results using real-life discussion / debate posts demonstrate the effectiveness of the proposed techniques .

Online discussion / debate forums allow people with common interests to freely ask and answer questions , to express their views and opinions on any subject matter , and to discuss issues of common interest .

A large part of such discussions is about social , political , and religious issues .

On such issues , there are often heated discussions / debates , i.e. , people agree or disagree and argue with one another .

Such ideological discussions on a myriad of social and political issues have practical implications in the fields of communication and political science as they give social scientists an opportunity to study real-life discussions / debates of almost any issue and analyze participant behaviors in a large scale .

In this paper , we present such an application , which aims to perform fine-grained analysis of user-interactions in online discussions .

There have been some related works that focus on discovering the general topics and ideological perspectives in online discussions ( Ahmed and Xing , 2010 ) , placing users in support / oppose camps ( Agarwal et al . , 2003 ) , and classifying user stances ( Somasundaran and Wiebe , 2009 ) .

However , these works are at a rather coarser level and have not considered more fine-grained characteristics of debates / discussions where users interact with each other by quoting / replying each other to express agreement or disagreement and argue with one another .

In this work , we want to mine the following information :

The nature of interaction of each pair of users or participants who have engaged in the discussion of certain issues , i.e. , whether the two persons mostly agree or disagree with each other in their interactions .

What language expressions are often used to express agreement ( e.g. , " I agree " and " you 're right " ) and disagreement ( e.g. , " I disagree " and " you speak nonsense " ) .

We note that although agreement and disagreement expressions are distinct from traditional sentiment expressions ( words and phrases ) such as good , excellent , bad , and horrible , agreement and disagreement clearly express a kind of sentiment as well .

They are usually emitted during interactive exchanges of arguments in ideological discussions .

This idea prompted us to introduce the concept of ADsentiment .

We define the polarity of agreement expressions as positive and the polarity of disagreement expressions as negative .

We refer agreement and disagreement expressions as ADsentiment expressions , or AD-expressions for short .

AD-expressions are crucial for the analysis of interactive discussions and debates just as sentiment expressions are instrumental in sentiment analysis ( Liu , 2012 ) .

We thus regard this work as an extension to traditional sentiment analysis ( Pang and Lee , 2008 ; Liu , 2012 ) .

In our earlier work ( Mukherjee and Liu , 2012a ) , we proposed three topic models to mine contention points , which also extract ADexpressions .

In this paper , we further improve the work by coupling an information retrieval method to rank good candidate phrases with topic modeling in order to discover more accurate ADexpressions .

Furthermore , we apply the resulting AD-expressions to the new task of classifying the arguing or interaction nature of each pair of users .

Using discovered AD-expressions for classification has an important advantage over traditional classification because they are domain independent .

We employ a semi-supervised generative model called JTE-P to jointly model AD-expressions , pair interactions , and discussion topics simultaneously in a single framework .

With such complex interactions mined , we can produce many useful summaries of discussions .

For example , we can discover the most contentious pairs for each topic and ideological camps of participants , i.e. , people who often agree with each other are likely to belong to the same camp .

The proposed framework also facilitates tracking users ' ideology shifts and the resulting arguing nature .

The proposed methods have been evaluated both qualitatively and quantitatively using a large number of real-life discussion / debate posts from four domains .

Experimental results show that the proposed model is highly effective in performing its tasks and outperforms several baselines .

This paper studied the problem of modeling user pair interactions in online discussions with the purpose of discovering the interaction or arguing nature of each author pair and various ADexpressions emitted in debates .

A novel technique was also proposed to rank n-gram phrases where relevance based ranking was used in conjunction with a semi-supervised generative model .

This method enables us to find better ADexpressions .

Experiments using real-life online debate data showed the effectiveness of the model .

In our future work , we intend to extend the model to account for stances , and issue specific interactions which would pave the way for user profiling and behavioral modeling .


We present an exploration of generative probabilistic models for multi-document summarization .

Beginning with a simple word frequency based model ( Nenkova and Vanderwende , 2005 ) , we construct a sequence of models each injecting more structure into the representation of document set content and exhibiting ROUGE gains along the way .

Our final model , HIERSUM , utilizes a hierarchical LDA-style model ( Blei et al . , 2004 ) to represent content specificity as a hierarchy of topic vocabulary distributions .

At the task of producing generic DUC-style summaries , HIERSUM yields state-of-the-art ROUGE performance and in pairwise user evaluation strongly outperforms Toutanova et al . ( 2007 ) 's state-of-the-art discriminative system .

We also explore HIERSUM 's capacity to produce multiple ' topical summaries ' in order to facilitate content discovery and navigation .

Over the past several years , there has been much interest in the task of multi-document summarization .

In the common Document Understanding Conference ( DUC ) formulation of the task , a system takes as input a document set as well as a short description of desired summary focus and outputs a word length limited summary .

1 To avoid the problem of generating cogent sentences , many systems opt for an extractive approach , selecting sentences from the document set which best reflect its core content .

There are several approaches to modeling document content : simple word frequency-based methods ( Luhn , 1958 ; Nenkova and Vanderwende , 2005 ) , graph-based approaches ( Radev , 2004 ; Wan and Yang , 2006 ) , as well as more linguistically motivated techniques ( Mckeown et al . , 1999 ; Leskovec et al . , 2005 ; Harabagiu et al . , 2007 ) .

Another strand of work ( Barzilay and Lee , 2004 ; Daumé III and Marcu , 2006 ; Eisenstein and Barzilay , 2008 ) , has explored the use of structured probabilistic topic models to represent document content .

However , little has been done to directly compare the benefit of complex content models to simpler surface ones for generic multi-document summarization .

In this work we examine a series of content models for multi-document summarization and argue that LDA-style probabilistic topic models ( Blei et al . , 2003 ) can offer state-of-the-art summarization quality as measured by automatic metrics ( see section 5.1 ) and manual user evaluation ( see section 5.2 ) .

We also contend that they provide convenient building blocks for adding more structure to a summarization model .

In particular , we utilize a variation of the hierarchical LDA topic model ( Blei et al . , 2004 ) to discover multiple specific ' subtopics ' within a document set .

The resulting model , HIERSUM ( see section 3.4 ) , can produce general summaries as well as summaries for any of the learned sub-topics .

In this paper we have presented an exploration of content models for multi-document summarization and demonstrated that the use of structured topic models can benefit summarization quality as measured by automatic and manual metrics .


Simultaneous Machine Translation is the task of incrementally translating an input sentence before it is fully available .

Currently , simultaneous translation is carried out by translating each sentence independently of the previously translated text .

More generally , Streaming MT can be understood as an extension of Simultaneous MT to the incremental translation of a continuous input text stream .

In this work , a state-of-the-art simultaneous sentencelevel MT system is extended to the streaming setup by leveraging the streaming history .

Extensive empirical results are reported on IWSLT Translation Tasks , showing that leveraging the streaming history leads to significant quality gains .

In particular , the proposed system proves to compare favorably to the best performing systems .

Simultaneous Machine Translation ( MT ) is the task of incrementally translating an input sentence before it is fully available .

Indeed , simultaneous MT can be naturally understood in the scenario of translating a text stream as a result of an upstream Automatic Speech Recognition ( ASR ) process .

This setup defines a simultaneous Speech Translation ( ST ) scenario that is gaining momentum due to the vast number of industry applications that could be exploited based on this technology , from person-toperson communication to subtitling of audiovisual content , just to mention two main applications .

These real-world streaming applications motivate us to move from simultaneous to streaming MT , understanding streaming MT as the task of simultaneously translating a potentially unbounded and unsegmented text stream .

Streaming MT poses two main additional challenges over simultaneous MT .

First , the MT system must be able to leverage the streaming history beyond the sentence level both at training and inference time .

Second , the system must work under latency constraints over the entire stream .

With regard to exploiting streaming history , or more generally sentence context , it is worth mentioning the significant amount of previous work in offline MT at sentence level ( Tiedemann and Scherrer , 2017 ; Agrawal et al . , 2018 ) , document level ( Scherrer et al . , 2019 ; Ma et al . , 2020a ; Zheng et al . , 2020b ; Li et al . , 2020 ; Maruf et al . , 2021 ; Zhang et al . , 2021 ) , and in related areas such as language modelling ( Dai et al . , 2019 ) that has proved to lead to quality gains .

Also , as reported in ( Li et al . , 2020 ) , more robust ST systems can be trained by taking advantage of the context across sentence boundaries using a data augmentation strategy similar to the prefix training methods proposed in ( Niehues et al . , 2018 ; Ma et al . , 2019 ) .

This data augmentation strategy was suspected to boost re-translation performance when compared to conventional simultaneous MT systems ( Arivazhagan et al . , 2020 ) .

Nonetheless , with the notable exception of ( Schneider and Waibel , 2020 ) , sentences in simultaneous MT are still translated independently from each other ignoring the streaming history .

( Schneider and Waibel , 2020 ) proposed an end-to-end streaming MT model with a Transformer architecture based on an Adaptive Computation Time method with a monotonic encoderdecoder attention .

This model successfully uses the streaming history and a relative attention mechanism inspired by Transformer-XL ( Dai et al . , 2019 ) .

Indeed , this is an MT model that sequentially translates the input stream without the need for a segmentation model .

However , it is hard to interpret the latency of their streaming MT model because the authors observe that the current sentence-level latency measures , Average Proportion ( AP ) ( Cho and Esipova , 2016 ) , Average Lagging ( AL ) ( Ma et al . , 2019 ) and Differentiable Average Lagging ( DAL ) ( Cherry and Foster , 2019 ) do not perform well on a streaming setup .

This fact is closely related to the second challenge mentioned above , which is that the system must work under latency constraints over the entire stream .

Indeed , current sentence-level latency measures do not allow us to appropriately gauge the latency of streaming MT systems .

To this purpose , ( Iranzo-Sánchez et al . , 2021 ) recently proposed a stream-level adaptation of the sentence-level latency measures based on the conventional re-segmentation approach applied to the ST output in order to evaluate translation quality ( Matusov et al . , 2005 ) .

In this work , the simultaneous MT model based on a unidirectional encoder-decoder and training along multiple wait-k paths proposed by ( Elbayad et al . , 2020a ) is evolved into a streamingready simultaneous MT model .

To achieve this , model training is performed following a sentenceboundary sliding-window strategy over the parallel stream that exploits the idea of prefix training , while inference is carried out in a single forward pass on the source stream that is segmented by a Direct Segmentation ( DS ) model ( Iranzo-Sánchez et al . , 2020 ) .

In addition , a refinement of the unidirectional encoder-decoder that takes advantage of longer context for encoding the initial positions of the streaming MT process is proposed .

This streaming MT system is thoroughly assessed on IWSLT translation tasks to show how leveraging the streaming history provides systematic and significant BLEU improvements over the baseline , while reported stream-adapted latency measures are fully consistent and interpretable .

Finally , our system favourably compares in terms of translation quality and latency to the latest state-of-the-art simultaneous MT systems ( Ansari et al . , 2020 ) .

Next section provides a formal framework for streaming MT to accommodate streaming history in simultaneous MT .

Section 3 presents the streaming experimental setup whose results are reported and discussed in Section 4 .

Finally , conclusions and future work are drawn in Section 5 .

In this work , a formalization of streaming MT as a generalization of simultaneous MT has been proposed in order to define a theoretical framework in which our two contributions have been made .

On the one hand , we successfully leverage streaming history across sentence boundaries for a simultaneous MT system based on multiple wait-k paths that allows our system to greatly improve the results of the sentence-level baseline .

On the other hand , our PBE is able to take into account longer context information than its unidirectional counterpart , while keeping the same training efficiency .

Our proposed MT system has been evaluated under a realistic streaming setting being able to reach similar translation quality than a state-of-theart segmentation-free streaming MT system at a fraction of its latency .

Additionally , our system has been shown to be competitive when compared with state-of-the-art simultaneous MT systems optimized for sentence-level translation , obtaining excellent results using a single model across a wide range of latency levels , thanks to its flexible inference policy .

In terms of future work , additional training and inference procedures that take advantage of the streaming history in streaming MT are still open for research .

One important avenue of improvement is to devise more robust training methods , so that simultaneous models can perform as well as their offline counterparts when carrying out inference at higher latencies .

The segmentation model , though proved useful in a streaming setup , adds complexity and can greatly affect translation quality .

Thus , the development of segmentation-free streaming MT models is another interesting research topic .


This paper presents a statistical model which trains from a corpus annotated with Part-Of-Speech tags and assigns them to previously unseen text with state-of-the-art accuracy(96.6 % ) .

The model can be classified as a Maximum Entropy model and simultaneously uses many contextual " features " to predict the POS tag .

Furthermore , this paper demonstrates the use of specialized features to model difficult tagging decisions , discusses the corpus consistency problems discovered during the implementation of these features , and proposes a training strategy that mitigates these problems .

Many natural language tasks require the accurate assignment of Part-Of-Speech ( POS ) tags to previously unseen text .

Due to the availability of large corpora which have been manually annotated with POS information , many taggers use annotated text to " learn " either probability distributions or rules and use them to automatically assign POS tags to unseen text .

The experiments in this paper were conducted on the Wall Street Journal corpus from the Penn Treebank project ( Marcus et al . , 1994 ) , although the model can trai~n from any large corpus annotated with POS tags .

Since most realistic natural language applications must process words that were never seen before in training data , all experiments in this paper are conducted on test data that include unknown words .

Several recent papers ( Brill , 1994 , Magerman , 1995 ) have reported 96.5 % tagging accuracy on the Wall St. Journal corpus .

The experiments in this paper test the hypothesis that better use of context will improve the accuracy .

A Maximum Entropy model is well-suited for such experiments since it corn-bines diverse forms of contextual information in a principled manner , and does not impose any distributional assumptions on the training data .

Previous uses of this model include language modeling ( Lau et al . , 1993 ) , machine translation ( Berger et al . , 1996 ) , prepositional phrase attachment ( Ratnaparkhi et al . , 1994 ) , and word morphology ( Della Pietra et al . , 1995 ) .

This paper briefly describes the maximum entropy and maximum likelihood properties of the model , features used for POS tagging , and the experiments on the Penn Treebank Wall St. Journal corpus .

It then discusses the consistency problems discovered during an attempt to use specialized features on the word context .

Lastly , the results in this paper are compared to those from previous work on POS tagging .

The Maximum Entropy model is an extremely flexible technique for linguistic modelling , since it can use a virtually unrestricted and rich feature set in the framework of a probability model .

The implementation in this paper is a state-of-the-art POS tagger , as evidenced by the 96.6 % accuracy on the unseen Test set , shown in Table 11 .

The model with specialized features does not perform much better than the baseline model , and further discovery or refinement of word-based features is difficult given the inconsistencies in the training data .

A model trained and tested on data from a single annotator performs at .5 % higher accuracy than the baseline model and should produce more consistent input for applications that require tagged text .


Neural-based summarization models suffer from the length limitation of text encoder .

Long documents have to been truncated before they are sent to the model , which results in huge loss of summary-relevant contents .

To address this issue , we propose the sliding selector network with dynamic memory for extractive summarization of long-form documents , which employs a sliding window to extract summary sentences segment by segment .

Moreover , we adopt memory mechanism to preserve and update the history information dynamically , allowing the semantic flow across different windows .

Experimental results on two large-scale datasets that consist of scientific papers demonstrate that our model substantially outperforms previous state-of-the-art models .

Besides , we perform qualitative and quantitative investigations on how our model works and where the performance gain comes from .

Text summarization is an important task of natural language processing which aims to distil salient contents from a textual document .

Existing summarization models can be roughly classified into two categories , which are abstractive and extractive .

Abstractive summarization usually adopts natural language generation technology to produce a wordby-word summary .

In general , these approaches are flexible but may yield disfluent summaries ( Liu and Lapata , 2019a ) .

By comparison , extractive approaches aim to select a subset of the sentences in the source document , thereby enjoying better fluency and efficiency ( Cao et al . , 2017 ) .

Although many summarization approaches have demonstrated their success on relatively short documents , such as news articles , they usually fail Paragraph 1 : Medical tourism is illustrated as occurrence in which individuals travel abroad to receive healthcare services .

It is a multibillion dollar industry and countries like India , Thailand , Israel , Singapore , … Paragraph 2 : The prime driving factors in medical tourism are increased medical costs , increased insurance premiums , increasing number of uninsured or partially insured individuals in developed countries , … … …

Paragraph 5 : It is generally presumed in marketing that products with similar characteristics will be equally preferred by the consumers , however , attributes , which make the product similar to other products , will not … .

to achieve desired performance when directly applied in long-form documents , such as scientific papers .

This inferior performance is partly due to the truncation operation , which inevitably leads to information loss , especially for extractive models because parts of gold sentences would be inaccessible .

In addition , the accurate modeling of long texts remains a challenge ( Frermann and Klementiev , 2019 ) .

A practical solution for this problem is to use a sliding window to process documents separately .

This approach is used in other NLP tasks , such as machine reading comprehension ( Wang et al . , 2019b ) .

However , such a paradigm is not suitable for summarization task because the concatenation of summaries that are independently extracted from local contexts is usually inconsistent with the gold summary of the entire document .

Figure 1 shows an example to illustrate this problem .

The core topic of the source document is " medical tourism , " which is discussed in Paragraphs 1 and 2 .

How-ever , the 5-th paragraph is mainly about " consumer and product . "

As a consequence , the paragraphby-paragraph extraction approach might produce a both repetitive and noisy summary .

Under this circumstance , the supervised signals will have a negative effect on model behaviors because understanding why Paragraph 5 should output an empty result without information conveying from previous texts is confused for the model .

In this paper , we propose a novel extractive summarization model for long-form documents .

We split the input document into multiple windows and encode them with a sliding encoder sequentially .

During this process , we introduce a memory to preserve salient information learned from previous windows , which is used to complete and enrich local texts .

Intuitively , our model has the following advantages : 1 ) In each window , the text encoder processes a relatively short segment , thereby yielding more accurate representations .

The local text representations can capture beyond-window contextual information via the memory module .

The previous selection results are also parameterized in the memory block , allowing the collaboration among summary sentences .

To sum up , our contributions are threefold .

( 1 ) We propose a novel extractive summarization model that can summarize documents of arbitrary length without truncation loss .

Also , it employs the memory mechanism to address context fragmentation .

To the best of our knowledge , we are the first to propose applying memory networks into extractive text summarization task .

( 2 ) The proposed framework ( i.e. , a sliding encoder combined with dynamic memory ) provides a general solution for summarizing long documents and can be easily extended to other abstractive and extractive summarization models .

( 3 ) Our model achieves the state-of-the-art results on two widely used datasets for long document summarization .

Moreover , we conduct extensive analysis to understand how our model works and where the performance gain comes from .

In this study , we propose a novel extractive summarization that can summarize long-form documents without content loss .

We conduct extensive experiments on two well-studied datasets that consist of scientific papers .

Experimental results demonstrate that our model outperforms previous stateof-the-art models .

In the future , we will extend our framework ( i.e. , a sliding encoder combined with long-range memory modeling ) to abstractive summarization models .


In this paper , we present work on extracting social networks from unstructured text .

We introduce novel features derived from semantic annotations based on FrameNet .

We also introduce novel semantic tree kernels that help us improve the performance of the best reported system on social event detection and classification by a statistically significant margin .

We show results for combining the models for the two aforementioned subtasks into the overall task of social network extraction .

We show that a combination of features from all three levels of abstractions ( lexical , syntactic and semantic ) are required to achieve the best performing system .

Social network extraction from text has recently been gaining a considerable amount of attention ( Agarwal and Rambow , 2010 ; Elson et al . , 2010 ; Agarwal et al . , 2013a ;

Agarwal et al . , 2013b ; He et al . , 2013 ) .

One of the reason for this attention , we believe , is that being able to extract social networks from unstructured text may provide a powerful new tool for historians , political scientists , scholars of literature , and journalists to analyze large collections of texts around entities and their interactions .

The tool would allow researchers to quickly extract networks and assess their size , nature , and cohesiveness , a task that would otherwise be impossible with corpora numbering millions of documents .

It would also make it possible to make falsifiable claims about these networks , bringing the experimental method to disciplines like history , where it is still relatively rare .

In our previous work ( Agarwal et al . , 2010 ) , we proposed a definition of a network based on interactions : nodes are entities and links are social events .

We defined two broad types of links : one-directional links ( one person thinking about or talking about another person ) and bi-directional links ( two people having a conversation , a meeting , etc . ) .

For example , in the following sentence , we would add two links to the network : a one-directional link between Toujan Faisal and the committee , triggered by the word said ( because Toujan is talking about the committee ) and a bi-directional link between the same entities triggered by the word informed ( a mutual interaction ) .

( 1 ) [ Toujan Faisal ] , 54 , said [ she ] was informed of the refusal by an [ Interior Ministry committee ] overseeing election preparations .

In this paper , we extract networks using the aforementioned definition of social networks .

We introduce and add tree kernel representations and features derived from frame-semantic parses to our previously proposed system .

Our results show that hand-crafted frame semantic features , which are linguistically motivated , add less value to the overall performance in comparison with the frame-semantic tree kernels .

We believe this is due to the fact that hand-crafted features require frame parses to be highly accurate and complete .

In contrast , tree kernels are able to find and leverage less strict patterns without requiring the semantic parse to be entirely accurate or complete .

Apart from introducing semantic features and tree structures , we evaluate on the task of social network extraction , which is a combination of two sub-tasks : social event detection and social event classification .

In our previous work ( Agarwal and Rambow , 2010 ) , we presented results for the two sub-tasks , but no evaluation was presented for the task of social network extraction .

We experiment with two different designs of combining models for the two sub-tasks : 1 ) One-versus-All and 2 ) Hierarchical .

We find that the hierarchical design outperforms the more commonly used Oneversus-All by a statistically significant margin .

Following are the contributions of this paper :

We design and propose novel frame semantic features and tree-based representations and show that tree kernels are well suited to work with noisy semantic parses .

We show that in order to achieve the best performing system , we need to include features and tree structures from all levels of abstractions , lexical , syntactic , and semantic , and that the convolution kernel framework is well-suited for creating such a combination .

We combine the previously proposed subtasks ( social event detection and classification ) into a single task , social network extraction , and show that combining the models using a hierarchical design is significantly better than the one-versus-all design .

The rest of the paper is structured as follows :

In Section 2 , we give a precise definition of the task and describe the data .

In Section 3 , we give a brief overview of frame semantics and motivate the need to use frame semantics for the tasks addressed in this paper .

In Section 4 , we present semantic features and tree kernel representations designed for the tasks .

In Section 5 , we briefly review tree kernels and support vector machines ( SVM ) .

In Section 6 we present experiments and discuss the results .

We conclude and give future directions of work in Section 8 .

This work has only scratched the surface of possibilities for using frame semantic features and tree structures for the task of social event extraction .

We have shown that tree kernels are well suited to work with possibly inaccurate semantic parses in contrast to hand-crafted features that require the semantic parses to be completely accurate .

We have also extended our previous work by designing and evaluating a full system for social network extraction .

A more natural data representation for semantic parses is a graph structure .

We are actively exploring the design of semantic graph structures that may be brought to bear with the use of graph kernels ( Vishwanathan et al . , 2010 ) .


User-generated textual data is rich in content and has been used in many user behavioral modeling tasks .

However , it could also leak user private-attribute information that they may not want to disclose such as age and location .

User 's privacy concerns mandate data publishers to protect privacy .

One effective way is to anonymize the textual data .

In this paper , we study the problem of textual data anonymization and propose a novel Reinforcement Learning-based Text Anonymizor , RLTA , which addresses the problem of private-attribute leakage while preserving the utility of textual data .

Our approach first extracts a latent representation of the original text w.r.t .

a given task , then leverages deep reinforcement learning to automatically learn an optimal strategy for manipulating text representations w.r.t .

the received privacy and utility feedback .

Experiments show the effectiveness of this approach in terms of preserving both privacy and utility .

Social media users generate a tremendous amount of data such as profile information , network connections and online reviews and posts .

Online vendors use this data to understand users preferences and further predict their future needs .

However , user-generated data is rich in content and malicious attackers can infer users ' sensitive information .

AOL search data leak in 2006 is an example of privacy breaches which results in users re-identification according to the published AOL search logs and queries ( Pass et al . , 2006 ) .

Therefore , these privacy concerns mandate that data be anonymized before publishing .

Recent research has shown that textual data alone may contain sufficient information about users ' private-attributes that they do not want to disclose such as age , gender , location , political views and sexual orienta-tion ( Mukherjee and Liu , 2010 ; Volkova et al . , 2015 ) .

Little attention has been paid to protect users textual information ( Li et al . , 2018 ; Zhang et al . , 2018 ; Anandan et al . , 2012 ; Saygin et al . , 2006 ) .

Anonymizing textual information comes at the cost of losing utility of data for future applications .

Some existing work shows the degraded quality of textual information ( Anandan et al . , 2012 ; Zhang et al . , 2018 ; Saygin et al . , 2006 ) .

Another related problem setting is when the latent representation of the user generated texts is shared for different tasks .

It is very common to use recurrent neural networks to create a representation of user generated text to use for different machine learning tasks .

Hitaj el al . show text representations can leak users ' private information such as location ( Hitaj et al . , 2017 ) .

This work aims to anonymize users ' textual information against private-attribute inference attacks .

Adversarial learning is the state-of-the-art approach for creating a privacy preserving text embedding ( Li et al . , 2018 ; Coavoux et al . , 2018 ) .

In these methods , a model is trained to create a text embedding , but we can not control the privacyutility balance .

Recent success of reinforcement learning ( RL ) ( Paulus et al . , 2017 ; Sun and Zhang , 2018 ) shows a feasible alternative : by leveraging reinforcement learning , we can include feedback of attackers and utility in a reward function that allows for the control of the privacy-utility balance .

Furthermore , an RL agent can perturb parts of an embedded text for preserving both utility and privacy , instead of retraining an embedding as in adversarial learning .

Therefore , we propose a novel Reinforcement Learning-based Text Anonymizer , namely , RLTA , composed of two main components : 1 ) an attention based task-aware text representation learner to extract latent embedding representation of the original text 's content w.r.t .

a given task , and 2 ) a deep reinforcement learning based privacy and utility preserver to convert the problem of text anonymization to a one-player game in which the agent 's goal is to learn the optimal strategy for text embedding manipulation to satisfy both privacy and utility .

The Deep Q-Learning algorithm is then used to train the agent capable of changing the text embedding w.r.t .

the received feedback from the privacy and utility subcomponents .

We investigate the following challenges : 1 ) How could we extract the textual embedding w.r.t .

2 ) How could we perturb the extracted text embedding to ensure that user privateattribute information is obscured ?

and 3 ) How could we preserve the utility of text embedding during anonymization ?

Our main contributions are : ( 1 ) we study the problem of text anonymization by learning a reinforced task-aware text anonymizer , ( 2 ) we corporate a data-utility taskaware checker to ensure that the utility of textual embeddings is preserved w.r.t .

( 3 ) we conduct experiments on real-world data to demonstrate the effectiveness of RLTA in an important natural language processing task .

In this paper , we propose a deep reinforcement learning based text anonymization , RLTA , which creates a text embedding such that does not leak user 's private-attribute information while preserving its utility w.r.t .

RLTA has two main components : ( 1 ) an attention based taskaware text representation learner , and ( 2 ) a deep RL based privacy and utility preserver .

Our results illustrate the effectiveness of RLTA in preserving privacy and utility .

One future direction is to generate privacy preserving text rather than embeddings .

We also adopt deep Q-learning to train the agent .

A future direction is to apply different RL algorithms and investigate how it impacts results .

It would be also interesting to adopt RLTA for other types of data .


The freedom of the Deep Web offers a safe place where people can express themselves anonymously

but they also can conduct illegal activities .

In this paper , we present and make publicly available 1 a new dataset for Darknet active domains , which we call it " Darknet Usage Text Addresses " ( DUTA ) .

We built DUTA by sampling the Tor network during two months and manually labeled each address into 26 classes .

Using DUTA , we conducted a comparison between two well-known text representation techniques crossed by three different supervised classifiers to categorize the Tor hidden services .

We also fixed the pipeline elements and identified the aspects that have a critical influence on the classification results .

We found that the combination of TF-IDF words representation with Logistic Regression classifier achieves 96.6 % of 10 folds cross-validation accuracy and a macro F1 score of 93.7 % when classifying a subset of illegal activities from DUTA .

The good performance of the classifier might support potential tools to help the authorities in the detection of these activities .

If we think about the web as an ocean of data , the Surface Web is no more than the slight waves that float on the top .

While in the depth , there is a lot of sunken information that is not reached by the traditional search engines .

The web can be divided into Surface Web and Deep Web .

The Surface Web is the portion of the web that can be crawled and 1 The dataset is available upon request to the first author ( email ) .

indexed by the standard search engines , such as Google or Bing .

However , despite their existence , there is still an enormous part of the web remained without indexing due to its vast size and the lack of hyperlinks , i.e. not referenced by the other web pages .

This part , that can not be found using a search engine , is known as Deep Web ( Noor et al . , 2011 ; Boswell , 2016 ) .

Additionally , the content might be locked and requires human interaction to access e.g. to solve a CAPTCHA or to enter a log-in credential to access .

This type of web pages is referred to as " database-driven " websites .

Moreover , the traditional search engines do not examine the underneath layers of the web , and consequently , do not reach the Deep Web .

The Darknet , which is also known as Dark Web , is a subset of the Deep Web .

It is not only not indexed and isolated , but also requires a specific software or a dedicated proxy server to access it .

The Darknet works over a virtual sub-network of the World Wide Web ( WWW ) that provides an additional layer of anonymity for the network users .

The most popular ones are " The Onion Router"2 also known as Tor network , " Invisible Internet Project " I2P3 , and Freenet4 .

The community of Tor refers to Darknet websites as " Hidden Services " ( HS ) which can be accessed via a special browser called Tor Browser5 .

A study by Bergman et al . ( 2001 ) has stated astonishing statistics about the Deep Web .

For example , only on Deep Web there are more than 550 billion individual documents comparing to only 1 billion on Surface Web .

Furthermore , in the study of Rudesill et al . ( 2015 ) they emphasized on the immensity of the Deep Web which was estimated to be 400 to 500 times wider than the Surface Web .

The concepts of Darknet and Deep Net have ex-isted since the establishment of World Wide Web ( WWW ) , but what make it very popular in the recent years is when the FBI had arrested Dread Pirate Roberts , the owner of Silk Road black market , in October 2013 .

The FBI has estimated the sales on Silk Road to be 1.2 Billion dollars by July 2013 .

The trading network covered among 150,000 anonymous customers and approximately 4,000 vendors ( Rudesill et al . , 2015 ) .

The cryptocurrency ( Nakamoto , 2008 ) is a hot topic in the field of Darknet since it anonymizes the financial transactions and hides the trading parties identities ( Ron and Shamir , 2014 ) .

The Darknet is often associated with illegal activities .

In a study carried out by Intelliagg group ( 2015 ) over 1 K samples of hidden services , they claimed that 68 % of Darknet contents would be illegal .

( 2016 ) showed , after analyzing 5 K onion domains , that the most common usages for Tor HS are criminal and illegal activities , such as drugs , weapons and all kind of pornography .

It is worth to mention about dramatic increase in the proliferation of Darknet domains which doubled their size from 30 K to 60 K between August 2015 and 2016 ( Figure 1 ) .

However , the publicly reachable domains are no more than 6 K to 7 K due to the ambiguity nature of the Darknet ( Ciancaglini et al . , 2016 ) .

Motivated by the critical buried contents on the Darknet and its high abuse , we focused our research in designing and building a system that classifies the illegitimate practices on Darknet .

In this paper , we present the first publicly available dataset called " Darknet Usage Text Addresses " ( DUTA ) that is extracted from the Tor HS Darknet .

DUTA contains 26 categories that cover all the legal and the illegal activities monitored on Darknet during our sampling period .

Our objective is to create a precise categorization of the Darknet via classifying the textual content of the HS .

In order to achieve our target , we designed and compared different combinations of some of the most wellknown text classification techniques by identifying the key stages that have a high influence on the method performance .

We set a baseline methodology by fixing the elements of text classification pipeline which allows the scientific community to compare their future research with this baseline under the defined pipeline .

The fixed methodology we propose might represent a significant contribution into a tool for the authorities who monitor the Darknet abuse .

The rest of the paper is organized as follows : Section 2 presents the related work .

Next , Section 3 explains the proposed dataset DUTA and its characteristics .

After that , Section 4 describes the set of the designed classification pipelines .

Then , in Section 5 we discuss the experiments performed and the results .

In Section 6 we describe the technical implementation details and how we employed the successful classifier in an application .

Finally , in Section 7 we present our conclusions with a pointing to our future work .

In this paper , we have categorized illegal activities of Tor HS by using two text representation methods , TF-IDF and BOW , combined with three classifiers , SVM , LR , and NB .

To support the classification pipelines , we built the dataset DUTA , We found that the combination of the TF-IDF text representation with the Logistic Regression classifier can achieve 96.6 % accuracy over 10 folds of cross-validation and 93.7 % macro F1 score .

We noticed that our classifier suffers from overfitting due to the difficulty of reaching more samples of onion hidden services for some classes like counterfeiting personal identification or illegal drugs .

However , our results are encouraging , and yet there is still a wide margin for future improvements .

We are looking forward to enlarging the dataset by digging deeper into the Darknet by adding more HS sources , even from I2P and Freenet , and exploring ports other than the HTTP port .

Moreover , we plan to get the benefit of the HTML tags and the hyperlinks by weighting some tags or parsing the hyperlinks text .

Also , during the manual labeling of the dataset , we realized that a wide portion of the hidden services advertise their illegal products graphically , i.e. the service owner uses the images instead of the text .

Therefore , our aim is to build an image classifier to work in parallel with the text classification .

The high accuracy we have obtained in this work might represent an opportunity to insert our research into a tool that supports the authorities in monitoring the Darknet .

The Darknet , which is also known as Dark Web , is a subset of the Deep Web .

The Darknet is often associated with illegal activities .