Cybersecurity risks and malware threats are becoming increasingly dangerous and common .

Despite the severity of the problem , there has been few NLP efforts focused on tackling cybersecurity .

In this paper , we discuss the construction of a new database for annotated malware texts .

An annotation framework is introduced based around the MAEC vocabulary for defining malware characteristics , along with a database consisting of 39 annotated APT reports with a total of 6,819 sentences .

We also use the database to construct models that can potentially help cybersecurity researchers in their data collection and analytics efforts .

In 2010 , the malware known as Stuxnet physically damaged centrifuges in Iranian nuclear facilities ( Langner , 2011 ) .

More recently in 2016 , a botnet known as Mirai used infected Internet of Things ( IoT ) devices to conduct large-scale Distributed Denial of Service ( DDoS ) attacks and disabled Internet access for millions of users in the US West Coast ( US-CERT , 2016 ) .

These are only two cases in a long list ranging from ransomeware on personal laptops ( Andronio et al . , 2015 ) to taking over control of moving cars ( Checkoway et al . , 2011 ) .

Attacks such as these are likely to become increasingly frequent and dangerous as more devices and facilities become connected and digitized .

Recently , cybersecurity defense has also been recognized as one of the " problem areas likely to be important both for advancing AI and for its long-run impact on society " ( Sutskever et al . , 2016 ) .

In particular , we feel that natural language processing ( NLP ) has the potential for substantial contribution in cybersecurity and that this is a critical research area given the urgency and risks involved .

There exists a large repository of malwarerelated texts online , such as detailed malware reports by various cybersecurity agencies such as Symantec ( DiMaggio , 2015 ) and Cylance ( Gross , 2016 ) and in various blog posts .

Cybersecurity researchers often consume such texts in the process of data collection .

However , the sheer volume and diversity of these texts make it difficult for researchers to quickly obtain useful information .

A potential application of NLP can be to quickly highlight critical information from these texts , such as the specific actions taken by a certain malware .

This can help researchers quickly understand the capabilities of a specific malware and search in other texts for malware with similar capabilities .

An immediate problem preventing application of NLP techniques to malware texts is that such texts are mostly unannotated .

This severely limits their use in supervised learning techniques .

In light of that , we introduce a database of annotated malware reports for facilitating future NLP work in cybersecurity .

To the best of our knowledge , this is the first database consisting of annotated malware reports .

It is intended for public release , where we hope to inspire contributions from other research groups and individuals .

The main contributions of this paper are :

• We initiate a framework for annotating malware reports and annotate 39 Advanced Persistent Threat ( APT ) reports ( containing 6,819 sentences ) with attribute labels from the Malware Attribute Enumeration and Characterization ( MAEC ) vocabulary ( Kirillov et al . , 2010 ) .

• We propose the following tasks , construct models for tackling them , and discuss the challenges :

• Classify if a sentence is useful for inferring malware actions and capabilities , • Predict token , relation and attribute labels for a given malware-related text , as defined by the earlier framework , and • Predict a malware 's signatures based only on text describing the malware .

In this paper , we presented a framework for annotating malware reports .

We also introduced a database with 39 annotated APT reports and proposed several new tasks and built models for extracting information from the reports .

Finally , we discuss several factors that make these tasks extremely challenging given currently available models .

We hope that this paper and the accompanying database serve as a first step towards NLP being applied in cybersecurity and that other researchers will be inspired to contribute to the database and to construct their own datasets and implementations .

More details about this database can be found at http://statnlp.org / research / re/.


Until quite recently , extending Phrase-based Statistical Machine Translation ( PBSMT ) with syntactic structure caused system performance to deteriorate .

In this work we show that incorporating lexical syntactic descriptions in the form of supertags can yield significantly better PBSMT systems .

We describe a novel PBSMT model that integrates supertags into the target language model and the target side of the translation model .

Two kinds of supertags are employed : those from Lexicalized Tree-Adjoining Grammar and Combinatory Categorial Grammar .

Despite the differences between these two approaches , the supertaggers give similar improvements .

In addition to supertagging , we also explore the utility of a surface global grammaticality measure based on combinatory operators .

We perform various experiments on the Arabic to English NIST 2005 test set addressing issues such as sparseness , scalability and the utility of system subcomponents .

Our best result ( 0.4688 BLEU ) improves by 6.1 % relative to a state-of-theart PBSMT model , which compares very favourably with the leading systems on the NIST 2005 task .

Within the field of Machine Translation , by far the most dominant paradigm is Phrase-based Statistical Machine Translation ( PBSMT ) ( Koehn et al . , 2003 ; Tillmann & Xia , 2003 ) .

However , unlike in rule-and example-based MT , it has proven difficult to date to incorporate linguistic , syntactic knowledge in order to improve translation quality .

Only quite recently have ( Chiang , 2005 ) and ( Marcu et al . , 2006 ) shown that incorporating some form of syntactic structure could show improvements over a baseline PBSMT system .

While ( Chiang , 2005 ) avails of structure which is not linguistically motivated , ( Marcu et al . , 2006 ) employ syntactic structure to enrich the entries in the phrase table .

In this paper we explore a novel approach towards extending a standard PBSMT system with syntactic descriptions : we inject lexical descriptions into both the target side of the phrase translation table and the target language model .

Crucially , the kind of lexical descriptions that we employ are those that are commonly devised within lexicon-driven approaches to linguistic syntax , e.g. Lexicalized Tree-Adjoining Grammar ( Joshi & Schabes , 1992 ; Bangalore & Joshi , 1999 ) and Combinary Categorial Grammar ( Steedman , 2000 ) .

In these linguistic approaches , it is assumed that the grammar consists of a very rich lexicon and a tiny , impoverished 1 set of combinatory operators that assemble lexical entries together into parse-trees .

The lexical entries consist of syntactic constructs ( ' supertags ' ) that describe information such as the POS tag of the word , its subcategorization information and the hierarchy of phrase categories that the word projects upwards .

In this work we employ the lexical entries but exchange the algebraic combinatory operators with the more robust and efficient supertagging approach : like standard taggers , supertaggers employ probabilities based on local context and can be implemented using finite state technology , e.g. Hidden Markov Models ( Bangalore & Joshi , 1999 ) .

There are currently two supertagging approaches available : LTAG-based ( Bangalore & Joshi , 1999 ) and CCG-based ( Clark & Curran , 2004 ) .

Both the LTAG ( Chen et al . , 2006 ) and the CCG supertag sets ( Hockenmaier , 2003 ) were acquired from the WSJ section of the Penn-II Treebank using handbuilt extraction rules .

Here we test both the LTAG and CCG supertaggers .

We interpolate ( log-linearly ) the supertagged components ( language model and phrase table ) with the components of a standard PBSMT system .

Our experiments on the Arabic-English NIST 2005 test suite show that each of the supertagged systems significantly improves over the baseline PBSMT system .

Interestingly , combining the two taggers together diminishes the benefits of supertagging seen with the individual LTAG and CCG systems .

In this paper we discuss these and other empirical issues .

The remainder of the paper is organised as follows : in section 2 we discuss the related work on enriching PBSMT with syntactic structure .

In section 3 , we describe the baseline PBSMT system which our work extends .

Section 5 describes the experiments carried out , together with the results obtained .

Section 6 concludes , and provides avenues for further work .

SMT practitioners have on the whole found it difficult to integrate syntax into their systems .

In this work , we have presented a novel model of PBSMT which integrates supertags into the target language model and the target side of the translation model .

Using LTAG supertags gives the best improvement over a state-of-the-art PBSMT system for a smaller data set , while CCG supertags work best on a large 2 million-sentence pair training set .

Adding grammaticality factors based on algebraic compositional operators gives the best result , namely 0.4688 BLEU , or a 6.1 % relative increase over the baseline .

This result compares favourably with the best systems on the NIST 2005 Arabic-English task .

We expect more work on system integration to improve results still further , and anticipate that similar increases are to be seen for other language pairs .


Existing work in fine-grained sentiment analysis focuses on sentences and phrases but ignores the contribution of individual words and their grammatical connections .

This is because of a lack of both ( 1 ) annotated data at the word level and ( 2 ) algorithms that can leverage syntactic information in a principled way .

We address the first need by annotating articles from the information technology business press via crowdsourcing to provide training and testing data .

To address the second need , we propose a suffix-tree data structure to represent syntactic relationships between opinion targets and words in a sentence that are opinion-bearing .

We show that a factor graph derived from this data structure acquires these relationships with a small number of word-level features .

We demonstrate that our supervised model performs better than baselines that ignore syntactic features and constraints .

The terms " sentiment analysis " and " opinion mining " cover a wide body of research on and development of systems that can automatically infer emotional states from text ( after Pang and Lee ( 2008 ) we use the two names interchangeably ) .

Sentiment analysis plays a large role in business , politics , and is itself a vibrant research area ( Bollen et al . , 2010 ) .

Effective sentiment analysis for texts such as newswire depends on the ability to extract who ( source ) is saying what ( target ) .

Fine-grained sentiment analysis requires identifying the sources and targets directly relevant to sentiment bearing expressions ( Ruppenhofer et al . , 2008 ) .

For example , consider the following sentence from a major information technology ( IT ) business journal : Lloyd Hession , chief security officer at BT Radianz in New York , said that virtualization also opens up a slew of potential network access control issues .

There are three entities in the sentence that have the capacity to express an opinion : Lloyd Hession , BT Radianz , and New York .

There are also a number of mentioned concepts that could serve as the topic of an opinion in the sentence , or target .

These include all the sources , but also " virtualization " , " network access control " , " network " , and so on .

The challenging task is to discriminate between these mentions and choose the ones that are relevant to the user .

Furthermore , such a system must also indicate the content of the opinion itself .

This means that we are actually searching for all triples { source , target , opinion } in this sentence ( Kim and Hovy , 2006 ) and throughout each document in the corpus .

In this case , we want to identify that Lloyd Hession is the source of an opinion , " slew of network issues , " about a target , virtualization .

Providing such fine-grained annotations would enrich information extraction , question answering , and corpus exploration applications by letting users see who is saying what with what opinion ( Wilson et al . , 2005 ; Stoyanov and Cardie , 2006 ) .

We motivate the need for a grammatically-focused approach to fine-grained opinion mining and situate it within the context of existing work in Section 2 .

We propose a supervised technique for learning opiniontarget relations from dependency graphs in a way that preserves syntactic coherence and semantic compositionality .

In addition to being theoretically sound -a lacuna identified in many sentiment systems1 -such approaches improve downstream sentiment tasks ( Moilanen and Pulman , 2007 ) .

There are multiple types of downstream tasks that potentially require the retrieval of { source , target , opinion } relations on a sentence-by-sentence basis .

An increasingly significant application area is in the use of large corpora in social science .

This area of research requires the exploration and aggregation of data about the relationships between discourses , organizations , and people .

For example , the IT business press data that we use in this work belongs to a larger research program ( Tsui et al . , 2009 ; Sayeed et al . , 2010 ) of exploring industry opinion leadership .

IT business press text is one type of text in which many entities and opinions can appear intermingled with one another in a small amount of text .

Another application for fine-grained sentiment relation retrieval of this type is paraphrasing , where attribution of which opinion belongs to which entities may be important for producing useful and accurate output , since source and target identification errors can change the entire meaning of an output text .

Unlike previous approaches that ignore syntax , we use a sentence 's syntactic structure to build a probabilistic model that encodes whether a word is opinion bearing as a latent variable .

We build a data structure we call a " syntactic relatedness trie " ( Section 3 ) that serves as the skeleton for a graphical model over the sentiment relevance of words ( Section 4 ) .

This approach allows us to learn features that predict opinion bearing constructions from grammatical structures .

Because of a dearth of resources for this fine-grained task , we also develop new crowdsourcing techniques for labeling word-level , syntactically informed sen-timent ( Section 5 ) .

We use inference techniques to uncover grammatical patterns that connect opinionexpressing words and target entities ( Section 6 ) performing better than using syntactically uninformed methods .

In this work , we have applied machine learning to produce a robust modeling of syntactic structure for an information extraction application .

A solution to the problem of modeling these structures requires the development of new techniques that model complex linguistic relationships in an application-dependent way .

We have shown that we can mine these relationships without being overcome by the data-sparsity issues that typically stymie learning over complex linguistic structure .

The limitations on these techniques ultimately find their root in the difficulty in modeling complex syntactic structures that simultaneously exclude irrelevant portions of the structure while maintaining connected relations .

Our technique uses a structurelabelling scheme that enforces connectedness .

Enforcing connected structure is not only necessary to produce useful results but also to improve accuracy .

Further performance gains might be possible by enriching the feature set .

For example , the POS tagset used by the Stanford parser contains multiple verb tags that represent different English tenses and numbers .

For the purpose of sentiment relations , it is possible that the differences between verb tags are too small to matter and are causing data sparsity issues .

Thus , we could additional features that " back off " to general verb tags .


Many language generation tasks require the production of text conditioned on both structured and unstructured inputs .

We present a novel neural network architecture which generates an output sequence conditioned on an arbitrary number of input functions .

Crucially , our approach allows both the choice of conditioning context and the granularity of generation , for example characters or tokens , to be marginalised , thus permitting scalable and effective training .

Using this framework , we address the problem of generating programming code from a mixed natural language and structured specification .

We create two new data sets for this paradigm derived from the collectible trading card games Magic the Gathering and Hearthstone .

On these , and a third preexisting corpus , we demonstrate that marginalising multiple predictors allows our model to outperform strong benchmarks .

The generation of both natural and formal languages often requires models conditioned on diverse predictors ( Koehn et al . , 2007 ;

Most models take the restrictive approach of employing a single predictor , such as a word softmax , to predict all tokens of the output sequence .

To illustrate its limitation , suppose we wish to generate the answer to the question " Who wrote The Foundation ? " as " The Foundation was written by Isaac Asimov " .

The generation of the words " Issac Asimov " and " The Foundation " from a word softmax trained on annotated data is unlikely to succeed as these words are sparse .

A robust model might , for example , employ one pre- dictor to copy " The Foundation " from the input , and a another one to find the answer " Issac Asimov " by searching through a database .

However , training multiple predictors is in itself a challenging task , as no annotation exists regarding the predictor used to generate each output token .

Furthermore , predictors generate segments of different granularity , as database queries can generate multiple tokens while a word softmax generates a single token .

In this work we introduce Latent Predictor Networks ( LPNs ) , a novel neural architecture that fulfills these desiderata : at the core of the architecture is the exact computation of the marginal likelihood over latent predictors and generated segments allowing for scalable training .

We introduce a new corpus for the automatic generation of code for cards in Trading Card Games ( TCGs ) , on which we validate our model1 .

TCGs , such as Magic the Gathering ( MTG ) and Hearthstone ( HS ) , are games played between two players that build decks from an ever expanding pool of cards .

Examples of such cards are shown in Figure 1 .

Each card is identified by its attributes ( e.g. , name and cost ) and has an effect that is described in a text box .

Digital implementations of these games implement the game logic , which includes the card effects .

This is attractive from a data extraction perspective as not only are the data annotations naturally generated , but we can also view the card as a specification communicated from a designer to a software engineer .

This dataset presents additional challenges to prior work in code generation ( Wong and Mooney , 2006 ; Jones et al . , 2012 ; Lei et al . , 2013 ; Artzi et al . , 2015 ; Quirk et al . , 2015 ) , including the handling of structured input-i.e .

cards are composed by multiple sequences ( e.g. , name and description)-and attributes ( e.g. , attack and cost ) , and the length of the generated sequences .

Thus , we propose an extension to attention-based neural models ( Bahdanau et al . , 2014 ) to attend over structured inputs .

Finally , we propose a code compression method to reduce the size of the code without impacting the quality of the predictions .

Experiments performed on our new datasets , and a further pre-existing one , suggest that our extensions outperform strong benchmarks .

The paper is structured as follows : We first describe the data collection process ( Section 2 ) and formally define our problem and our baseline method ( Section 3 ) .

Then , we propose our extensions , namely , the structured attention mechanism ( Section 4 ) and the LPN architecture ( Section 5 ) .

We follow with the description of our code compression algorithm ( Section 6 ) .

Our model is validated by comparing with multiple benchmarks ( Section 7 ) .

Finally , we contextualize our findings with related work ( Section 8) and present the conclusions of this work ( Section 9 ) .

We introduced a neural network architecture named Latent Prediction Network , which allows efficient marginalization over multiple predictors .

Under this architecture , we propose a generative model for code generation that combines a character level softmax to generate language-specific tokens and multiple pointer networks to copy keywords from the input .

Along with other extensions , namely structured attention and code compression , our model is applied on on both existing datasets and also on a newly created one with implementations of TCG game cards .

Our experiments show that our model out-performs multiple benchmarks , which demonstrate the importance of combining different types of predictors .


The main work in bilingual lexicon extraction from comparable corpora is based on the implicit hypothesis that corpora are balanced .

However , the historical contextbased projection method dedicated to this task is relatively insensitive to the sizes of each part of the comparable corpus .

Within this context , we have carried out a study on the influence of unbalanced specialized comparable corpora on the quality of bilingual terminology extraction through different experiments .

Moreover , we have introduced a regression model that boosts the observations of word cooccurrences used in the context-based projection method .

Our results show that the use of unbalanced specialized comparable corpora induces a significant gain in the quality of extracted lexicons .

The bilingual lexicon extraction task from bilingual corpora was initially addressed by using parallel corpora ( i.e. a corpus that contains source texts and their translation ) .

However , despite good results in the compilation of bilingual lexicons , parallel corpora are scarce resources , especially for technical domains and for language pairs not involving English .

For these reasons , research in bilingual lexicon extraction has focused on another kind of bilingual corpora comprised of texts sharing common features such as domain , genre , sampling period , etc . without having a source text / target text relationship ( McEnery and Xiao , 2007 ) .

These corpora , well known now as comparable corpora , have also initially been introduced as non-parallel corpora ( Fung , 1995 ; Rapp , 1995 ) , and non-aligned corpora ( Tanaka and Iwasaki , 1996 ) .

According to Fung and Che-ung ( 2004 ) , who range bilingual corpora from parallel corpora to quasi-comparable corpora going through comparable corpora , there is a continuum from parallel to comparable corpora ( i.e. a kind of filiation ) .

The bilingual lexicon extraction task from comparable corpora inherits this filiation .

For instance , the historical context-based projection method ( Fung , 1995 ; Rapp , 1995 ) , known as the standard approach , dedicated to this task seems implicitly to lead to work with balanced comparable corpora in the same way as for parallel corpora ( i.e. each part of the corpus is composed of the same amount of data ) .

In this paper we want to show that the assumption that comparable corpora should be balanced for bilingual lexicon extraction task is unfounded .

Moreover , this assumption is prejudicial for specialized comparable corpora , especially when involving the English language for which many documents are available due the prevailing position of this language as a standard for international scientific publications .

Within this context , our main contribution consists in a re-reading of the standard approach putting emphasis on the unfounded assumption of the balance of the specialized comparable corpora .

In specialized domains , the comparable corpora are traditionally of small size ( around 1 million words ) in comparison with comparable corpus-based general language ( up to 100 million words ) .

Consequently , the observations of word co-occurrences which is the basis of the standard approach are unreliable .

To make them more reliable , our second contribution is to contrast different regression models in order to boost the observations of word co-occurrences .

This strategy allows to improve the quality of extracted bilingual lexicons from comparable corpora .

In this paper , we have studied how an unbalanced specialized comparable corpus could influence the quality of the bilingual lexicon extraction .

This aspect represents a significant interest when working with specialized comparable corpora for which the quantity of the data collected may differ depending on the languages involved , especially when involving the English language as many scientific documents are available .

More precisely , our different experiments show that using an unbalanced specialized comparable corpus always improves the quality of word translations .

Thus , the MAP goes up from 29.6 % ( best result on the balanced corpora ) to 42.3 % ( best result on the unbalanced corpora ) in the breast cancer domain , and from 16.5 % to 26.0 % in the diabetes domain .

Additionally , these results can be improved by using a prediction model of the word co-occurrence counts .

Here , the MAP goes up from 42.3 % ( best result on the unbalanced corpora ) to 46.9 % ( best result on the unbalanced corpora with prediction ) in the breast cancer domain , and from 26.0 % to 29.8 % in the diabetes domain .

We hope that this study will pave the way for using specialized unbalanced comparable corpora for bilingual lexicon extraction .


Word sense disambiguation is the process of determining which sense of a word is used in a given context .

Due to its importance in understanding semantics of natural languages , word sense disambiguation has been extensively studied in Computational Linguistics .

However , existing methods either are brittle and narrowly focus on specific topics or words , or provide only mediocre performance in real-world settings .

Broad coverage and disambiguation quality are critical for a word sense disambiguation system .

In this paper we present a fully unsupervised word sense disambiguation method that requires only a dictionary and unannotated text as input .

Such an automatic approach overcomes the problem of brittleness suffered in many existing methods and makes broad-coverage word sense disambiguation feasible in practice .

We evaluated our approach using SemEval 2007 Task 7 ( Coarse-grained English All-words Task ) , and our system significantly outperformed the best unsupervised system participating in Se-mEval 2007 and achieved the performance approaching top-performing supervised systems .

Although our method was only tested with coarse-grained sense disambiguation , it can be directly applied to fine-grained sense disambiguation .

In many natural languages , a word can represent multiple meanings / senses , and such a word is called a homograph .

Word sense disambiguation(WSD ) is the process of determining which sense of a homograph is used in a given context .

WSD is a long-standing problem in Computational Linguistics , and has significant impact in many real-world applications including machine translation , information extraction , and information retrieval .

Generally , WSD methods use the context of a word for its sense disambiguation , and the context information can come from either annotated / unannotated text or other knowledge resources , such as Word-Net ( Fellbaum , 1998 ) , SemCor ( SemCor , 2008 ) , Open Mind Word Expert ( Chklovski and Mihalcea , 2002 ) , eXtended WordNet ( Moldovan and Rus , 2001 ) , Wikipedia ( Mihalcea , 2007 ) , parallel corpora ( Ng , Wang , and Chan , 2003 ) .

In ( Ide and Véronis , 1998 ) many different WSD approaches were described .

Usually , WSD techniques can be divided into four categories ( Agirre and Edmonds , 2006 ) ,

• Dictionary and knowledge based methods .

These methods use lexical knowledge bases such as dictionaries and thesauri , and hypothesize that context knowledge can be extracted from definitions of words .

For example , Lesk disambiguated two words by finding the pair of senses with the greatest word overlap in their dictionary definitions ( Lesk , 1986 ) .

Supervised methods mainly adopt context to disambiguate words .

A supervised method includes a training phase and a testing phase .

In the training phase , a sense-annotated training corpus is required , from which syntactic and semantic features are extracted to create a classifier using machine learning techniques , such as Support Vector Machine ( Novischi et al . , 2007 ) .

In the following testing phase , a word is classified into senses ( Mihalcea , 2002 ) ( Ng and Lee , 1996 ) .

Currently supervised methods achieve the best disambiguation quality ( about 80 % precision and recall for coarse-grained WSD in the most recent WSD evaluation conference SemEval 2007 ( Navigli et al . , 2007 ) ) .

Nevertheless , since training corpora are manually annotated and expensive , supervised methods are often brittle due to data scarcity , and it is hard to annotate and acquire sufficient contextual information for every sense of a large number of words existing in natural languages .

To overcome the knowledge acquisition bottleneck problem suffered by supervised methods , these methods make use of a small annotated corpus as seed data in a bootstrapping process ( Hearst , 1991 ) ( Yarowsky , 1995 ) .

A word-aligned bilingual corpus can also serve as seed data ( Ng , Wang , and Chan , 2003 ) .

These methods acquire contextual information directly from unannotated raw text , and senses can be induced from text using some similarity measure ( Lin , 1997 ) .

However , automatically acquired information is often noisy or even erroneous .

In the most recent SemEval 2007 ( Navigli et al . , 2007 ) , the best unsupervised systems only achieved about 70 % precision and 50 % recall .

Disambiguation of a limited number of words is not hard , and necessary context information can be carefully collected and hand-crafted to achieve high disambiguation accuracy as shown in ( Yarowsky , 1995 ) .

However , such approaches suffer a significant performance drop in practice when domain or vocabulary is not limited .

Such a " cliff-style " performance collapse is called brittleness , which is due to insufficient knowledge and shared by many techniques in Artificial Intelligence .

The main challenge of a WSD system is how to overcome the knowledge acquisition bottleneck and efficiently collect the huge amount of context knowledge .

More precisely , a practical WSD need figure out how to create and maintain a comprehensive , dynamic , and up-todate context knowledge base in a highly automatic manner .

The context knowledge required in WSD has the following properties :

The context knowledge need cover a large number of words and their usage .

Such a requirement of broad coverage is not trivial because a natural language usually contains thousands of words , and some popular words can have dozens of senses .

For example , the Oxford English Dictionary has approximately 301,100 main entries ( Oxford , 2003 ) , and the average polysemy of the WordNet inventory is 6.18 ( Fellbaum , 1998 ) .

Clearly acquisition of such a huge amount of knowledge can only be achieved with automatic techniques .

2 . Natural language is not a static phenomenon .

New usage of existing words emerges , which creates new senses .

New words are created , and some words may " die " over time .

It is estimated that every year around 2,500 new words appear in English ( Kister , 1992 ) .

Such dynamics requires a timely maintenance and updating of context knowledge base , which makes manual collection even more impractical .

Taking into consideration the large amount and dynamic nature of context knowledge , we only have limited options when choosing knowledge sources for WSD .

WSD is often an unconscious process to human beings .

With a dictionary and sample sentences / phrases an average educated person can correctly disambiguate most polysemous words .

Inspired by human WSD process , we choose an electronic dictionary and unannotated text samples of word instances as context knowledge sources for our WSD system .

Both sources can be automatically accessed , provide an excellent coverage of word meanings and usage , and are actively updated to reflect the current state of languages .

In this paper we present a fully unsupervised WSD system , which only requires WordNet sense inventory and unannotated text .

In the rest of this paper , section 2 describes how to acquire and represent the context knowledge for WSD .

We present our WSD algorithm in section 3 .

Our WSD system is evaluated with SemEval-2007 Task 7 ( Coarse-grained English All-words Task ) data set , and the experiment results are discussed in section 4 .

Broad coverage and disambiguation quality are critical for WSD techniques to be adopted in practice .

This paper proposed a fully unsupervised WSD method .

We have evaluated our approach with SemEval-2007 Task 7 ( Coarse-grained English Allwords Task ) data set , and we achieved F-scores approaching the top performing supervised WSD systems .

By using widely available unannotated text and a fully unsupervised disambiguation approach , our method may provide a viable solution to the problem of WSD .

1 . Continue to build the knowledge base , enlarge the coverage and improve the system performance .

The experiment results in Section 4.2 clearly show that more word instances can improve the disambiguation accuracy and recall scores ;

2 . WSD is often an unconscious process for human beings .

It is unlikely that a reader examines all surrounding words when determining the sense of a word , which calls for a smarter and more selective matching strategy than what we have tried in Section 4.1 ;

3 . Test our WSD system on fine-grained SemEval 2007 WSD task 17 .

Although we only evaluated our approach with coarse-grained senses , our method can be directly applied to finegrained WSD without any modifications .

This paper proposed a fully unsupervised WSD method .


Lexica distinguishing all morphologically related forms of each lexeme are crucial to many language technologies , yet building them is expensive .

We propose Frugal Paradigm Completion , an approach that predicts all related forms in a morphological paradigm from as few manually provided forms as possible .

It induces typological information during training which it uses to determine the best sources at test time .

We evaluate our language-agnostic approach on 7 diverse languages .

Compared to popular alternative approaches , our Frugal Paradigm Completion approach reduces manual labor by 16 - 63 % and is the most robust to typological variation .

From syntactic parsing ( Seeker and Kuhn , 2013 ) to text-to-speech ( Zen et al . , 2016 ; Wan et al . , 2019 ) , many linguistic technologies rely on accurate lexica decorated with morphological information .

Yet , building such lexica requires much human effort ( Buckwalter , 2002 ; Tadić and Fulgosi , 2003 ; Forsberg et al . , 2006 ; Sagot , 2010 ; Eskander et al . , 2013 ) .

We present a language-agnostic method for minimizing the manual labor required to add new paradigms to an existing lexicon .

Formally , let each lexicon entry , or realization , be a triple ( P , C , f ) .

P marks membership in some paradigm P of morphologically related words , C defines a cell in P as a bundle of morphosyntactic features , and f is the form realizing C in P. Hence , paradigm SING can be expressed ( in the UniMorph schema ( Kirov et al . , 2018 ) ) as a set of realizations : { ( SING , NFIN , sing ) , ( SING , 3.SG.PRES , sings ) , . . .

For each paradigm to be added to the lexicon , e.g. , FLY , we aim to select as few sources as pos-sible to be manually realized , e.g. , { ( FLY , NFIN , fly ) , ( FLY , PST , flew ) } such that the forms realizing the remaining cells can be predicted , i.e. , flies , flying , flown .

Here , sources are manually provided realizations .

Targets are realizations whose forms must be predicted from sources .

Our work differs from traditional paradigm completion ( Durrett and DeNero , 2013 ) in that sources are not given blindly , but the system must strategically select which sources it wants to be given at test time .

Paradigm completion from one source is typically non-deterministic due to multiple inflection classes realizing different exponents in some cells , e.g. , suffixing + ed generates the past tense for WALK , but not for SING or FLY which are members of different classes .

Hence , many works discuss paradigm completion in the context of ( implicit ) inflection class disambiguation ( Ackerman et al . , 2009 ; Montermini and Bonami , 2013 ;

Finkel and Stump ( 2007 ) propose three approaches to select the fewest sources required to deterministically identify class .

Yet , neural sequence models can often complete paradigms accurately from less sources without fully disambiguating inflection class ( Kann and Schütze , 2016 ; Aharoni and Goldberg , 2017 ; Wu and Cotterell , 2019 ) .

See Elsner et al . ( 2019 ) for an overview of the application of neural sequence models to morphological theory .

We propose Frugal Paradigm Completion ( FPC ) , inspired by work on inflection class disambiguation and neural sequence modeling .

We train a source selection agent ( SSA ) to induce typological knowledge regarding the distribution of complexity in paradigms and use this to request informative source cells to be realized by an oracle .

Sources are fed to a predictor to generate target forms .

For each paradigm , SSA iteratively requests sources until the oracle confirms all cells have been realized correctly .

We introduce a novel metric , auto-rate , to quantify the manual labour ( performed by the oracle ) needed to complete each paradigm .

Using this metric , we demonstrate that FPC reduces labor by 63 % over predicting targets from lemmata , and 47 % over predicting them from the smallest set of sources that fully disambiguates inflection class .

We propose a new typology for discussing the organization of complexity in paradigms which helps explain why strategies perform better or worse on certain languages while FPC , being sensitive to typological variation , performs robustly .

After discussing related paradigm completion approaches in Section 2 , we describe FPC in Section 3 .

Section 4 covers all data and experimental set up details .

We discuss results in Section 5 and analyze FPC 's behavior in Section 6 .

We presented Frugal Paradigm Completion , which reduces the manual labor required to expand a morphological lexicon by 16 - 63 % over competitive approaches across 7 languages .

We demonstrated that typologically distinct morphological systems require unique treatment and benefit from our SSA , that learns its strategy from data .

We found that inducing this strategy is not as challenging as previously suggested ( Finkel and Stump , 2007 ) .

Thus , SSA might be replaced with a less costly architecture while our model might be improved by conditioning on semantics and jointly decoding from a variable number of sources .


Several attempts have been made to learn phrase translation probabilities for phrasebased statistical machine translation that go beyond pure counting of phrases in word-aligned training data .

Most approaches report problems with overfitting .

We describe a novel leavingone-out approach to prevent over-fitting that allows us to train phrase models that show improved translation performance on the WMT08 Europarl German-English task .

In contrast to most previous work where phrase models were trained separately from other models used in translation , we include all components such as single word lexica and reordering models in training .

Using this consistent training of phrase models we are able to achieve improvements of up to 1.4 points in BLEU .

As a side effect , the phrase table size is reduced by more than 80 % .

A phrase-based SMT system takes a source sentence and produces a translation by segmenting the sentence into phrases and translating those phrases separately ( Koehn et al . , 2003 ) .

The phrase translation table , which contains the bilingual phrase pairs and the corresponding translation probabilities , is one of the main components of an SMT system .

The most common method for obtaining the phrase table is heuristic extraction from automatically word-aligned bilingual training data ( Och et al . , 1999 ) .

In this method , all phrases of the sentence pair that match constraints given by the alignment are extracted .

At extraction time it does not matter , whether the phrases are extracted from a highly probable phrase alignment or from an unlikely one .

Phrase model probabilities are typically defined as relative frequencies of phrases extracted from word-aligned parallel training data .

The joint counts C ( f , ẽ ) of the source phrase f and the target phrase ẽ in the entire training data are normalized by the marginal counts of source and target phrase to obtain a conditional probability

The translation process is implemented as a weighted log-linear combination of several models h m ( e I 1 , s K 1 , f J 1 ) including the logarithm of the phrase probability in source-to-target as well as in target-to-source direction .

The phrase model is combined with a language model , word lexicon models , word and phrase penalty , and many others .

The best translation êÎ 1 as defined by the models then can be written as

In this work , we propose to directly train our phrase models by applying a forced alignment procedure where we use the decoder to find a phrase alignment between source and target sentences of the training data and then updating phrase translation probabilities based on this alignment .

In contrast to heuristic extraction , the proposed method provides a way of consistently training and using phrase models in translation .

We use a modified version of a phrase-based decoder to perform the forced alignment .

This way we ensure that all models used in training are identical to the ones used at decoding time .

An illustration of the basic idea can be seen in Figure 1 .

In the literature this method by itself has been shown to be problematic because it suffers from over-fitting ( DeNero et al . , 2006 ) , ( Liang et al . , 2006 ) .

Since our initial phrases are extracted from the same training data , that we want to align , very long phrases can be found for segmentation .

As these long phrases tend to occur in only a few training sentences , the EM algorithm generally overestimates their probability and neglects shorter phrases , which better generalize to unseen data and thus are more useful for translation .

In order to counteract these effects , our training procedure applies leaving-one-out on the sentence level .

Our results show , that this leads to a better translation quality .

Ideally , we would produce all possible segmentations and alignments during training .

However , this has been shown to be infeasible for real-world data ( DeNero and Klein , 2008 ) .

As training uses a modified version of the translation decoder , it is straightforward to apply pruning as in regular decoding .

Additionally , we consider three ways of approximating the full search space :

the single-best Viterbi alignment , 2 . the n-best alignments , 3 .

all alignments remaining in the search space after pruning .

The performance of the different approaches is measured and compared on the German-English Europarl task from the ACL 2008 Workshop on Statistical Machine Translation ( WMT08 ) .

Our results show that the proposed phrase model training improves translation quality on the test set by 0.9 BLEU points over our baseline .

We find that by interpolation with the heuristically extracted phrases translation performance can reach up to 1.4 BLEU improvement over the baseline on the test set .

After reviewing the related work in the following section , we give a detailed description of phrasal alignment and leaving-one-out in Section 3 .

Section 4 explains the estimation of phrase models .

The empirical evaluation of the different approaches is done in Section 5 .

We have shown that training phrase models can improve translation performance on a state-ofthe-art phrase-based translation model .

This is achieved by training phrase translation probabilities in a way that they are consistent with their use in translation .

A crucial aspect here is the use of leaving-one-out to avoid over-fitting .

We have shown that the technique is superior to limiting phrase lengths and smoothing with lexical probabilities alone .

While models trained from Viterbi alignments already lead to good results , we have demonstrated that considering the 100-best alignments allows to better model the ambiguities in phrase segmentation .

The proposed techniques are shown to be superior to previous approaches that only used lexical probabilities to smooth phrase tables or imposed limits on the phrase lengths .

On the WMT08 Europarl task we show improvements of 0.9 BLEU points with the trained phrase table and 1.4 BLEU points when interpolating the newly trained model with the original , heuristically extracted phrase table .

In TER , improvements are 0.4 and 1.7 points .

In addition to the improved performance , the trained models are smaller leading to faster and smaller translation systems .

In TER , improvements are 0.4 and 1.7 points .


Abuse on the Internet represents a significant societal problem of our time .

Previous research on automated abusive language detection in Twitter has shown that communitybased profiling of users is a promising technique for this task .

However , existing approaches only capture shallow properties of online communities by modeling followerfollowing relationships .

In contrast , working with graph convolutional networks ( GCNs ) , we present the first approach that captures not only the structure of online communities but also the linguistic behavior of the users within them .

We show that such a heterogeneous graph-structured modeling of communities significantly advances the current state of the art in abusive language detection .

Matthew Zook ( 2012 ) carried out an interesting study showing that the racist tweets posted in response to President Obama 's re-election were not distributed uniformly across the United States but instead formed clusters .

This phenomenon is known as homophily : i.e. , people , both in real life and online , tend to cluster with those who appear similar to themselves .

To model homophily , recent research in abusive language detection on Twitter ( Mishra et al . , 2018a ) incorporates embeddings for authors ( i.e. , users who have composed tweets ) that encode the structure of their surrounding communities .

The embeddings ( called author profiles ) are generated by applying a node embedding framework to an undirected unlabeled community graph where nodes denote the authors and edges the follower-following relationships amongst them on Twitter .

However , these profiles do not capture the linguistic behavior of the authors and their communities and do not convey whether their tweets tend to be abusive or not .

In contrast , we represent the community of authors as a heterogeneous graph consisting of two types of nodes , authors and their tweets , rather than a homogeneous community graph of authors only .

The primary advantage of such heterogeneous representations is that they enable us to model both community structure as well as the linguistic behavior of authors in these communities .

To generate richer author profiles , we then propose a semi-supervised learning approach based on graph convolutional networks ( GCNs ) applied to the heterogeneous graph representation .

To the best of our knowledge , our work is the first to use GCNs to model online communities in social media .

We demonstrate that our methods provide significant improvements over existing techniques .

In this paper , we built on the work of Mishra et al . ( 2018a ) that introduces community-based profiling of authors for abusive language detection .

We proposed an approach based on graph convolutional networks to show that author profiles that directly capture the linguistic behavior of authors along with the structural traits of their community significantly advance the current state of the art .


This paper investigates cross-lingual textual entailment as a semantic relation between two text portions in different languages , and proposes a prospective research direction .

We argue that cross-lingual textual entailment ( CLTE ) can be a core technology for several cross-lingual NLP applications and tasks .

Through preliminary experiments , we aim at proving the feasibility of the task , and providing a reliable baseline .

We also introduce new applications for CLTE that will be explored in future work .

Textual Entailment ( TE ) ( Dagan and Glickman , 2004 ) has been proposed as a generic framework for modeling language variability .

Given two texts T and H , the task consists in deciding if the meaning of H can be inferred from the meaning of T. So far , TE has been only applied in a monolingual setting , where both texts are assumed to be written in the same language .

In this work , we propose and investigate a cross-lingual extension of TE , where we assume that T and H are written in different languages .

The great potential of integrating ( monolingual ) TE recognition components into NLP architectures has been reported in several works , such as question answering ( Harabagiu and Hickl , 2006 ) , information retrieval ( Clinchant et al . , 2006 ) , information extraction ( Romano et al . , 2006 ) , and document summarization ( Lloret et al . , 2008 ) .

To the best of our knowledge , mainly due to the absence of cross-lingual TE ( CLTE ) recognition components , similar improvements have not been achieved yet in any cross-lingual application .

As a matter of fact , despite the great deal of attention that TE has received in recent years ( also witnessed by five editions of the Recognizing Textual Entailment Challenge1 ) , interest for cross-lingual extensions has not been in the mainstream of TE research , which until now has been mainly focused on the English language .

Nevertheless , the strong interest towards crosslingual NLP applications ( both from the market and research perspectives , as demonstrated by successful evaluation campaigns such as CLEF2 ) is , to our view , a good reason to start investigating CLTE , as well .

Along such direction , research can now benefit from recent advances in other fields , especially machine translation ( MT ) , and the availability of : i ) large amounts of parallel and comparable corpora in many languages , ii ) open source software to compute word-alignments from parallel corpora , and iii ) open source software to set-up strong MT baseline systems .

We strongly believe that all these resources can potentially help in developing inference mechanisms on multilingual data .

Building on these considerations , this paper aims to put the basis for future research on the crosslingual Textual Entailment task , in order to allow for semantic inference across languages in different NLP tasks .

Among these , as a long-term goal , we plan to adopt CLTE to support the alignment of text portions that express the same meaning in different languages .

As a possible application scenario , CLTE can be used to address content merging tasks in tidy multilingual environments , such as commercial Web sites , digital libraries , or user generated content collections .

Within such framework , as it will be discussed in the last section of this paper , CLTE components can be used for automatic content synchronization in a concurrent , collaborative , and multilingual editing setting , e.g. Wikipedia .

This paper presented a preliminary investigation towards cross-lingual Textual Entailment , focusing on possible research directions and alternative methodologies .

Baseline results have been provided to demonstrate the potentialities of a simple approach that integrates MT and monolingual TE components .

Overall , our work sets a novel framework for further studies and experiments to improve crosslingual NLP tasks .

In particular , CLTE can be scaled to more complex problems , such as cross-lingual content merging and synchronization .


Recognition of tone and intonation is essential for speech recognition and language understanding .

However , most approaches to this recognition task have relied upon extensive collections of manually tagged data obtained at substantial time and financial cost .

In this paper , we explore two approaches to tone learning with substantially reductions in training data .

We employ both unsupervised clustering and semi-supervised learning to recognize pitch accent in English and tones in Mandarin Chinese .

In unsupervised Mandarin tone clustering experiments , we achieve 57 - 87 % accuracy on materials ranging from broadcast news to clean lab speech .

For English pitch accent in broadcast news materials , results reach 78 % .

In the semi-supervised framework , we achieve Mandarin tone recognition accuracies ranging from 70 % for broadcast news speech to 94 % for read speech , outperforming both Support Vector Machines ( SVMs ) trained on only the labeled data and the 25 % most common class assignment level .

These results indicate that the intrinsic structure of tone and pitch accent acoustics can be exploited to reduce the need for costly labeled training data for tone learning and recognition .

Tone and intonation play a crucial role across many languages .

However , the use and structure of tone varies widely , ranging from lexical tone which determines word identity to pitch accent signalling information status .

Here we consider the recognition of lexical tones in Mandarin Chinese syllables and pitch accent in English .

Although intonation is an integral part of language and is requisite for understanding , recognition of tone and pitch accent remains a challenging problem .

The majority of current approaches to tone recognition in Mandarin and other East Asian tone languages integrate tone identification with the general task of speech recognition within a Hidden Markov Model framework .

In some cases tone recognition is done only implicitly when a word or syllable is constrained jointly by the segmental acoustics and a higher level language model and the word identity determines tone identity .

Other strategies build explicit and distinct models for the syllable final region , the vowel and optionally a final nasal , for each tone .

Recent research has demonstrated the importance of contextual and coarticulatory influences on the surface realization of tones .

The overall shape of the tone or accent can be substantially modified by the local effects of adjacent tone and intonational elements .

Furthermore , broad scale phenomena such as topic and phrase structure can affect pitch height , and pitch shape may be variably affected by the presence of boundary tones .

These findings have led to explicit modeling of tonal context within the HMM framework .

In addition to earlier approaches that employed phrase structure ( Fujisaki , 1983 ) , several recent approaches to tone recognition in East Asian languages ( Wang and Seneff , 2000 ; Zhou et al . , 2004 ) have incorporated elements of local and broad range contextual influence on tone .

Many of these techniques create explicit context-dependent models of the phone , tone , or accent for each context in which they appear , either using the tone sequence for left or right context or using a simplified high-low contrast , as is natural for integration in a Hidden Markov Model speech recognition framework .

In pitch accent recognition , recent work by ( Hasegawa-Johnson et al . , 2004 ) has integrated pitch accent and boundary tone recognition with speech recognition using prosodically conditioned models within an HMM framework , improving both speech and prosodic recognition .

Since these approaches are integrated with HMM speech recognition models , standard HMM training procedures which rely upon large labeled training sets are used for tone recognition as well .

Other tone and pitch accent recognition approaches using other classification frameworks such as support vector machines ( Thubthong and Kijsirikul , 2001 ) and decision trees with boosting and bagging ( Sun , 2002 ) have relied upon large labeled training setsthousands of instances -for classifier learning .

This labelled training data is costly to construct , both in terms of time and money , with estimates for some intonation annotation tasks reaching tens of times realtime .

This annotation bottleneck as well as a theoretical interest in the learning of tone motivates the use of unsupervised or semi-supervised approaches to tone recognition whereby the reliance on this often scarce resource can be reduced .

Little research has been done in the application of unsupervised and semi-supervised techniques for tone and pitch accent recognition .

Some preliminary work by ( Gauthier et al . , 2005 ) employs selforganizing maps and measures of f0 velocity for tone learning .

In this paper we explore the use of spectral and standard k-means clustering for unsupervised acquisition of tone , and the framework of manifold regularization for semi-supervised tone learning .

We find that in clean read speech , unsupervised techniques can identify the underlying Mandarin tone categories with high accuracy , while even on noisier broadcast news speech , Mandarin tones can be recognized well above chance levels , with English pitch accent recognition at near the levels achieved with fully supervised Support Vector Machine ( SVM ) classifiers .

Likewise in the semi-supervised framework , tone classification outperforms both most common class assignment and a comparable SVM trained on only the same small set of labeled instances , without recourse to the unlabeled instances .

The remainder of paper is organized as follows .

Section 2 describes the data sets on which English pitch accent and Mandarin tone learning are performed and the feature extraction process .

Section 3 describes the unsupervised and semisupervised techniques employed .

Sections 4 and 5 describe the experiments and results in unsupervised and semi-supervised frameworks respectively .

Section 6 presents conclusions and future work .

We have demonstrated the effectiveness of both unsupervised and semi-supervised techniques for recognition of Mandarin Chinese syllable tones and English pitch accents using acoustic features alone to capture pitch target height and slope .

Although outperformed by fully supervised classification techniques using much larger samples of labelled training data , these unsupervised and semi-supervised techniques perform well above most common class assignment , in the best cases approaching 90 % of supervised levels , and , where comparable , well above a good discriminative classifier trained on a comparably small set of labelled data .

Unsupervised techniques achieve accuracies of 87 % on the cleanest read speech , reaching 57 % on data from a standard Mandarin broadcast news corpus , and over 78 % on pitch accent classification for English broadcast news .

Semi-supervised classification in the Mandarin four-class classification task reaches 94 % accuracy on read speech , 70 % on broadcast news data , improving dramatically over both the simple baseline of 25 % and a standard SVM with an RBF kernel trained only on the labeled examples .

Future work will consider a broader range of tone and intonation classification , including the richer tone set of Cantonese as well as Bantu family tone languages , where annotated data truly is very rare .

We also hope to integrate a richer contextual representation of tone and intonation consistent with phonetic theory within this unsupervised and semisupervised learning framework .

We will further explore improvements in classification accuracy based on increases in labeled and unlabeled training examples .


There is a need to develop a suitable computational grammar formalism for free word order languages for two reasons :

First , a suitably designed formalism is likely to be more efficient .

Second , such a formalism is also likely to be linguistically more elegant and satisfying .

In this paper , we describe such a formalism , called the Paninian framework , that has been successfully applied to Indian languages .

This paper shows that the Paninian framework applied to modern Indian languages gives an elegant account of the relation between surface form ( vibhakti ) and semantic ( karaka ) roles .

The same basic account also explains active-passives and complex sentences .

This suggests that the solution is not just adhoc but has a deeper underlying unity .

A constraint based parser is described for the framework .

The constraints problem reduces to bipartite graph matching problem because of the nature of constraints .

Efficient solutions are known for these problems .

It is interesting to observe that such a parser ( designed for free word order languages ) compares well in asymptotic time complexity with the parser for context free grammars ( CFGs ) which are basically designed for positional languages .

A majority of human languages including Indian and other languages have relatively free word order .

tn free word order languages , order of words contains only secondary information such as emphasis etc .

Primary information relating to ' gross ' meaning ( e.g. , one that includes semantic relationships ) is contained elsewhere .

Most existing computational grammars are based on context free grammars which are basically positional grammars .

It is important to develop a suitable computational grammar formalism for free word order languages for two reasons :

A suitably designed formalism will be more efficient because it will be able to make use of primary sources of information directly .

Such a formalism is also likely to be linguistically more elegant and satisfying .

Since it will be able to relate to primary sources of information , the grammar is likely to be more economical and easier to write .

It uses the notion of karaka relations between verbs and nouns in a sentence .

The notion of karaka relations is central to the Paninian model .

The karaka relations are syntactico-semantic ( or semantico-syntactic ) relations between the verbals and other related constituents in a sentence .

They by themselves do not give the semantics .

Instead they specify relations which mediate between vibhakti of nominals and verb forms on one hand and semantic relations on the other ( Kiparsky , 1982 ) ( Cardona ( 1976 ( Cardona ( ) , ( 1988 ) ) ) .

Two of the important karakas are karta karaka and karma karaka .

Frequently , the karta karaka maps to agent theta role , and the karma to theme or goal theta role .

Here we will not argue for the linguistic significance of karaka relations and differences with theta relations , as that has been done elsewhere ( Bharati et al . ( 1990 ) and ( 1992 ) ) .

In summary , karta karaka is that participant in the action that is most independent .

At times , it turns out to be the agent .

Thus , ' boy ' and ' key ' are respectively the karta karakas in the following sentences 1The Paninian framework was originally designed more than two millennia ago for writing a grammar of Sanskrit ; it has been adapted by us to deal with modern Indian languages .

Note that in the first sentence , the karta ( boy ) maps to agent theta role , while in the second , karta ( key ) maps to instrument theta role .

As part of this framework , a mapping is specified between karaka relations and vibhakti ( which covers A. 2 collectively case endings , post-positional markers , etc . ) .

This mapping between karakas and vibhakti depends on the verb and its tense aspect modality ( TAM ) label .

The mapping is represented by two structures : default karaka charts and karaka chart transformations .

The default karaka chart for a verb or a class of verbs gives the mapping for the TAM label tA_hE called basic .

It specifies the vibhakti permitted for the applicable karaka relations for a verb when the verb has the basic TAM label .

This basic TAM label roughly corresponds to present indefinite tense and is purely syntactic in nature .

For other B. 1 TAM labels there are karaka chart transformation rules .

Thus , for a given verb with some TAM label , appropriate karaka chart can be obtained using its basic karaka chart and the transformation rule B.2 depending on its TAM label .

In Hindi for instance , the basic TAM label is tA_hE ( which roughly stands for the present indefinite ) .

The default karaka chart for three of the B.3 karakas is given in Fig .

This explains the vibhaktis in sentences A.1 to A.2 .

In A.1 and A.2 , ' Ram ' is karta and ' Mohan ' is karma , because of their vibhakti markers ¢ and ko , respectively .

3 ( Note that B.4 ' rAma ' is followed by ¢ or empty postposition , and ' mohana ' by ' ko ' postposition . )

2The transformation rules are a device to represent the karaka charts more compactly .

However , as is obvious , they affect the karaka charts and not the parse structure .

Therefore , they are different from transformational granmlars .

Formally , these rules can be eliminated by having separate karaka charts for each TAM label .

But one would miss the liguistic generalization of relating the karaka charts based on TAM labels in a systematic manner .

3In the present examples karta and karma tm'n out to be agent and theme , respectively .

Fig . 3 gives some transformation rules for the default mapping for Hindi .

It explains the vibhakti in sentences B.1 to B.4 , where Ram is the karta but has different vibhaktis , ¢ , he , ko , se , respectively .

In each of the sentences , if we transform the karaka chart of Fig . 2 by the transformation rules of Fig . 3 , we get the desired vibhakti for the karta Ram .

In general , the transformations affect not only the vibhakti of karta but also that of other karakas .

They also ' delete ' karaka roles at times , that is , the ' deleted ' karaka roles must not occur in the sentence .

The Paninian framework is similar to the broad class of case based grammars .

What distinguishes the Paninian framework is the use of karaka relations rather than theta roles , and the neat dependence of the karaka vibhakti mapping on TAMs and the transformation rules , in case of Indian languages .

The same principle also solves the problem of karaka assignment for complex sentences ( Discussed later in Sec .

In summary , this paper makes several contributions :

• It shows that the Paninian framework applied to modern Indian languages gives an elegant account of the relation between vibhakti and karaka roles .

8The modified verb in the present sentences is the main verb .

The same basic account also explains activepassives and complex sentences in these languages .

This suggest that the solution is not just adhoc but has a deeper underlying unity .

• It shows how a constraint based parser can be built using the framework .

Efficient solutions are known for these problems .

A parser for Indian languages based on the Paninian theory is operational as part of a machine translation system .

As part of our future work , we plan to apply this framework to other free word order languages ( i.e. , other than the Indian languages ) .

This theory can also be attempted on positional languages such as English .

What is needed is the concept of generalized vibhakti in which position of a word gets incoporated in vibhakti .

Thus , for a pure free word order language , the generalized vibhakti contains preor post-positional markers , whereas for a pure positional language it contains position information of a word ( group ) .

Clearly , for most natural languages , generalized vibhakti would contain information pertaining to both markers and position .

• It shows how a constraint based parser can be built using the framework .


We present a novel feature attribution method for explaining text classifiers , and analyze it in the context of hate speech detection .

Although feature attribution models usually provide a single importance score for each token , we instead provide two complementary and theoreticallygrounded scores -necessity and sufficiencyresulting in more informative explanations .

We propose a transparent method that calculates these values by generating explicit perturbations of the input text , allowing the importance scores themselves to be explainable .

We employ our method to explain the predictions of different hate speech detection models on the same set of curated examples from a test suite , and show that different values of necessity and sufficiency for identity terms correspond to different kinds of false positive errors , exposing sources of classifier bias against marginalized groups .

Explainability in AI ( XAI ) is critical in reaching various objectives during a system 's development and deployment , including debugging the system , ensuring its fairness , safety and security , and understanding and appealing its decisions by end-users ( Vaughan and Wallach , 2021 ; Luo et al . , 2021 ) .

A popular class of local explanation techniques is feature attribution methods , where the aim is to provide scores for each feature according to how important that feature is for the classifier decision for a given input .

From an intuitive perspective , one issue with feature attribution scores is that it is not always clear how to interpret the assigned importance in operational terms .

Specifically , saying that a feature is ' important ' might translate to two different predictions .

The first interpretation is that if an important feature value is changed , then the prediction will change .

The second interpretation is that , as long as the feature remains , the prediction will not change .

The former interpretation corresponds to the necessity of the feature value , while the latter corresponds to its sufficiency .

To further illustrate the difference between necessity and sufficiency , we take an example from hate speech detection .

Consider the utterance " I hate women " .

For a perfect model , the token ' women ' should have low sufficiency for a positive prediction , since merely the mention of this identity group should not trigger a hateful prediction .

However , this token should have fairly high necessity , since a target identity is required for an abusive utterance to count as hate speech ( e.g. , " I hate oranges " should not be classified as hate speech ) .

In this paper , we develop a method to estimate the necessity and sufficiency of each word in the input , as explanations for a binary text classifier 's decisions .

Model-agnostic feature attribution methods like ours often perturb the input to be explained , obtain the predictions of the model for the perturbed instances , and aggregate the results to make conclusions about which input features are more influential on the model decision .

When applying these methods to textual data , it is common to either drop the chosen tokens , or replace them with the mask token for those models that have been trained by fine-tuning a masked language model such as BERT ( Devlin et al . , 2019 ) .

However , deleting tokens raises the possibility that a large portion of the perturbed examples are not fluent , and lie well outside the data manifold .

Replacing some tokens with the mask token partially remedies this issue , however it raises others .

Firstly , the explanation method ceases to be truly model-agnostic .

Secondly , a masked sentence is in-distribution for the pre-trained model but out-of-distribution for the fine-tuned model , because the learned manifolds deviate from those formed during pre-training in the fine-tuning step .

To avoid these problems we use a generative model to replace tokens with most probable n-grams .

Generating perturbations in this way ensures that the perturbed instances are close to the true data manifold .

It also provides an additional layer of transparency to the user , so they can decide whether to trust the explanation by checking how reasonable the perturbed examples seem .

Although supervised discriminative models rely fundamentally on correlations within the dataset , different models might rely on different correlations more or less depending on model architecture and biases , training methods , and other idiosyncrasies .

To capture the distinction between correlations in the data and the direct causes of the prediction , we turn to the notion of interventions from causal inference ( Pearl , 2009 ) .

Previous work employing causal definitions of necessity and sufficiency for XAI have assumed tabular data with binary or numerical features .

The situation in NLP is much more complex , since each feature is a word in context , and we have no concept of ' flipping ' or ' increasing ' feature values ( as in binary data and numerical data , respectively ) .

Instead , our method generates perturbations of the input text that have high probability of being fluent while minimizing the probability that the generated text will also be a direct cause of the prediction we aim to explain .

As our application domain we choose hate speech detection , a prominent NLP task with significant social outcomes ( Fortuna and Nunes , 2018 ; Kiritchenko et al . , 2021 ) .

It has been shown that contemporary hate speech classifiers tend to learn spurious correlations , including those between identity terms and the positive ( hate ) class , which can result in further discrimination of already marginalized groups ( Dixon et al . , 2018 ; Park et al . , 2018 ; Garg et al . , 2019 ) .

We apply our explainability metrics to test classifiers ' fairness towards identity-based groups ( e.g. , women , Muslims ) .

We show how necessity and sufficiency metrics calculated for identity terms over hateful sentences can explain the classifier 's behaviour on non-hateful statements , highlighting classifiers ' tendencies to over-rely on the presence of identity terms or to ignore the characteristics of the object of abuse ( e.g. , protected identity groups vs. non-human entities ) .

The contributions of this work are as follows :

• We present the first methodology for calculating necessity and sufficiency metrics for text data as a feature attribution method .

Arguably , this dual explainability measure is more informative and allows for deeper insights into a model 's inner workings than traditional single metrics .

• We use a generative model for producing input perturbations to avoid the out-of-distribution prediction issues that emerge with token deletion and masking techniques .

• To evaluate the new methodology , we apply it to the task of explaining hate speech classification , and demonstrate that it can detect and explain biases in hate speech classifiers .

We make the implementation code freely available to researchers to facilitate further advancement of explainability techniques for NLP.1

This work is a step towards more informative and transparent feature attribution metrics for explaining text classifiers .

We argue that standard token importance metrics can be ambiguous in terms of what ' importance ' means .

Instead , we adapt the theoretically-grounded concepts of necessity and sufficiency to explain text classifiers .

Besides being more informative , the process of generating these two metrics is intuitive and can be explained to lay people in terms of " how much the perturbations in input change the output of the classifier " .

Moreover , the input perturbations can be presented to the users , leading to a transparent and understandable explainability framework .

Considering the complexities of perturbing textual features , we introduced a practical implementation to compute the necessity and sufficiency of the input tokens .

Taking hate speech detection as an example application , we showed that sufficiency and necessity can be used to explain the expected differences between a classifier that is intended to detect identity-based hate speech and those trained for detecting general abuse .

We also leveraged these metrics to explain the observed over-sensitivity and under-sensitivity to mentions of target groups , issues that are tightly related to fairness in hate speech detection .

While the current work focused on binary hate speech detection for English-language social media posts , in future work , we will explore the effectiveness of these metrics in generating explanations for other applications and languages .

We will also explore how the new metrics can improve the debugging of the models or communicating the model 's decisionmaking process to the end-users .


Call centers handle customer queries from various domains such as computer sales and support , mobile phones , car rental , etc .

Each such domain generally has a domain model which is essential to handle customer complaints .

These models contain common problem categories , typical customer issues and their solutions , greeting styles .

Currently these models are manually created over time .

Towards this , we propose an unsupervised technique to generate domain models automatically from call transcriptions .

We use a state of the art Automatic Speech Recognition system to transcribe the calls between agents and customers , which still results in high word error rates ( 40 % ) and show that even from these noisy transcriptions of calls we can automatically build a domain model .

The domain model is comprised of primarily a topic taxonomy where every node is characterized by topic(s ) , typical Questions-Answers ( Q&As ) , typical actions and call statistics .

We show how such a domain model can be used for topic identification of unseen calls .

We also propose applications for aiding agents while handling calls and for agent monitoring based on the domain model .

Call center is a general term for help desks , information lines and customer service centers .

Many companies today operate call centers to handle customer issues .

It includes dialog-based ( both voice and online chat ) and email support a user receives from a professional agent .

Call centers have become a central focus of most companies as they allow them to be in direct contact with their customers to solve product-related and servicesrelated issues and also for grievance redress .

A typical call center agent handles over a hundred calls in a day .

Gigabytes of data is produced every day in the form of speech audio , speech transcripts , email , etc .

This data is valuable for doing analysis at many levels , e.g. , to obtain statistics about the type of problems and issues associated with different products and services .

This data can also be used to evaluate agents and train them to improve their performance .

Today 's call centers handle a wide variety of domains such as computer sales and support , mobile phones and apparels .

To analyze the calls in any domain , analysts need to identify the key issues in the domain .

Further , there may be variations within a domain , say mobile phones , based on the service providers .

The analysts generate a domain model through inspection of the call records ( audio , transcripts and emails ) .

Such a model can include a listing of the call categories , types of problems solved in each category , listing of the customer issues , typical questions-answers , appropriate call opening and closing styles , etc .

In essence , these models provide a structured view of the domain .

Manually building such models for various domains may become prohibitively resource intensive .

Another important point to note is that these models are dynamic in nature and change over time .

As a new version of a mobile phone is introduced , software is launched in a country , a sudden attack of a virus , the model may need to be refined .

Hence , an automated way of creating and maintaining such a model is important .

In this paper , we have tried to formalize the essential aspects of a domain model .

It comprises of primarily a topic taxonomy where every node is characterized by topic(s ) , typical Questions-Answers ( Q&As ) , typical actions and call statistics .

To build the model , we first automatically transcribe the calls .

Current automatic speech recognition technology for telephone calls have moderate to high word error rates ( Padmanabhan et al . , 2002 ) .

We applied various feature engineering techniques to combat the noise introduced by the speech recognition system and applied text clustering techniques to group topically similar calls together .

Using clustering at different granularity and identifying the relationship between groups at different granularity we generate a taxonomy of call types .

This taxonomy is augmented with various meta information related to each node as mentioned above .

Such a model can be used for identification of topics of unseen calls .

Towards this , we envision an aiding tool for agents to increase agent effectiveness and an administrative tool for agent appraisal and training .

Organization of the paper : We start by describing related work in relevant areas .

Section 3 talks about the call center dataset and the speech recognition system used .

The following section contains the definition and describes an unsupervised mechanism for building a topical model from automatically transcribed calls .

Section 5 demonstrates the usability of such a topical model and proposes possible applications .

We have shown that it is possible to retrieve useful information from noisy transcriptions of call center voice conversations .

We have shown that the extracted information can be put in the form of a model that succinctly captures the domain and provides a comprehensive view of it .

We briefly showed through experiments that this model is an accurate description of the domain .

We have also suggested useful scenarios where the model can be used to aid and improve call center performance .

A call center handles several hundred-thousand calls per year in various domains .

It is very difficult to monitor the performance based on manual processing of the calls .

The framework presented in this paper , allows a large part of this work to be automated .

A domain specific model that is automatically learnt and updated based on the voice conversations allows the call center to identify problem areas quickly and allocate resources more effectively .

In future we would like to semantically cluster the topic specific information so that redundant topics are eliminated from the list .

We can use Automatic Taxonomy Generation(ATG ) algorithms for document summarization ( Kummamuru et al . , 2004 ) to build topic taxonomies .

We would also like to link our model to technical manuals , catalogs , etc . already available on the different topics in the given domain .

To build the model , we first automatically transcribe the calls .


Designing measures that capture various aspects of language ability is a central task in the design of systems for automatic scoring of spontaneous speech .

In this study , we address a key aspect of language proficiency assessment -syntactic complexity .

We propose a novel measure of syntactic complexity for spontaneous speech that shows optimum empirical performance on real world data in multiple ways .

First , it is both robust and reliable , producing automatic scores that agree well with human rating compared to the stateof-the-art .

Second , the measure makes sense theoretically , both from algorithmic and native language acquisition points of view .

Assessment of a speaker 's proficiency in a second language is the main task in the domain of automatic evaluation of spontaneous speech ( Zechner et al . , 2009 ) .

Prior studies in language acquisition and second language research have conclusively shown that proficiency in a second language is characterized by several factors , some of which are , fluency in language production , pronunciation accuracy , choice of vocabulary , grammatical sophistication and accuracy .

The design of automated scoring systems for non-native speaker speaking proficiency is guided by these studies in the choice of pertinent objective measures of these key aspects of language proficiency .

The focus of this study is the design and performance analysis of a measure of the syntactic complexity of non-native English responses for use in automatic scoring systems .

The state-ofthe art automated scoring system for spontaneous speech ( Zechner et al . , 2009 ; Higgins et al . , 2011 ) currently uses measures of fluency and pronunciation ( acoustic aspects ) to produce scores that are in reasonable agreement with human-rated scores of proficiency .

Despite its good performance , there is a need to extend its coverage to higher order aspects of language ability .

Fluency and pronunciation may , by themselves , already be good indicators of proficiency in non-native speakers , but from a construct validity perspective1 , it is necessary that an automatic assessment model measure higher-order aspects of language proficiency .

Syntactic complexity is one such aspect of proficiency .

By " syntactic complexity " , we mean a learner 's ability to use a wide range of sophisticated grammatical structures .

This study is different from studies that focus on capturing grammatical errors in non-native speakers ( Foster and Skehan , 1996 ; Iwashita et al . , 2008 ) .

Instead of focusing on grammatical errors that are found to be highly representative of language proficiency , our interest is in capturing the range of forms that surface in language production and the degree of sophistication of such forms , collectively referred to as syntactic complexity in ( Ortega , 2003 ) .

The choice and design of objective measures of language proficiency is governed by two crucial constraints :

Validity : a measure should show high discriminative ability between various levels of language proficiency , and the scores produced by the use of this measure should show high agreement with human-assigned scores .

2 . Robustness : a measures should be derived automatically and should be robust to errors in the measure generation process .

A critical impediment to the robustness constraint in the state-of-the-art is the multi-stage au-tomated process , where errors in the speech recognition stage ( the very first stage ) affect subsequent stages .

Guided by studies in second language development , we design a measure of syntactic complexity that captures patterns indicative of proficient and non-proficient grammatical structures by a shallow-analysis of spoken language , as opposed to a deep syntactic analysis , and analyze the performance of the automatic scoring model with its inclusion .

We compare and contrast the proposed measure with that found to be optimum in Yoon and Bhat ( 2012 ) .

Our primary contributions in this study are :

• We show that the measure of syntactic complexity derived from a shallow-analysis of spoken utterances satisfies the design constraint of high discriminative ability between proficiency levels .

In addition , including our proposed measure of syntactic complexity in an automatic scoring model results in a statistically significant performance gain over the state-of-the-art . •

The proposed measure , derived through a completely automated process , satisfies the robustness criterion reasonably well .

In the domain of native language acquisition , the presence or absence of a grammatical structure indicates grammatical development .

We observe that the proposed approach elegantly and effectively captures this presencebased criterion of grammatical development , since the feature indicative of presence or absence of a grammatical structure is optimal from an algorithmic point of view .

Seeking alternatives to measuring syntactic complexity of spoken responses via syntactic parsers , we study a shallow-analysis based approach for use in automatic scoring .

Empirically , we show that the proposed measure , based on a maximum entropy classification , satisfied the constraints of the design of an objective measure to a high degree .

In addition , the proposed measure was found to be relatively robust to ASR errors .

The measure outperformed a related measure of syntactic complexity ( also based on shallow-analysis of spoken response ) previously found to be well-suited for automatic scoring .

Including the measure of syntactic complexity in an automatic scoring model resulted in statistically significant performance gains over the stateof-the-art .

We also make an interesting observation that the impressionistic evaluation of syntactic complexity is better approximated by the presence or absence of grammar and usage patterns ( and not by their frequency of occurrence ) , an idea supported by studies in native language acquisition .


Dialogue act classification is a central challenge for dialogue systems .

Although the importance of emotion in human dialogue is widely recognized , most dialogue act classification models make limited or no use of affective channels in dialogue act classification .

This paper presents a novel affect-enriched dialogue act classifier for task-oriented dialogue that models facial expressions of users , in particular , facial expressions related to confusion .

The findings indicate that the affectenriched classifiers perform significantly better for distinguishing user requests for feedback and grounding dialogue acts within textual dialogue .

The results point to ways in which dialogue systems can effectively leverage affective channels to improve dialogue act classification .

Dialogue systems aim to engage users in rich , adaptive natural language conversation .

For these systems , understanding the role of a user 's utterance in the broader context of the dialogue is a key challenge ( Sridhar , Bangalore , & Narayanan , 2009 ) .

Central to this endeavor is dialogue act classification , which categorizes the intention behind the user 's move ( e.g. , asking a question , providing declarative information ) .

Automatic dialogue act classification has been the focus of a large body of research , and a variety of approaches , including sequential models ( Stolcke et al . , 2000 ) , vector-based models ( Sridhar , Bangalore , & Narayanan , 2009 ) , and most recently , featureenhanced latent semantic analysis ( Di Eugenio , Xie , & Serafin , 2010 ) , have shown promise .

These models may be further improved by leveraging regularities of the dialogue from both linguistic and extra-linguistic sources .

Users ' expressions of emotion are one such source .

Human interaction has long been understood to include rich phenomena consisting of verbal and nonverbal cues , with facial expressions playing a vital role ( Knapp & Hall , 2006 ; McNeill , 1992 ; Mehrabian , 2007 ; Russell , Bachorowski , & Fernandez-Dols , 2003 ; Schmidt & Cohn , 2001 ) .

While the importance of emotional expressions in dialogue is widely recognized , the majority of dialogue act classification projects have focused either peripherally ( or not at all ) on emotion , such as by leveraging acoustic and prosodic features of spoken utterances to aid in online dialogue act classification ( Sridhar , Bangalore , & Narayanan , 2009 ) .

Other research on emotion in dialogue has involved detecting affect and adapting to it within a dialogue system ( Forbes-Riley , Rotaru , Litman , & Tetreault , 2009 ; López-Cózar , Silovsky , & Griol , 2010 ) , but this work has not explored leveraging affect information for automatic user dialogue act classification .

Outside of dialogue , sentiment analysis within discourse is an active area of research ( López-Cózar et al . , 2010 ) , but it is generally lim-ited to modeling textual features and not multimodal expressions of emotion such as facial actions .

Such multimodal expressions have only just begun to be explored within corpus-based dialogue research ( Calvo & D'Mello , 2010 ; Cavicchio , 2009 ) .

This paper presents a novel affect-enriched dialogue act classification approach that leverages knowledge of users ' facial expressions during computer-mediated textual human-human dialogue .

Intuitively , the user 's affective state is a promising source of information that may help to distinguish between particular dialogue acts ( e.g. , a confused user may be more likely to ask a question ) .

We focus specifically on occurrences of students ' confusion-related facial actions during taskoriented tutorial dialogue .

Confusion was selected as the focus of this work for several reasons .

First , confusion is known to be prevalent within tutoring , and its implications for student learning are thought to run deep ( Graesser , Lu , Olde , Cooper-Pye , & Whitten , 2005 ) .

Second , while identifying the " ground truth " of emotion based on any external display by a user presents challenges , prior research has demonstrated a correlation between particular facial action units and confusion during learning ( Craig , D'Mello , Witherspoon , Sullins , & Graesser , 2004 ; D'Mello , Craig , Sullins , & Graesser , 2006 ; McDaniel et al . , 2007 ) .

Finally , automatic facial action recognition technologies are developing rapidly , and confusion-related facial action events are among those that can be reliably recognized automatically ( Bartlett et al . , 2006 ; Cohn , Reed , Ambadar , Xiao , & Moriyama , 2004 ; Pantic & Bartlett , 2007 ; Zeng , Pantic , Roisman , & Huang , 2009 ) .

This promising development bodes well for the feasibility of automatic real-time confusion detection within dialogue systems .

Emotion plays a vital role in human interactions .

In particular , the role of facial expressions in humanhuman dialogue is widely recognized .

Facial expressions offer a promising channel for understanding the emotions experienced by users of dialogue systems , particularly given the ubiquity of webcam technologies and the increasing number of dialogue systems that are deployed on webcamenabled devices .

This paper has reported on a first step toward using knowledge of user facial expressions to improve a dialogue act classification model for tutorial dialogue , and the results demonstrate that facial expressions hold great promise for distinguishing the pedagogically relevant dialogue act REQUEST FOR FEEDBACK , and the conversational moves of GROUNDING .

These early findings highlight the importance of future work in this area .

Dialogue act classification models have not fully leveraged some of the techniques emerging from work on sentiment analysis .

These approaches may prove particularly useful for identifying emotions in dialogue utterances .

Another important direction for future work involves more fully exploring the ways in which affect expression differs between textual and spoken dialogue .

Finally , as automatic facial tagging technologies mature , they may prove powerful enough to enable broadly deployed dialogue systems to feasibly leverage facial expression data in the near future .