Existing techniques for disambiguating named entities in text mostly focus on Wikipedia as a target catalog of entities . Yet for many types of entities , such as restaurants and cult movies , relational databases exist that contain far more extensive information than Wikipedia . This paper introduces a new task , called Open-Database Named-Entity Disambiguation ( Open-DB NED ) , in which a system must be able to resolve named entities to symbols in an arbitrary database , without requiring labeled data for each new database . We introduce two techniques for Open-DB NED , one based on distant supervision and the other based on domain adaptation . In experiments on two domains , one with poor coverage by Wikipedia and the other with near-perfect coverage , our Open-DB NED strategies outperform a state-of-the-art Wikipedia NED system by over 25 % in accuracy . Named-entity disambiguation ( NED ) is the task of linking names mentioned in text with an established catalog of entities ( Bunescu and Pasca , 2006 ; Ratinov et al . , 2011 ) . It is a vital first step for semantic understanding of text , such as in grounded semantic parsing ( Kwiatkowski et al . , 2011 ) , as well as for information retrieval tasks like person name search ( Chen and Martin , 2007 ; Mann and Yarowsky , 2003 ) . NED requires a catalog of symbols , called referents , to which named-entities will be resolved . Most NED systems today use Wikipedia as the catalog of referents , but exclusive focus on Wikipedia as a target for NED systems has significant drawbacks : despite its breadth , Wikipedia still does not contain all or even most real-world entities mentioned in text . As one example , it has poor coverage of entities that are mostly important in a small geographical region , such as hotels and restaurants , which are widely discussed on the Web . 57 % of the named-entities in the Text Analysis Conference 's ( TAC ) 2009 entity linking task refer to an entity that does not appear in Wikipedia ( McNamee et al . , 2009 ) . Wikipedia is clearly a highly valuable resource , but it should not be thought of as the only one . Instead of relying solely on Wikipedia , we propose a novel approach to NED , which we refer to as Open-DB NED : the task is to resolve an entity to Wikipedia or to any relational database that meets mild conditions about the format of the data , described below . Leveraging structured , relational data should allow systems to achieve strong accuracy , as with domain-specific or database-specific NED techniques like Hoffart et al . 's NED system for YAGO ( Hoffart et al . , 2011 ) . And because of the availability of huge numbers of databases on the Web , many for specialized domains , a successful system for this task will cover entities that a Wikipedia NED or database-specific system can not . We investigate two complementary learning strategies for Open-DB NED , both of which significantly relax the assumptions of traditional NED systems . The first strategy , a distant supervision approach , uses the relational information in a given database and a large corpus of unlabeled text to learn a database-specific model . The second strat-egy , a domain adaptation approach , assumes a single source database that has accompanying labeled data . Classifiers in this setting must learn a model that transfers from the source database to any new database , without requiring new training data for the new database . Experiments show that both strategies outperform a state-of-the-art Wikipedia NED system by wide margins without requiring any labeled data from the test domain , highlighting the significant advantage of having domain-specific relational data . The next section contrasts Open-DB NED with previous work . Section 3 formalizes the task . Sections 4 and 5 present our distant supervision strategy and domain-adaptation strategy , respectively . Section 6 introduces a technique that is a hybrid of the two learning strategies . Section 7 describes our experiments , and Section 8 concludes . This paper introduces the task of Open-DB Named Entity Disambiguation , and presents two distinct strategies for solving this task . Experiments indicate that a mixture of the two strategies significantly outperforms a state-of-the-art Wikipedia NED system , on a dataset where Wikipedia has good coverage and on another dataset where Wikipedia has poor coverage . The results indicate that there is a significant benefit to leveraging other sources of knowledge in addition to Wikipedia , and that it is possible to leverage this knowledge without requiring labeled data for each new source . The initial success of these Open-DB NED approaches indicates that this task is a promising area for future research , including exciting extensions that link large numbers of domainspecific databases to text .

Existing approaches to disambiguate named entities solely use Wikipedia as a catalogue however Many kinds of named entities are missed in Wikipedia.

They propose a task where systems need to reference arbitrary databases for finding named entities not only Wikipedia, together with methods to achieve domain adaptation.

A mixture of two domain adaptation methods outperforms existing systems that only rely on Wikipedia for their new Open-DB Named Entity Disambiguation task.


Solving algebraic word problems has recently emerged as an important natural language processing task . To solve algebraic word problems , recent studies suggested neural models that generate solution equations by using ' Op ( operator / operand ) ' tokens as a unit of input / output . However , such a neural model suffered two issues : expression fragmentation and operand-context separation . To address each of these two issues , we propose a pure neural model , Expression-Pointer Transformer ( EPT ) , which uses ( 1 ) ' Expression ' token and ( 2 ) operand-context pointers when generating solution equations . The performance of the EPT model is tested on three datasets : ALG514 , DRAW-1 K , and MAWPS . Compared to the state-of-the-art ( SoTA ) models , the EPT model achieved a comparable performance accuracy in each of the three datasets ; 81.3 % on ALG514 , 59.5 % on DRAW-1 K , and 84.5 % on MAWPS . The contribution of this paper is two-fold ; ( 1 ) We propose a pure neural model , EPT , which can address the expression fragmentation and the operandcontext separation . ( 2 ) The fully automatic EPT model , which does not use hand-crafted features , yields comparable performance to existing models using hand-crafted features , and achieves better performance than existing pure neural models by at most 40 % . Solving algebraic word problems has recently become an important research task in that automatically generating solution equations requires understanding natural language . Table 1 shows a sample algebraic word problem , along with corresponding solution equations that are used to generate answers for the problem . To solve such problems with deep learning technology , researchers recently suggested neural models that generate solution equations automatically ( Huang Problem One number is eight more than twice another and their sum is 20 . What are their numbers ? Numbers 1('one ' ) , 8('eight ' ) , 2('twice ' ) , 20 . Equations x 0 -2x 1 = 8 , x 0 + x 1 = 20 Answers ( 16 , 4 ) et al . , 2018 ; Amini et al . , 2019 ; Chiang and Chen , 2019 ; Wang et al . , 2019 ) . However , suggested neural models showed a fairly large performance gap compared to existing state-of-the-art models based on hand-crafted features in popular algebraic word problem datasets , such as ALG514 ( 44.5 % for pure neural model vs. 83.0 % for using hand-crafted features ) ( Huang et al . , 2018 ; Upadhyay and Chang , 2016 ) . To address the large performance gap in this study , we propose a larger unit of input / output ( I / O ) token called " Expressions " for a pure neural model . Figure 1 illustrates conventionally used " Op ( operator / operands ) " versus our newly proposed " Expression " token . To improve the performance of pure neural models that can solve algebraic word problems , we identified two issues that can be addressed using Expression tokens , which are shown in Figure 1 : ( 1 ) expression fragmentation and ( 2 ) operandcontext separation . First , the expression fragmentation issue is a segmentation of an expression tree , which represents a computational structure of equations that are used to generate a solution . This issue arises when Op , rather than the whole expression tree , is used as an input / output unit of a problem-solving model . For example , as shown in Figure 1 ( a ) , using Op tokens as an input to a problem-solving model disassembles a tree structure into operators ( " × " ) and operands ( " x 1 " and " 2 " ) . Meanwhile , we propose using the Figure 1 : Illustration using the word problem in Table 1 for the ( a ) expression fragmentation issue , ( b ) operandcontext separation issue , and ( c ) our solution for these two issues . " Expression " ( ×(x 1 , 2 ) ) token , which can explicitly capture a tree structure as a whole , as shown in Figure 1 ( c ) . The second issue of operand-context separation is the disconnection between an operand and a number that is associated with the operand . This issue arises when a problem-solving model substitutes a number stated in an algebraic word problem into an abstract symbol for generalization . As shown in Figure 1 ( b ) , when using an Op token , the number 8 is changed into an abstract symbol ' N 1 ' . Meanwhile , when using an Expression token , the number 8 is not transformed into a symbol . Rather a pointer is made to the location where the number 8 occurred in an algebraic word problem . Therefore , using such an " operand-context pointer " enables a model to access contextual information about the number directly , as shown in Figure 1 ( c ) ; thus , the operand-context separation issue can be addressed . In this paper , we propose a pure neural model called Expression-Pointer Transformer ( EPT ) to address the two issues above . The contribution of this paper is two-fold ; 1 . We propose a pure neural model , Expression-Pointer Transformer ( EPT ) , which can address the expression fragmentation and operandcontext separation issues . 2 . The EPT model is the first pure neural model that showed comparable accuracy to the existing state-of-the-art models , which used handcrafted features . Compared to the state-ofthe-art pure neural models , the EPT achieves better performance by about 40 % . In the rest of the paper , we introduce existing approaches to solve algebraic word problems in Section 2 . Next , Section 3 introduces our proposed model , EPT , and Section 4 reports the experimental settings . Then in Section 5 , results of two studies are presented . Section 5.1 presents a performance comparison between EPT and existing SoTA models . Section 5.2 presents an ablation study examining the effects of Expression tokens and applying operand-context pointers . Finally , in Section 6 , a conclusion is presented with possible future directions for our work . In this study , we proposed a neural algebraic word problem solver , Expression-Pointer Transformer ( EPT ) , and examined its characteristics . We designed EPT to address two issues : expression fragmentation and operand-context separation . The EPT resolves the expression fragmentation issue by generating ' Expression ' tokens , which simultaneously generate an operator and required operands . In addition , the EPT resolves the operand-context separation issue by applying operand-context pointers . Our work is meaningful in that we demonstrated a possibility for alleviating the costly procedure of devising hand-crafted features in the domain of solving algebraic word problems . As future work , we plan to generalize the EPT to other datasets , including non-English word problems or non-algebraic domains in math , to extend our model .

Neural models largely underperform hand-crafted feature-based models on algebraic word datasets such as ALG514 because of two issues namely expression fragmentation and operand-context separation.

They propose a model which generates an operator and required operands and applies operand-context pointers to resolve the expression fragmentation and operand-context separation issues respectively.

The proposed model performs comparable results to the state-of-the-art models with hand-crafted features and outperforms neural models by 40% on three datasets.


In this paper I will report the result of a quantitative analysis of the dynamics of the constituent elements of Japanese terminology . In Japanese technical terms , the linguistic contribution of morphemes greatly differ according to their types of origin . To analyse this aspect , a quantitative method is applied , which can properly characterise the dynamic nature of morphemes in terminology on the basis of a small sample . In computational linguistics , the interest in terminological applications such as automatic term extraction is growing , and many studies use the quantitative information ( cf . Kageura & Umino , 1996 ) . However , the basic quantitative nature of terminological structure , which is essential for terminological theory and applications , has not yet been exploited . The static quantitative descriptions are not sufficient , as there are terms which do not appear in the sample . So it is crucial to establish some models , by which the terminological structure beyond the sample size can be properly described . In Japanese terminology , the roles of morphemes are different according to their types of origin , i.e. the morphemes borrowed mainly from Western languages ( borrowed morphemes ) and the native morphemes including Chineseorigined morphemes which are the majority . There are some quantitative studies ( Ishii , 1987 ; Nomura & Ishii , 1989 ) , but they only treat the static nature of the sample . Located in the intersection of these two backgrounds , the aim of the present study is twofold , i.e. ( 1 ) to introduce a quantitative framework in which the dynamic nature of terminology can be described , and to examine its theoretical validity , and ( 2 ) to describe the quantitative dynamics of morphemes as a ' mass ' in Japanese terminology , with reference to the types of origin . Using the LNRE models , we have succesfully analysed the dynamic nature of the morphemes in Japanese terminology . As the majority of the terminological data is located in the LNRE zone , it is important to use the statistical framework which allows for the LNRE characteristics . The LNRE models give the suitable means . We are currently extending our research to integrating the quantitative nature of morphological distributions to the qualitative mode ] of term formation , by taking into account the po-sitional and combinatorial nature of morphemes and the distributions of term length .

Static quantitative descriptions are not sufficient to analyse Japanese terminology because of the dynamic nature of samples calling for a method beyond the sample size.

They apply a quantitative method which can characterise the dynamic nature of morphemes using a small sample of Japanese terminology.

They show that the method can successfully analyze the dynamic nature of the morphemes in Japanese terminology with suitable means.


We study the problem of leveraging the syntactic structure of text to enhance pre-trained models such as BERT and RoBERTa . Existing methods utilize syntax of text either in the pre-training stage or in the fine-tuning stage , so that they suffer from discrepancy between the two stages . Such a problem would lead to the necessity of having human-annotated syntactic information , which limits the application of existing methods to broader scenarios . To address this , we present a model that utilizes the syntax of text in both pre-training and fine-tuning stages . Our model is based on Transformer with a syntax-aware attention layer that considers the dependency tree of the text . We further introduce a new pre-training task of predicting the syntactic distance among tokens in the dependency tree . We evaluate the model on three downstream tasks , including relation classification , entity typing , and question answering . Results show that our model achieves state-of-the-art performance on six public benchmark datasets . We have two major findings . First , we demonstrate that infusing automatically produced syntax of text improves pre-trained models . Second , global syntactic distances among tokens bring larger performance gains compared to local head relations between contiguous tokens . 1 * Work is done during internship at Microsoft . † For questions , please contact D. Tang and Z. Xu . Pre-trained models such as BERT ( Devlin et al . , 2019 ) , GPT ( Radford et al . , 2018 ) , and RoBERTa ( Liu et al . , 2019 ) have advanced the state-of-the-art performances of various natural language processing tasks . The successful recipe is that a model is first pre-trained on a huge volume of unsupervised data with self-supervised objectives , and then is fine-tuned on supervised data with the same data scheme . Dominant pre-trained models represent a text as a sequence of tokens2 . The merits are that such basic text representations are available from vast amounts of unsupervised data , and that models pre-trained and fine-tuned with the same paradigm usually achieve good accuracy in practice ( Guu et al . , 2020 ) . However , an evident limitation of these methods is that richer syntactic structure of text is ignored . In this paper , we seek to enhance pre-trained models with syntax of text . Related studies attempt to inject syntax information either only in the finetuning stage ( Nguyen et al . , 2020 ; Sachan et al . , 2020 ) , or only in the pre-training stage ( Wang et al . , 2020 ) , which results in discrepancies . When only fusing syntax information in the fine-tuning phase , Sachan et al . ( 2020 ) finds that there is no performance boost unless high quality human-annotated dependency parses are available . However , this requirement would limit the application of the model to broader scenarios where human-annotated dependency information is not available . To address this , we conduct a large-scale study on injecting automatically produced syntax of text in both the pre-training and fine-tuning stages . We construct a pre-training dataset by applying an offthe-shelf dependency parser ( Qi et al . , 2020 ) to one billion sentences from common crawl news . With these data , we introduce a syntax-aware pretraining task , called dependency distance prediction , which predicts the syntactic distance between tokens in the dependency structure . Compared with the pre-training task of dependency head prediction ( Wang et al . , 2020 ) that only captures local syntactic relations among words , dependency distance prediction leverages global syntax of the text . In addition , we developed a syntax-aware attention layer , which can be conveniently integrated into Transformer ( Vaswani et al . , 2017 ) to allow tokens to selectively attend to contextual tokens based on their syntactic distance in the dependency structure . We conduct experiments on entity typing , question answering and relation classification on six benchmark datasets . Experimental results show that our method achieves state-of-the-art performance on all six datasets . Further analysis shows that our model can indicate the importance of syntactic information on downstream tasks , and that the newly introduced dependency distance prediction task could capture the global syntax of the text , performs better than dependency head prediction . In addition , compared with experimental results of injecting syntax information in either the pre-training or fine-tuning stage , injecting syntax information in both stages achieves the best performance . In summary , the contribution of this paper is threefold . ( 1 ) We demonstrate that infusing automatically produced dependency structures into the pre-trained model shows superior performance over downstream tasks . ( 2 ) We propose a syntax-aware attention layer and a pre-training task for infusing syntactic information into the pre-trained model . ( 3 ) We find that the newly introduced dependency distance prediction task performs better than the dependency head prediction task . In this paper , we present SEPREM that leverage syntax information to enhance pre-trained models . To inject syntactic information , we introduce a syntax-aware attention layer and a newly designed pre-training task are proposed . Experimental results show that our method achieves state-of-theart performance over six datasets . Further analysis shows that the proposed dependency distance prediction task performs better than dependency head prediction task .

Existing ways of injecting syntactic knowledge into pretraining models cause discrepancies between pretraining and fine-tuning and require expensive annotation.

They propose to inject syntactic features obtained by an off-the-shelf parser into pretraining models coupled with a new syntax-aware attention layer.

The proposed model achieves state-of-the-art in relation classification, entity typing, and question answering tasks.


This paper proposes an unsupervised approach for segmenting a multiauthor document into authorial components . The key novelty is that we utilize the sequential patterns hidden among document elements when determining their authorships . For this purpose , we adopt Hidden Markov Model ( HMM ) and construct a sequential probabilistic model to capture the dependencies of sequential sentences and their authorships . An unsupervised learning method is developed to initialize the HMM parameters . Experimental results on benchmark datasets have demonstrated the significant benefit of our idea and our approach has outperformed the state-of-the-arts on all tests . As an example of its applications , the proposed approach is applied for attributing authorship of a document and has also shown promising results . Authorship analysis is a process of inspecting documents in order to extract authorial information about these documents . It is considered as a general concept that embraces several types of authorship subjects , including authorship verif ication , plagiarism detection and author attribution . Authorship verification ( Brocardo et al . , 2013 ; Potha and Stamatatos , 2014 ) decides whether a given document is written by a specific author . Plagiarism detection ( Stein et al . , 2011 ; Kestemont et al . , 2011 ) seeks to expose the similarity between two texts . However , it is un-able to determine if they are written by the same author . In author attribution ( Juola , 2006 ; Savoy , 2015 ) , a real author of an anonymous document is predicted using labeled documents of a set of candidate authors . Another significant subject in authorship analysis , which has received comparatively less attention from research community , is authorship-based document decomposition ( ABDD ) . This subject is to group the sentences of a multi-author document to different classes , of which each contains the sentences written by only one author . Many applications can take advantage of such a subject , especially those in forensic investigation , which aim to determine the authorship of sentences in a multi-author document . Furthermore , this kind of subject is beneficial for detecting plagiarism in a document and defining contributions of authors in a multi-author document for commercial purpose . ABDD can also be applied to identify which source ( regarded as an ' author ' in this paper ) a part of a document is copied from when the document is formed by taking contents from various sources . In despite of the benefits of ABDD , there has been little research reported on this subject . Koppel et al . ( 2011 ) are the first researchers who implemented an unsupervised approach for ABDD . However , their approach is restricted to Hebrew documents only . The authors of Akiva and Koppel ( 2013 ) addressed the drawbacks of the above approach by proposing a generic unsupervised approach for ABDD . Their approach utilized distance measurements to increase the precision and accuracy of clustering and classification phases , respectively . The accuracy of their approach is highly dependent on the number of au-thors . When the number of authors increases , the accuracy of the approach is significantly dropped . Giannella ( 2015 ) presented an improved approach for ABDD when the number of authors of the document is known or unknown . In his approach , a Bayesian segmentation algorithm is applied , which is followed by a segment clustering algorithm . However , the author tested his approach by using only documents with a few transitions among authors . Furthermore , the accuracy of the approach is very sensitive to the setting of its parameters . In Aldebei et al . ( 2015 ) , the authors presented an unsupervised approach ABDD by exploiting the differences in the posterior probabilities of a Naive-Bayesian model in order to increase the precision and the classification accuracy , and to be less dependent on the number of authors in comparing with the approach in Akiva and Koppel ( 2013 ) . Their work was tested on documents with up to 400 transitions among authors and the accuracy of their approach was not sensitive to the setting of parameters , in contrast with the approach in Giannella ( 2015 ) . However , the performance of their approach greatly depends on a threshold , of which the optimal value for an individual document is not easy to find . Some other works have focused on segmenting a document into components according to their topics . For applications where the topics of documents are unavailable , these topicbased solutions will fail . In this paper , the ABDD approach is independent of documents ' topics . All of the existing works have assumed that the observations ( i.e. , sentences ) are independent and identically distributed ( i.i.d . ) . No consideration has been given to the contextual information between the observations . However , in some cases , the i.i.d . assumption is deemed as a poor one ( Rogovschi et al . , 2010 ) . In this paper , we will relax this assumption and consider sentences of a document as a sequence of observations . We make use of the contextual information hidden between sentences in order to identify the authorship of each sentence in a document . In other words , the authorships of the " previous " and " subsequent " sentences have relationships with the authorship of the current sentence . There-fore , in this paper , a well-known sequential model , Hidden Markov Model ( HMM ) , is used for modelling the sequential patterns of the document in order to describe the authorship relationships . The contributions of this article are summarized as follows . 1 . We capture the dependencies between consecutive elements in a document to identify different authorial components and construct an HMM for classification . It is for the first time the sequential patterns hidden among document elements is considered for such a problem . 2 . To build and learn the HMM model , an unsupervised learning method is first proposed to estimate its initial parameters , and it does not require any information of authors or document 's context other than how many authors have contributed to write the document . 3 . Different from the approach in Aldebei et al . ( 2015 ) , the proposed unsupervised approach no longer relies on any predetermined threshold for ABDD . 4 . Comprehensive experiments are conducted to demonstrate the superior performance of our ideas on both widely-used artificial benchmark datasets and an authentic scientific document . As an example of its applications , the proposed approach is also applied for attributing authorship on a popular dataset . The proposed approach can not only correctly determine the author of a disputed document but also provide a way for measuring the confidence level of the authorship decision for the first time . The rest of this article is organised as follows . Section 2 reviews the HMM . Section 3 presents the details of our proposed approach , including the processes for initialization and learning of HMM parameters , and the Viterbi decoding process for classification . Experiments are conducted in Section 4 , followed by the conclusion in Section 5 . We have developed an unsupervised approach for decomposing a multi-author document based on authorship . Different from the stateof-the-art approaches , we have innovatively made use of the sequential information hidden among document elements . For this purpose , we have used HMM and constructed a sequential probabilistic model , which is used to find the best sequence of authors that represents the sentences of the document . An unsupervised learning method has also been developed to estimate the initial parameter values of HMM . Comparative experiments conducted on benchmark datasets have demonstrated the effectiveness of our ideas with superior perfor-mance achieved on both artificial and authentic documents . An application of the proposed approach on authorship attribution has also achieved perfect results of 100 % accuracies together with confidence measurement for the first time .

There is no method for multiauthor segmentation of a document into author components which can be applied to authorship verification, plagiarism detection and author attribution.

They propose a HMM-based sequential probabilistic model that captures the dependencies of sequential sentences and their authors coupled with an unsupervised initialization method.

Experiments with artificial and authentic scientific document datasets show that the proposed model outperforms existing methods and also be able to provide confidence scores.


We investigate two systems for automatic disfluency detection on English and Mandarin conversational speech data . The first system combines various lexical and prosodic features in a Conditional Random Field model for detecting edit disfluencies . The second system combines acoustic and language model scores for detecting filled pauses through constrained speech recognition . We compare the contributions of different knowledge sources to detection performance between these two languages . Speech disfluencies are common phenomena in spontaneous speech . They consist of spoken words and phrases that represent self-correction , hesitation , and floor-grabbing behaviors , but do not add semantic information ; removing them yields the intended , fluent utterance . The presence of disfluencies in conversational speech data can cause problems for both downstream processing ( parsing and other natural language processing tasks ) and human readability of speech transcripts . There has been much research effort on automatic disfluency detection in recent years ( Shriberg and Stolcke , 1997 ; Snover et al . , 2004 ; Liu et al . , 2006 ; Lin and Lee , 2009 ; Schuler et al . , 2010 ; Georgila et al . , 2010 ; Zwarts and Johnson , 2011 ) , particularly from the DARPA EARS ( Effective , Affordable , Reusable Speech-to-Text ) MDE ( MetaData Extraction ) ( DARPA Information Processing Technology Office , 2003 ) program , which focused on the automatic transcription of sizable amounts of speech data and rendering such transcripts in readable form , for both conversational telephone speech ( CTS ) and broadcast news ( BN ) . However , the EARS MDE effort was focused on English only , and there has n't been much research on the effectiveness of similar automatic disfluency detection approaches for multiple languages . This paper presents three main innovations . First , we extend the EARS MDE-style disfluency detection approach combining lexical and prosodic features using a Conditional Random Field ( CRF ) model , which was employed for detecting disfluency on English conversational speech data ( Liu et al . , 2005 ) , to Mandarin conversational speech , as presented in Section 2 . Second , we implement an automatic filled pause detection approach through constrained speech recognition , as presented in Section 3 . Third , for both disfluency detection systems , we compare side-by-side contributions of different knowledge sources to detection performance for two languages , English and Mandarin , as presented in Section 4 . Conclusions appear in Section 5 . In conclusion , we have presented two automatic disfluency detection systems , one combining various lexical and prosodic features , and the other combining LVCSR acoustic and language model knowledge sources . We observed significant improvements in combining lexical and prosodic features over just employing word n-gram features , for both languages . When combining AM and LM knowledge sources for FP detection in constrained speech recognition , we found increasing LM weight improved both false alarm and miss rates for Mandarin but degraded the miss rate for English .

Existing works on detecting speech disfluency which can be a problem for downstream processing and creating transcripts only focus on English.

They evaluate a Conditional Random Field-based edit disfluency detection model and a system which combines acoustic and language model that detects filled pauses in Mandarin.

Their system comparisons in English and Mandarin show that combining lexical and prosodic features achieves improvements in both languages.


We propose a method to generate large-scale encyclopedic knowledge , which is valuable for much NLP research , based on the Web . We first search the Web for pages containing a term in question . Then we use linguistic patterns and HTML structures to extract text fragments describing the term . Finally , we organize extracted term descriptions based on word senses and domains . In addition , we apply an automatically generated encyclopedia to a question answering system targeting the Japanese Information-Technology Engineers Examination . Reflecting the growth in utilization of the World Wide Web , a number of Web-based language processing methods have been proposed within the natural language processing ( NLP ) , information retrieval ( IR ) and artificial intelligence ( AI ) communities . A sample of these includes methods to extract linguistic resources ( Fujii and Ishikawa , 2000 ; Resnik , 1999 ; Soderland , 1997 ) , retrieve useful information in response to user queries ( Etzioni , 1997 ; McCallum et al . , 1999 ) and mine / discover knowledge latent in the Web ( Inokuchi et al . , 1999 ) . In this paper , mainly from an NLP point of view , we explore a method to produce linguistic resources . Specifically , we enhance the method proposed by Fujii and Ishikawa ( 2000 ) , which extracts encyclopedic knowledge ( i.e. , term descriptions ) from the Web . In brief , their method searches the Web for pages containing a term in question , and uses linguistic expressions and HTML layouts to extract fragments describing the term . They also use a language model to discard non-linguistic fragments . In addition , a clustering method is used to divide descriptions into a specific number of groups . On the one hand , their method is expected to enhance existing encyclopedias , where vocabulary size is relatively limited , and therefore the quantity problems has been resolved . On the other hand , encyclopedias extracted from the Web are not comparable with existing ones in terms of quality . In hand-crafted encyclopedias , term descriptions are carefully organized based on domains and word senses , which are especially effective for human usage . However , the output of Fujii 's method is simply a set of unorganized term descriptions . Although clustering is optionally performed , resultant clusters are not necessarily related to explicit criteria , such as word senses and domains . To sum up , our belief is that by combining extraction and organization methods , we can enhance both quantity and quality of Web-based encyclopedias . Motivated by this background , we introduce an organization model to Fujii 's method and reformalize the whole framework . In other words , our proposed method is not only extraction but generation of encyclopedic knowledge . Section 2 explains the overall design of our encyclopedia generation system , and Section 3 elaborates on our organization model . Section 4 then explores a method for applying our resultant encyclopedia to NLP research , specifically , question answering . Section 5 performs a number of experiments to evaluate our methods . The World Wide Web has been an unprecedentedly enormous information source , from which a number of language processing methods have been explored to extract , retrieve and discover various types of information . In this paper , we aimed at generating encyclopedic knowledge , which is valuable for many applications including human usage and natural language understanding . For this purpose , we reformalized an existing Web-based extraction method , and proposed a new statistical organization model to improve the quality of extracted data . Given a term for which encyclopedic knowledge ( i.e. , descriptions ) is to be generated , our method sequentially performs a ) retrieval of Web pages contain-ing the term , b ) extraction of page fragments describing the term , and c ) organizing extracted descriptions based on domains ( and consequently word senses ) . In addition , we proposed a question answering system , which answers interrogative questions associated with what , by using a Web-based encyclopedia as a knowledge base . For the purpose of evaluation , we used as test inputs technical terms collected from the Class II IT engineers examination , and found that the encyclopedia generated through our method was of operational quality and quantity . We also used test questions from the Class II examination , and evaluated the Web-based encyclopedia in terms of question answering . We found that our Webbased encyclopedia improved the system coverage obtained solely with an existing dictionary . In addition , when we used both resources , the performance was further improved . Future work would include generating information associated with more complex interrogations , such as ones related to how and why , so as to enhance Webbased natural language understanding .

Existing methods that extract encyclopedic knowledge from the Web output unorganized clusters of term descriptions not necessarily related to explicit criteria while clustering is performed.

They propose to use word senses and domains for organizing extracted term descriptions on questions extracted from the Web to improve the quality.

The generated encyclopedia is applied to a Japanese question and answering system and improves over a system which solely depends on a dictionary.


Research on modeling time series text corpora has typically focused on predicting what text will come next , but less well studied is predicting when the next text event will occur . In this paper we address the latter case , framed as modeling continuous inter-arrival times under a log-Gaussian Cox process , a form of inhomogeneous Poisson process which captures the varying rate at which the tweets arrive over time . In an application to rumour modeling of tweets surrounding the 2014 Ferguson riots , we show how interarrival times between tweets can be accurately predicted , and that incorporating textual features further improves predictions . Twitter is a popular micro-blogging service which provides real-time information on events happening across the world . Evolution of events over time can be monitored there with applications to disaster management , journalism etc . For example , Twitter has been used to detect the occurrence of earthquakes in Japan through user posts ( Sakaki et al . , 2010 ) . Modeling the temporal dynamics of tweets provides useful information about the evolution of events . Inter-arrival time prediction is a type of such modeling and has application in many settings featuring continuous time streaming text corpora , including journalism for event monitoring , real-time disaster monitoring and advertising on social media . For example , journalists track several rumours related to an event . Predicted arrival times of tweets can be applied for ranking rumours according to their activity and narrow the interest to investigate a rumour with a short interarrival time over that of a longer one . Modeling the inter-arrival time of tweets is a challenging task due to complex temporal patterns exhibited . Tweets associated with an event stream arrive at different rates at different points in time . For example , Figure 1a shows the arrival times ( denoted by black crosses ) of tweets associated with an example rumour around Ferguson riots in 2014 . Notice the existence of regions of both high and low density of arrival times over a one hour interval . We propose to address inter-arrival time prediction problem with log-Gaussian Cox process ( LGCP ) , an inhomogeneous Poisson process ( IPP ) which models tweets to be generated by an underlying intensity function which varies across time . Moreover , it assumes a non-parametric form for the intensity function allowing the model complexity to depend on the data set . We also provide an approach to consider textual content of tweets to model inter-arrival times . We evaluate the models using Twitter rumours from the 2014 Ferguson unrest , and demonstrate that they provide good predictions for inter-arrival times , beating the baselines e.g. homogeneous Poisson Process , Gaussian Process regression and univariate Hawkes Process . Even though the central application is rumours , one could apply the proposed approaches to model the arrival times of tweets corresponding to other types of memes , e.g. discussions about politics . This paper makes the following contributions : 1 . Introduces log-Gaussian Cox process to predict tweet arrival times . 2 . Demonstrates how incorporating text improves results of inter-arrival time prediction . This paper introduced the log-Gaussian Cox processes for the problem of predicting the interarrival times of tweets . We showed how text from posts helps to achieve significant improvements . Evaluation on a set of rumours from Ferguson riots showed efficacy of our methods comparing to baselines . The proposed approaches are generalizable to problems other than rumours , e.g. disaster management and advertisement campaigns .

Modeling the event inter-arrival time of tweets is challenging due to complex temporal patterns but few works aim to predict the next text event occurrence.

They propose to apply a log-Gaussian Cox process model which captures the varying arriving rate over time coupled with the textual contents of tweets.

The proposed model outperforms baseline models on an inter-arrival time prediction task around a riots rumour and shows that it improves with textual features.


Hypertext documents , such as web pages and academic papers , are of great importance in delivering information in our daily life . Although being effective on plain documents , conventional text embedding methods suffer from information loss if directly adapted to hyper-documents . In this paper , we propose a general embedding approach for hyper-documents , namely , hyperdoc2vec , along with four criteria characterizing necessary information that hyper-document embedding models should preserve . Systematic comparisons are conducted between hyperdoc2vec and several competitors on two tasks , i.e. , paper classification and citation recommendation , in the academic paper domain . Analyses and experiments both validate the superiority of hyperdoc2vec to other models w.r.t . the four criteria . The ubiquitous World Wide Web has boosted research interests on hypertext documents , e.g. , personal webpages ( Lu and Getoor , 2003 ) , Wikipedia pages ( Gabrilovich and Markovitch , 2007 ) , as well as academic papers ( Sugiyama and Kan , 2010 ) . Unlike independent plain documents , a hypertext document ( hyper-doc for short ) links to another hyper-doc by a hyperlink or citation mark in its textual content . Given this essential distinction , hyperlinks or citations are worth specific modeling in many tasks such as link-based classification ( Lu and Getoor , 2003 ) , web retrieval ( Page et al . , 1999 ) , entity linking ( Cucerzan , 2007 ) , and citation recommendation ( He et al . , 2010 ) . To model hypertext documents , various efforts ( Cohn and Hofmann , 2000 ; Kataria et al . , 2010 ; Perozzi et al . , 2014 ; Zwicklbauer et al . , 2016 ; Wang et al . , 2016 ) have been made to depict networks of hyper-docs as well as their content . Among potential techniques , distributed representation ( Mikolov et al . , 2013 ; Le and Mikolov , 2014 ) tends to be promising since its validity and effectiveness are proven for plain documents on many natural language processing ( NLP ) tasks . Conventional attempts on utilizing embedding techniques in hyper-doc-related tasks generally fall into two types . The first type ( Berger et al . , 2017 ; Zwicklbauer et al . , 2016 ) simply downcasts hyper-docs to plain documents and feeds them into word2vec ( Mikolov et al . , 2013 ) ( w2v for short ) or doc2vec ( Le and Mikolov , 2014 ) ( d2v for short ) . These approaches involve downgrading hyperlinks and inevitably omit certain information in hyper-docs . However , no previous work investigates the information loss , and how it affects the performance of such downcasting-based adaptations . The second type designs sophisticated embedding models to fulfill certain tasks , e.g. , citation recommendation ( Huang et al . , 2015b ) , paper classification ( Wang et al . , 2016 ) , and entity linking ( Yamada et al . , 2016 ) , etc . These models are limited to specific tasks , and it is yet unknown whether embeddings learned for those particular tasks can generalize to others . Based on the above facts , we are interested in two questions : • What information should hyper-doc embedding models preserve , and what nice property should they possess ? • Is there a general approach to learning taskindependent embeddings of hyper-docs ? To answer the two questions , we formalize the hyper-doc embedding task , and propose four criteria , i.e. , content awareness , context awareness , newcomer friendliness , and context intent aware-ness , to assess different models . Then we discuss simple downcasting-based adaptations of existing approaches w.r.t . the above criteria , and demonstrate that none of them satisfy all four . To this end , we propose hyperdoc2vec ( h-d2v for short ) , a general embedding approach for hyperdocs . Different from most existing approaches , h-d2v learns two vectors for each hyper-doc to characterize its roles of citing others and being cited . Owning to this , h-d2v is able to directly model hyperlinks or citations without downgrading them . To evaluate the learned embeddings , we employ two tasks in the academic paper domain1 , i.e. , paper classification and citation recommendation . Experimental results demonstrate the superiority of h-d2v . Comparative studies and controlled experiments also confirm that h-d2v benefits from satisfying the above four criteria . We summarize our contributions as follows : • We propose four criteria to assess different hyper-document embedding models . • We propose hyperdoc2vec , a general embedding approach for hyper-documents . • We systematically conduct comparisons with competing approaches , validating the superiority of h-d2v in terms of the four criteria . We focus on the hyper-doc embedding problem . We propose that hyper-doc embedding algorithms should be content aware , context aware , newcomer friendly , and context intent aware . To meet all four criteria , we propose a general approach , hyperdoc2vec , which assigns two vectors to each hyper-doc and models citations in a straightforward manner . In doing so , the learned embeddings satisfy all criteria , which no existing model is able to . For evaluation , paper classification and citation recommendation are conducted on three academic paper datasets . Results confirm the effectiveness of our approach . Further analyses also demonstrate that possessing the four properties helps h-d2v outperform other models .

Existing text embedding methods do not take structures of hyper-documents into account losing useful properties for downstream tasks.

They propose an embedding method for hyper-documents that learns citation information along with four criteria to assess the properties the models should preserve.

The proposed model satisfies all of the introduced criteria and performs two tasks in the academic domain better than existing models.


Recurrent neural networks ( RNNs ) have achieved impressive results in a variety of linguistic processing tasks , suggesting that they can induce non-trivial properties of language . We investigate here to what extent RNNs learn to track abstract hierarchical syntactic structure . We test whether RNNs trained with a generic language modeling objective in four languages ( Italian , English , Hebrew , Russian ) can predict long-distance number agreement in various constructions . We include in our evaluation nonsensical sentences where RNNs can not rely on semantic or lexical cues ( " The colorless green ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas I ate with the chair sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep furiously " ) , and , for Italian , we compare model performance to human intuitions . Our language-model-trained RNNs make reliable predictions about long-distance agreement , and do not lag much behind human performance . We thus bring support to the hypothesis that RNNs are not just shallowpattern extractors , but they also acquire deeper grammatical competence . Recurrent neural networks ( RNNs ; Elman , 1990 ) are general sequence processing devices that do not explicitly encode the hierarchical structure that is thought to be essential to natural language ( Everaert et al . , 2015 ) . Early work using artificial languages showed that they may nevertheless be able to approximate context-free languages ( Elman , 1991 ) . More recently , RNNs have achieved impressive results in large-scale tasks such as language modeling for speech recognition and machine translation , and are by now standard tools for sequential natural language tasks ( e.g. , Mikolov et al . , 2010 ; Graves , 2012 ; Wu et al . , 2016 ) . This suggests that RNNs may learn to track grammatical structure even when trained on noisier natural data . The conjecture is supported by the success of RNNs as feature extractors for syntactic parsing ( e.g. , Cross and Huang , 2016 ; Kiperwasser and Goldberg , 2016 ; Zhang et al . , 2017 ) . Linzen et al . ( 2016 ) directly evaluated the extent to which RNNs can approximate hierarchical structure in corpus-extracted natural language data . They tested whether RNNs can learn to predict English subject-verb agreement , a task thought to require hierarchical structure in the general case ( " the girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl the boys like . Their experiments confirmed that RNNs can , in principle , handle such constructions . However , in their study RNNs could only succeed when provided with explicit supervision on the target task . Linzen and colleagues argued that the unsupervised language modeling objective is not sufficient for RNNs to induce the syntactic knowledge necessary to cope with long-distance agreement . . . is is is is is is is is is is is is is is is is is The current paper reevaluates these conclusions . We strengthen the evaluation paradigm of Linzen and colleagues in several ways . Most importantly , their analysis did not rule out the possibility that RNNs might be relying on semantic or collocational / frequency-based information , rather than purely on syntactic structure . In " dogs dogs dogs dogs dogs about what typically barks ( dogs , not neighbourhoods ) , without relying on more abstract structural cues . In a follow-up study to Linzen and colleagues ' , Bernardy and Lappin ( 2017 ) observed that RNNs are better at long-distance agreement when they construct rich lexical representations of words , which suggests effects of this sort might indeed be at play . We introduce a method to probe the syntactic abilities of RNNs that abstracts away from potential lexical , semantic and frequency-based confounds . Inspired by Chomsky 's ( 1957 ) We extend the previous work in three additional ways . First , alongside English , which has few morphological cues to agreement , we examine Italian , Hebrew and Russian , which have richer morphological systems . Second , we go beyond subject-verb agreement and develop an automated method to harvest a variety of long-distance number agreement constructions from treebanks . Finally , for Italian , we collect human judgments for the tested sentences , providing an important comparison point for RNN performance . 1We focus on the more interesting unsupervised setup , where RNNs are trained to perform generic , large-scale language modeling ( LM ): they are not given explicit evidence , at training time , that they must focus on long-distance agreement , but they are rather required to track a multitude of cues that might help with word prediction in general . Our results are encouraging . RNNs trained with a LM objective solve the long-distance agreement problem well , even on nonce sentences . The pattern is consistent across languages , and , crucially , not far from human performance in Italian . Moreover , RNN performance on language modeling ( measured in terms of perplexity ) is a good predictor of long-distance agreement accuracy . This suggests that the ability to capture structural generalizations is an important aspect of what makes the best RNN architectures so good at language modeling . Since our positive results contradict , to some extent , those of Linzen et al . ( 2016 ) , we also replicate their relevant experiment using our best RNN ( an LSTM ) . We outperform their models , suggesting that a careful architecture / hyperparameter search is crucial to obtain RNNs that are not only good at language modeling , but able to extract syntactic generalizations . 2 Constructing a long-distance agreement benchmark Overview . We construct our number agreement test sets as follows . Original sentences are automatically extracted from a dependency treebank . They are then converted into nonce sentences by substituting all content words with random words with the same morphology , resulting in grammatical but nonsensical sequences . An LM is evaluated on its predictions for the target ( second ) word in the dependency , in both the original and nonce sentences . Long-distance agreement constructions . Agreement relations , such as subject-verb agreement in English , are an ideal test bed for the syntactic abilities of LMs , because the form of the second item ( the target ) is predictable from the first item ( the cue ) . Crucially , the cue and the target are linked by a structural relation , where linear order in the word sequence does not matter ( Everaert et al . , 2015 ) In all these cases , the number of the main verb " thinks " is determined by its subject ( " girl " ) , and this relation depends on the syntactic structure of the sentence , not on the linear sequence of words . As the last sentence shows , the word directly preceding the verb can even be a noun with the opposite number ( " friends " ) , but this does not influence the structurally-determined form of the verb . When the cue and the target are adjacent ( " the girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks . . . " ) , an LM can predict the target without access to syntactic structure : it can simply extract the relevant morphosyntactic features of words ( e.g. , number ) and record the co-occurrence frequencies of patterns such as N P lur V P lur ( Mikolov et al . , 2013 ) . Thus , we focus here on long-distance agreement , where an arbitrary num- ber of words can occur between the elements of the agreement relation . We limit ourselves to number agreement ( plural or singular ) , as it is the only overt agreement feature shared by all of the languages we study . Identifying candidate constructions . We started by collecting pairs of part-of-speech ( POS ) tags connected by a dependency arc . Independently of which element is the head of the relation , we refer to the first item as the cue and to the second as the target . We additionally refer to the POS sequence characterizing the entire pattern as a construction , and to the elements in the middle as context . For each candidate construction , we collected all of the contexts in the corpus that intervene between the cue and the target ( we define contexts as the sequence of POS tags of the top-level nodes in the dependency subtrees ) . For example , for the English subject-verb agreement construction shown in Fig . 1a , the context is defined by VERB ( head of the relative clause ) and ADV ( adverbial modifier of the target verb ) , which together dominate the sequence " the boys like often " . For the Russian adjective-noun agreement construction in Fig . 1b , the context is NOUN , because in the dependency grammar we use the noun " moment " is the head of the prepositional phrase " at that moment " , which modifies the adjective " deep " . The candidate agreement pair and the context form a construction , which is characterized by a sequence of POS tags , e.g. , NOUN VERB ADV VERB or VERB NOUN CCONJ VERB ( Fig . 1c ) . is is is is is is is is is is is is is is is is is " and " girls girls girls girls girls girls girls girls girls girls girls girls girls girls girls girls girls who stayed at home were were were were were were were were were were were were were were were were were " . Conversely , standard syntactic structures might be split between different constructions , e.g. , relative clause contexts occur in both NOUN VERB VERB and NOUN VERB ADV VERB constructions ( the latter is illustrated by the English example in Fig . 1a ) . Construction contexts can contain a variable numbers of words . Since we are interested in challenging cases , we only considered cases in which at least three tokens intervened between the cue and the target . Excluding non-agreement constructions . In the next step , we excluded constructions in which the candidate cue and target did not agree in number in all of the instances of the construction in the treebank ( if both the cue and the target were morphologically annotated for number ) . This step retained English subject-verb constructions , for example , but excluded verb-object constructions , since any form of a verb can appear both with singular and plural objects . To focus on robust agreement patterns , we only kept constructions with at least 10 instances of both plural and singular agreement . When applied to the treebanks we used ( see Section 3 ) , this step resulted in between two ( English ) and 21 ( Russian ) constructions per lan-guage . English has the poorest morphology and consequently the lowest number of patterns with identifiable morphological agreement . Only the VP-conjunction construction ( Fig . 1c ) was identified in all four languages . Subject-verb agreement constructions were extracted in all languages but Russian ; Russian has relatively flexible word order and a noun dependent preceding a head verb is not necessarily its subject . The full list of extracted constructions in English and Italian is given in Tables 2 and 3 , respectively . For the other languages , see the Supplementary Material ( SM ) . 2 Original sentence test set . Our " original " sentence test set included all sentences from each construction where all words from the cue and up to and including the target occurred in the LM vocabulary ( Section 3 ) , and where the singular / plural counterpart of the target occurred in the treebank and in the language model vocabulary ( this is required by the evaluation procedure outlined below ) . The total counts of constructions and original sentences in our test sets are provided in Table 1 . The average number of context words separating the cue and the target ranged from 3.6 ( Hebrew ) to 4.5 ( Italian ) . Generating nonce sentences . We generated nine nonce variants of each original sentence as follows . Each content word ( noun , verb , adjective , proper noun , numeral , adverb ) in the sentence was substituted by another random content word from the treebank with matching POS and morphological features . To avoid forms that are ambiguous between several POS , which are particularly frequent in English ( e.g. , plural noun and singular verb forms ) , we excluded the forms that appeared with a different POS more than 10 % of the time in the treebank . Function words ( determiners , pronouns , adpositions , particles ) and punctuation were left intact . For example , we generated the nonce ( 1b ) from the original sentence ( 1a ): ( ( e.g. , " it stays the shuttle " in ( 1b ) ) . Evaluation procedure . For each sentence in our test set , we retrieved from our treebank the form that is identical to the agreement target in all morphological features except number ( e.g. , " finds " instead of " find " in ( 1b ) ) . Given a sentence with prefix p up to and excluding the target , we then compute the probabilities P ( t 1 |p ) and P ( t 2 |p ) for the singular and plural variants of the target , t 1 and t 2 , based on the language model . Following Linzen et al . ( 2016 ) , we say that the model identified the correct target if it assigned a higher probability to the form with the correct number . In ( 1b ) , for example , the model should assign a higher probability to " finds " than " find".3 3 Experimental setup Treebanks . We extracted our test sets from the Italian , English , Hebrew and Russian Universal Dependency treebanks ( UD , v2.0 , Nivre et al . , 2016 ) . The English and Hebrew treebanks were post-processed to obtain a richer morphological annotation at the word level ( see SM for details ) . We ran an extensive analysis of the abilities of RNNs trained on a generic language-modeling task to predict long-distance number agreement . Results were consistent across four languages and a number of constructions . They were above strong baselines even in the challenging case of nonsense sentences , and not far from human performance . We are not aware of other collections of human long-distance agreement judgments on nonsensical sentences , and we thus consider our publicly available data set an important contribution of our work , of interest to students of human language processing in general . The constructions we considered are quite infrequent ( according to a rough estimate based on the treebanks , the language in which they are most common is Hebrew , and even there they occur with average 0.8 % sentence frequency ) . Moreover , they vary in the contexts that separate the cue and the target . So , RNNs are not simply memorizing frequent morphosyntactic sequences ( which would already be impressive , for systems learning from raw text ) . We tentatively conclude that LMtrained RNNs can construct abstract grammatical representations of their input . This , in turn , suggests that the input itself contains enough information to trigger some form of syntactic learning in a system , such as an RNN , that does not contain an explicit prior bias in favour of syntactic structures . In future work , we would like to better understand what kind of syntactic information RNNs are encoding , and how . On the one hand , we plan to adapt methods to inspect information flow across RNN states ( e.g. , Hupkes et al . , 2017 ) . On the other , we would like to expand our empirical investigation by focusing on other long-distance phenomena , such as overt case assignment ( Blake , 2001 ) or parasitic gap licensing ( Culicover and Postal , 2001 ) . While it is more challenging to extract reliable examples of such phenomena from corpora , their study would probe more sophisticated syntactic capabilities , possibly even shedding light on the theoretical analysis of the underlying linguistic structures . Finally , it may be useful to complement the corpus-driven approach used in the current paper with constructed evaluation sentences that isolate particular syntactic phenomena , independent of their frequency in a natural corpus , as is common in psycholinguistics ( Enguehard et al . , 2017 ) .

Previous work only shows that RNNs can handle constructions that require hierarchical structure when explicit supervision on the target task is given.

They introduce a probing method for syntactic abilities to evaluate long-distance agreement on standard and nonsensical sentences in multiple languages with different morphological systems.

The RNNs trained on an LM objective can solve long-distance agreement problems well even on nonsensical sentences consistently across languages indicating their deeper grammatical competence.


Current system combination methods usually use confusion networks to find consensus translations among different systems . Requiring one-to-one mappings between the words in candidate translations , confusion networks have difficulty in handling more general situations in which several words are connected to another several words . Instead , we propose a lattice-based system combination model that allows for such phrase alignments and uses lattices to encode all candidate translations . Experiments show that our approach achieves significant improvements over the state-ofthe-art baseline system on Chinese-to-English translation test sets . System combination aims to find consensus translations among different machine translation systems . It has been proven that such consensus translations are usually better than the output of individual systems ( Frederking and Nirenburg , 1994 ) . In recent several years , the system combination methods based on confusion networks developed rapidly ( Bangalore et al . , 2001 ; Matusov et al . , 2006 ; Sim et al . , 2007 ; Rosti et al . , 2007a ; Rosti et al . , 2007b ; Rosti et al . , 2008 ; He et al . , 2008 ) , which show state-of-the-art performance in benchmarks . A confusion network consists of a sequence of sets of candidate words . Each candidate word is associated with a score . The optimal consensus translation can be obtained by selecting one word from each set to maximizing the overall score . To construct a confusion network , one first need to choose one of the hypotheses ( i.e. , candidate translations ) as the backbone ( also called " skeleton " in the literature ) and then decide the word alignments of other hypotheses to the backbone . Hypothesis alignment plays a crucial role in confusionnetwork-based system combination because it has a direct effect on selecting consensus translations . However , a confusion network is restricted in such a way that only 1-to-1 mappings are allowed in hypothesis alignment . This is not the fact even for word alignments between the same languages . It is more common that several words are connected to another several words . For example , " be capable of " and " be able to " have the same meaning . Although confusion-network-based approaches resort to inserting null words to alleviate this problem , they face the risk of producing degenerate translations such as " be capable to " and " be able of " . In this paper , we propose a new system combination method based on lattices . As a more general form of confusion network , a lattice is capable of describing arbitrary mappings in hypothesis alignment . In a lattice , each edge is associated with a sequence of words rather than a single word . Therefore , we select phrases instead of words in each candidate set and minimize the chance to produce unexpected translations such as " be capable to " . We compared our approach with the state-of-the-art confusion-network-based system ( He et al . , 2008 ) and achieved a significant absolute improvement of 1.23 BLEU points on the NIST 2005 Chinese-to-English test set and 0.93 BLEU point on the NIST 2008 Chinese-to-English test set . The alignment model plays an important role in system combination . Because of the expression limitation of confusion networks , only 1-to-1 mappings are employed in the confusion-network-based model . This paper proposes a lattice-based system combination model . As a general form of confusion networks , lattices can express n-to-n mappings . a confusion-network-based model processes words only . As a result , phrase pairs must be extracted before constructing a lattice . On NIST MT05 test set , the lattice-based system gave better results with an absolute improvement of 1.23 BLEU points over the confusionnetwork-based system ( He et al . , 2008 ) and 3.73 BLEU points over the best single system . On NIST MT08 test set , the lattice-based system outperformed the confusion-network-based system by 0.93 BLEU point and outperformed the best single system by 3.0 BLEU points .

System combination methods based on confusion networks only allow word level 1-to-1 mappings, and some workarounds cause another type of problem such as degeneration.

They propose to use lattices to combine systems that enable to process of a sequence of words rather than one word that can mitigate degeneration.

They show that their approach significantly outperforms the state-of-the-art confusion-network-based systems on Chinese-to-English translation tasks.


The growing demand for structured knowledge has led to great interest in relation extraction , especially in cases with limited supervision . However , existing distance supervision approaches only extract relations expressed in single sentences . In general , cross-sentence relation extraction is under-explored , even in the supervised-learning setting . In this paper , we propose the first approach for applying distant supervision to crosssentence relation extraction . At the core of our approach is a graph representation that can incorporate both standard dependencies and discourse relations , thus providing a unifying way to model relations within and across sentences . We extract features from multiple paths in this graph , increasing accuracy and robustness when confronted with linguistic variation and analysis error . Experiments on an important extraction task for precision medicine show that our approach can learn an accurate cross-sentence extractor , using only a small existing knowledge base and unlabeled text from biomedical research articles . Compared to the existing distant supervision paradigm , our approach extracted twice as many relations at similar precision , thus demonstrating the prevalence of cross-sentence relations and the promise of our approach . The accelerating pace in technological advance and scientific discovery has led to an explosive growth in knowledge . The ensuing information overload creates new urgency in assimilating frag-mented knowledge for integration and reasoning . A salient case in point is precision medicine ( Bahcall , 2015 ) . The cost of sequencing a person 's genome has fallen below $ 10001 , enabling individualized diagnosis and treatment of complex genetic diseases such as cancer . The availability of measurement for 20,000 human genes makes it imperative to integrate all knowledge about them , which grows rapidly and is scattered in millions of articles in PubMed2 . Traditional extraction approaches require annotated examples , which makes it difficult to scale to the explosion of extraction demands . Consequently , there has been increasing interest in indirect supervision ( Banko et al . , 2007 ; Poon and Domingos , 2009 ; Toutanova et al . , 2015 ) , with distant supervision ( Craven et al . , 1998 ; Mintz et al . , 2009 ) emerging as a particularly promising paradigm for augmenting existing knowledge bases from unlabeled text ( Poon et al . , 2015 ; Parikh et al . , 2015 ) . This progress is exciting , but distantsupervision approaches have so far been limited to single sentences , thus missing out on relations crossing the sentence boundary . Consider the following example:"The p56Lck inhibitor Dasatinib was shown to enhance apoptosis induction by dexamethasone in otherwise GC-resistant CLL cells . This finding concurs with the observation by Sade showing that Notch-mediated resistance of a mouse lymphoma cell line could be overcome by inhibiting p56Lck . " Together , the two sentences convey the fact that the drug Dasatinib could overcome resistance conferred by mutations to the Notch gene , which can not be inferred from either sentence alone . The impact of missed opportunities is especially pronounced in the long tail of knowledge . Such information is crucial for integrative reasoning as it includes the newest findings in specialized domains . In this paper , we present DISCREX , the first approach for distant supervision to relation extraction beyond the sentence boundary . The key idea is to adopt a document-level graph representation that augments conventional intra-sentential dependencies with new dependencies introduced for adjacent sentences and discourse relations . It provides a unifying way to derive features for classifying relations between entity pairs . As we augment this graph with new arcs , the number of possible paths between entities grow . We demonstrate that feature extraction along multiple paths leads to more robust extraction , allowing the learner to find structural patterns even when the language varies or the parser makes an error . The cross-sentence scenario presents a new challenge in candidate selection . This motivates our concept of minimal-span candidates in Section 3.2 . Excluding non-minimal candidates substantially improves classification accuracy . There is a long line of research on discourse phenomena , including coreference ( Haghighi and Klein , 2007 ; Poon and Domingos , 2008 ; Rahman and Ng , 2009 ; Raghunathan et al . , 2010 ) , narrative structures ( Chambers and Jurafsky , 2009 ; Cheung et al . , 2013 ) , and rhetorical relations ( Marcu , 2000 ) . For the most part , this work has not been connected to relation extraction . Our proposed extraction framework makes it easy to integrate such discourse relations . Our experiments evaluated the impact of coreference and discourse parsing , a preliminary step toward in-depth integration with discourse research . We conducted experiments on extracting druggene interactions from biomedical literature , an important task for precision medicine . By bootstrapping from a recently curated knowledge base ( KB ) with about 162 known interactions , our DIS-CREX system learned to extract inter-sentence drug-gene interactions at high precision . Crosssentence extraction doubled the yield compared to single-sentence extraction . Overall , by applying distant supervision , we extracted about 64,000 distinct interactions from about one million PubMed Central full-text articles , attaining two orders of magnitude increase compared to the original KB . We present the first approach for applying distant supervision to cross-sentence relation extraction , by adopting a document-level graph representation that incorporates both intra-sentential dependencies and inter-sentential relations such as adjacency and discourse relations . We conducted both automatic and manual evaluation on extracting drug-gene interactions from biomedical literature . With cross-sentence extraction , our DIS-CREX system doubled the yield of unique interactions , while maintaining the same accuracy . Using distant supervision , DISCREX improved the coverage of the Gene Drug Knowledge Database ( GDKD ) by two orders of magnitude , without requiring annotated examples . Future work includes : further exploration of features ; improved integration with coreference and discourse parsing ; combining distant supervision with active learning and crowd sourcing ; evaluate the impact of extractions to precision medicine ; applications to other domains .

Existing distance supervision methods for relation extraction cannot capture relations crossing the sentence boundary which is important in specialized domains with long-tail knowledge.

They propose a method for applying distance supervision to cross-sentence relation extraction by adopting a document-level graph representation that incorporates intra-sentential dependencies and inter-sentential relations.

Experiments on extracting drug-gene interactions from biomedical literature show that the proposed method doubles the performance of single-sentence extraction methods.


Convolution tree kernel has shown promising results in semantic role classification . However , it only carries out hard matching , which may lead to over-fitting and less accurate similarity measure . To remove the constraint , this paper proposes a grammardriven convolution tree kernel for semantic role classification by introducing more linguistic knowledge into the standard tree kernel . The proposed grammar-driven tree kernel displays two advantages over the previous one : 1 ) grammar-driven approximate substructure matching and 2 ) grammardriven approximate tree node matching . The two improvements enable the grammardriven tree kernel explore more linguistically motivated structure features than the previous one . Experiments on the CoNLL-2005 SRL shared task show that the grammardriven tree kernel significantly outperforms the previous non-grammar-driven one in SRL . Moreover , we present a composite kernel to integrate feature-based and tree kernel-based methods . Experimental results show that the composite kernel outperforms the previously best-reported methods . Given a sentence , the task of Semantic Role Labeling ( SRL ) consists of analyzing the logical forms expressed by some target verbs or nouns and some constituents of the sentence . In particular , for each predicate ( target verb or noun ) all the constituents in the sentence which fill semantic arguments ( roles ) of the predicate have to be recognized . Typical semantic roles include Agent , Patient , Instrument , etc . and also adjuncts such as Locative , Temporal , Manner , and Cause , etc . Generally , semantic role identification and classification are regarded as two key steps in semantic role labeling . Semantic role identification involves classifying each syntactic element in a sentence into either a semantic argument or a non-argument while semantic role classification involves classifying each semantic argument identified into a specific semantic role . This paper focuses on semantic role classification task with the assumption that the semantic arguments have been identified correctly . Both feature-based and kernel-based learning methods have been studied for semantic role classification ( Carreras and Màrquez , 2004 ; Carreras and Màrquez , 2005 ) . In feature-based methods , a flat feature vector is used to represent a predicateargument structure while , in kernel-based methods , a kernel function is used to measure directly the similarity between two predicate-argument structures . As we know , kernel methods are more effective in capturing structured features . Moschitti ( 2004 ) and Che et al . ( 2006 ) used a convolution tree kernel ( Collins and Duffy , 2001 ) for semantic role classification . The convolution tree kernel takes sub-tree as its feature and counts the number of common sub-trees as the similarity between two predicate-arguments . This kernel has shown very promising results in SRL . However , as a general learning algorithm , the tree kernel only carries out hard matching between any two sub-trees without considering any linguistic knowledge in kernel design . This makes the kernel fail to handle similar phrase structures ( e.g. , " buy a car " vs. " buy a red car " ) and near-synonymic grammar tags ( e.g. , the POS variations between " high / JJ degree / NN " 1 and " higher / JJR degree / NN " ) 2 . To some degree , it may lead to over-fitting and compromise performance . This paper reports our preliminary study in addressing the above issue by introducing more linguistic knowledge into the convolution tree kernel . To our knowledge , this is the first attempt in this research direction . In detail , we propose a grammar-driven convolution tree kernel for semantic role classification that can carry out more linguistically motivated substructure matching . Experimental results show that the proposed method significantly outperforms the standard convolution tree kernel on the data set of the CoNLL-2005 SRL shared task . The remainder of the paper is organized as follows : Section 2 reviews the previous work and Section 3 discusses our grammar-driven convolution tree kernel . Section 4 shows the experimental results . We conclude our work in Section 5 . In this paper , we propose a novel grammar-driven convolution tree kernel for semantic role classification . More linguistic knowledge is considered in the new kernel design . The experimental results verify that the grammar-driven kernel is more effective in capturing syntactic structure features than the previous convolution tree kernel because it allows grammar-driven approximate matching of substructures and node features . We also discuss the criteria to determine the optional nodes in a CFG rule in defining our grammar-driven convolution tree kernel . The extension of our work is to improve the performance of the entire semantic role labeling system using the grammar-driven tree kernel , including all four stages : pruning , semantic role identification , classification and post inference . In addition , a more interesting research topic is to study how to integrate linguistic knowledge and tree kernel methods to do feature selection for tree kernelbased NLP applications ( Suzuki et al . , 2004 ) . In detail , a linguistics and statistics-based theory that can suggest the effectiveness of different substructure features and whether they should be generated or not by the tree kernels would be worked out .

Despite its success in semantic role classification, convolution tree kernels based on the hard matching between two sub-trees suffer from over-fitting.

They propose to integrate a linguistically motivated grammar-baed convolution tree kernel into a standard tree kernel to achieve better substructure matching and tree node matching.

The new grammar-driven tree kernel significantly outperforms baseline kernels on the CoNLL-2005 task.


We explore the problem of resolving the second person English pronoun you in multi-party dialogue , using a combination of linguistic and visual features . First , we distinguish generic and referential uses , then we classify the referential uses as either plural or singular , and finally , for the latter cases , we identify the addressee . In our first set of experiments , the linguistic and visual features are derived from manual transcriptions and annotations , but in the second set , they are generated through entirely automatic means . Results show that a multimodal system is often preferable to a unimodal one . The English pronoun you is the second most frequent word in unrestricted conversation ( after I and right before it ) . 1 Despite this , with the exception of Gupta et al . ( 2007b ; 2007a ) , its resolution has received very little attention in the literature . This is perhaps not surprising since the vast amount of work on anaphora and reference resolution has focused on text or discourse -mediums where second-person deixis is perhaps not as prominent as it is in dialogue . For spoken dialogue pronoun resolution modules however , resolving you is an essential task that has an important impact on the capabilities of dialogue summarization systems . Besides being important for computational implementations , resolving you is also an interesting and challenging research problem . As for third person pronouns such as it , some uses of you are not strictly referential . These include discourse marker uses such as you know in example ( 1 ) , and generic uses like ( 2 ) , where you does not refer to the addressee as it does in ( 3 ) . ( 1 ) It 's not just , you know , noises like something hitting . ( 2 ) Often , you need to know specific button sequences to get certain functionalities done . ( 3 ) I think it 's good . You 've done a good review . However , unlike it , you is ambiguous between singular and plural interpretations -an issue that is particularly problematic in multi-party conversations . While you clearly has a plural referent in ( 4 ) , in ( 3 ) the number of its referent is ambiguous . 2(4 ) I do n't know if you guys have any questions . When an utterance contains a singular referential you , resolving the you amounts to identifying the individual to whom the utterance is addressed . This is trivial in two-person dialogue since the current listener is always the addressee , but in conversations with multiple participants , it is a complex problem where different kinds of linguistic and visual information play important roles ( Jovanovic , 2007 ) . One of the issues we investigate here is how this applies to the more concrete problem of resolving the second person pronoun you . We approach this issue as a three-step problem . Using the AMI Meeting Corpus ( McCowan et al . , 2005 ) of multi-party dialogues , we first discriminate between referential and generic uses of you . Then , within the referential uses , we distinguish between singular and plural , and finally , we resolve the singular referential instances by identifying the intended addressee . We use multimodal features : initially , we extract discourse features from manual transcriptions and use visual information derived from manual annotations , but then we move to a fully automatic approach , using 1-best transcriptions produced by an automatic speech recognizer ( ASR ) and visual features automatically extracted from raw video . In the next section of this paper , we give a brief overview of related work . We describe our data in Section 3 , and explain how we extract visual and linguistic features in Sections 4 and 5 respectively . Section 6 then presents our experiments with manual transcriptions and annotations , while Section 7 , those with automatically extracted information . We end with conclusions in Section 8 . We have investigated the automatic resolution of the second person English pronoun you in multi-party dialogue , using a combination of linguistic and visual features . We conducted a first set of experiments where our features were derived from manual transcriptions and annotations , and then a second set where they were generated by entirely automatic means . To our knowledge , this is the first attempt at tackling this problem using automatically extracted multimodal information . Our experiments showed that visual information can be highly predictive in resolving the addressee of singular referential uses of you . Visual features significantly improved the performance of both our manual and automatic systems , and the latter achieved an encouraging 75 % accuracy . We also found that our visual features had predictive power for distinguishing between generic and referential uses of you , and between referential singulars and plurals . Indeed , for the latter task , they significantly improved the manual system 's performance . The listeners ' gaze features were useful here : in our data set it was apparently the case that the speaker would often use the whiteboard / projector screen when addressing the group , thus drawing the listeners ' gaze in this direction . Future work will involve expanding our dataset , and investigating new potentially predictive features . In the slightly longer term , we plan to integrate the resulting system into a meeting assistant whose purpose is to automatically extract useful information from multi-party meetings .

Although the word "you" is frequently used and has several possible meanings, such as reference or generic, it is not well studied yet.

They first manually automatically separate the word "you" between generic and referential, then later use a multimodal system for automation.

They show that visual features can help distinguish the word "you" in multi-party conversations.


There is a growing research interest in opinion retrieval as on-line users ' opinions are becoming more and more popular in business , social networks , etc . Practically speaking , the goal of opinion retrieval is to retrieve documents , which entail opinions or comments , relevant to a target subject specified by the user 's query . A fundamental challenge in opinion retrieval is information representation . Existing research focuses on document-based approaches and documents are represented by bag-of-word . However , due to loss of contextual information , this representation fails to capture the associative information between an opinion and its corresponding target . It can not distinguish different degrees of a sentiment word when associated with different targets . This in turn seriously affects opinion retrieval performance . In this paper , we propose a sentence-based approach based on a new information representation , namely topic-sentiment word pair , to capture intra-sentence contextual information between an opinion and its target . Additionally , we consider inter-sentence information to capture the relationships among the opinions on the same topic . Finally , the two types of information are combined in a unified graph-based model , which can effectively rank the documents . Compared with existing approaches , experimental results on the COAE08 dataset showed that our graph-based model achieved significant improvement . In recent years , there is a growing interest in sharing personal opinions on the Web , such as product reviews , economic analysis , political polls , etc . These opinions can not only help independent users make decisions , but also obtain valuable feedbacks ( Pang et al . , 2008 ) . Opinion oriented research , including sentiment classifica-tion , opinion extraction , opinion question answering , and opinion summarization , etc . are receiving growing attention ( Wilson , et al . , 2005 ; Liu et al . , 2005 ; Oard et al . , 2006 ) . However , most existing works concentrate on analyzing opinions expressed in the documents , and none on how to represent the information needs required to retrieve opinionated documents . In this paper , we focus on opinion retrieval , whose goal is to find a set of documents containing not only the query keyword(s ) but also the relevant opinions . This requirement brings about the challenge on how to represent information needs for effective opinion retrieval . In order to solve the above problem , previous work adopts a 2-stage approach . In the first stage , relevant documents are determined and ranked by a score , i.e. tf-idf value . In the second stage , an opinion score is generated for each relevant document ( Macdonald and Ounis , 2007 ; Oard et al . , 2006 ) . The opinion score can be acquired by either machine learning-based sentiment classifiers , such as SVM ( Zhang and Yu , 2007 ) , or a sentiment lexicons with weighted scores from training documents ( Amati et al . , 2007 ; Hannah et al . , 2007 ; Na et al . , 2009 ) . Finally , an overall score combining the two is computed by using a score function , e.g. linear combination , to re-rank the retrieved documents . Retrieval in the 2-stage approach is based on document and document is represented by bag-of-word . This representation , however , can only ensure that there is at least one opinion in each relevant document , but it can not determine the relevance pairing of individual opinion to its target . In general , by simply representing a document in bag-of-word , contextual information i.e. the corresponding target of an opinion , is neglected . This may result in possible mismatch between an opinion and a target and in turn affects opinion retrieval performance . By the same token , the effect to documents consisting of mul-tiple topics , which is common in blogs and on-line reviews , is also significant . In this setting , even if a document is regarded opinionated , it can not ensure that all opinions in the document are indeed relevant to the target concerned . Therefore , we argue that existing information representation i.e. bag-of-word , can not satisfy the information needs for opinion retrieval . In this paper , we propose to handle opinion retrieval in the granularity of sentence . It is observed that a complete opinion is always expressed in one sentence , and the relevant target of the opinion is mostly the one found in it . Therefore , it is crucial to maintain the associative information between an opinion and its target within a sentence . We define the notion of a topic-sentiment word pair , which is composed of a topic term ( i.e. the target ) and a sentiment word ( i.e. opinion ) of a sentence . Word pairs can maintain intra-sentence contextual information to express the potential relevant opinions . In addition , inter-sentence contextual information is also captured by word pairs to represent the relationship among opinions on the same topic . In practice , the inter-sentence information reflects the degree of a word pair . Finally , we combine both intra-sentence and inter-sentence contextual information to construct a unified undirected graph to achieve effective opinion retrieval . The rest of the paper is organized as follows . In Section 2 , we describe the motivation of our approach . Section 3 presents a novel unified graph-based model for opinion retrieval . We evaluated our model and the results are presented in Section 4 . We review related works on opinion retrieval in Section 5 . Finally , in Section 6 , the paper is concluded and future work is suggested . In this work we focus on the problem of opinion retrieval . Different from existing approaches , which regard document relevance as the key indicator of opinion relevance , we propose to explore the relevance of individual opinion . To do that , opinion retrieval is performed in the granularity of sentence . We define the notion of word pair , which can not only maintain the association between the opinion and the corresponding target in the sentence , but it can also build up the relationship among sentences through the same word pair . Furthermore , we convert the relationships between word pairs and sentences into a unified graph , and use the HITS algorithm to achieve document ranking for opinion retrieval . Finally , we compare our approach with existing methods . Experimental results show that our proposed model performs well on COAE08 dataset . The novelty of our work lies in using word pairs to represent the information needs for opinion retrieval . On the one hand , word pairs can identify the relevant opinion according to intra-sentence contextual information . On the other hand , word pairs can measure the degree of a relevant opinion by taking inter-sentence contextual information into consideration . With the help of word pairs , the information needs for opinion retrieval can be represented appropriately . In the future , more research is required in the following directions : ( 1 ) Since word pairs can indicate relevant opinions effectively , it is worth further study on how they could be applied to other opinion oriented applications , e.g. opinion summarization , opinion prediction , etc . ( 2 ) The characteristics of blogs will be taken into consideration , i.e. , the post time , which could be helpful to create a more time sensitivity graph to filter out fake opinions . ( 3 ) Opinion holder is another important role of an opinion , and the identification of opinion holder is a main task in NTCIR . It would be interesting to study opinion holders , e.g. its seniority , for opinion retrieval .

Existing approaches to the opinion retrieval task represent documents using bag-of-words disregarding contextual information between an opinion and its corresponding text.

They propose a sentence-based approach which captures both inter and intra sentence contextual information combined with a unified undirected graph.

The proposed method outperforms existing approaches on the COAE08 dataset showing that word pairs can represent information for opinion retrieval well.