{
  "name": "323.pdf",
  "metadata": {
    "source": "docling-granite",
    "title": "A Neural Local Coherence Model",
    "authors": [],
    "emails": [],
    "sections": [
      {
        "heading": "Anonymous ACL submission",
        "text": ""
      },
      {
        "heading": "1 Introduction and Motivation",
        "text": "What distinguishes a coherent text from a random sequence of sentences is that it binds the sentences together to express a meaning as a whole - the interpretation of a sentence usually depends on the meaning of its neighbors. Coherence models that can distinguish a coherent from incoherent texts have a wide range of applications in text generation, summarization, and coherence scoring.\n\nSeveral formal theories of coherence have been proposed (Mann and Thompson, 1988a; Grosz et al., 1995; Asher and Lascarides, 2003), and their principles have inspired development of existing coherence models (Barzilay and Lapata, 2008; Lin et al., 2011; Li and Hovy, 2014). Among these models, the entity grid (Barzilay and Lapata, 2008), which is based on Centering Theory (Grosz et al., 1995), is arguably the most popular, and has seen a number of improvements over the years. As shown in Figure 2, the entity grid model represents a text by a grid that captures how grammatical roles of different entities change from sentence to sentence. The grid is then converted into a feature vector containing probabilities of local entity transitions, which enables machine learning models to learn the degree of text coherence. Extensions of this basic grid model incorporate entity-specific features (Elsner and Charniak, 2011), multiple ranks (Feng and Hirst, 2012), and coherence relations (Feng et al., 2014).\n\nWhile the entity grid and its extensions have been successful in many applications, they are limited in several ways. Firstly, they use discrete representation for grammatical roles and features, which limits the model to consider sufficiently long transitions (Bengio et al., 2003). Secondly, feature vector computation in existing models is decoupled from the target task, which limits the models to learn task-specific features.\n\nIn this paper, we propose a neural architecture for coherence assessment that can capture long range entity transitions along with arbitrary entityspecific features. Our model obtains generalization through distributed representations of entity transitions and entity features. We also present an end-to-end training method to learn task-specific high level features automatically in our model.\n\nWe evaluate our approach on three different evaluation tasks: discrimination, insertion, and summary coherence rating, proposed previously for evaluating coherence models (Barzilay and Lapata, 2008; Elsner and Charniak, 2011). Discrimination and insertion involve identifying the right order of the sentences in a text with different levels of difficulty. In summary coherence rating task, we compare the rankings, given by the model, against human pairwise judgments of coherence.\n\nThe experimental results show that our neural models consistently improve over the nonneural counterparts (i.e., existing entity grid models) yielding absolute gains of about 4% on discrimination, up to 2 . 5% on insertion, and more than 4% on summary coherence rating. Our model achieves state of the art results in all these tasks. We have released our code with the submission.\n\nThe remainder of this paper is organized as follows. We describe entity grid, its extensions, and its limitations in Section 2. In Section 3, we present our neural model. We describe evaluation tasks and results in Sections 4 and 5. We give a brief account of related work in Section 6. Finally, we conclude with future directions in Section 7."
      },
      {
        "heading": "2 Entity Grid and Its Extensions",
        "text": "Motivated by Centering Theory (Grosz et al., 1995), Barzilay and Lapata (2008) proposed an entity-based model for representing and assessing text coherence. Their model represents a text by a two-dimensional array called entity grid that captures transitions of discourse entities across sentences. As shown in Figure 2, the rows of the grid correspond to sentences, and the columns correspond to discourse entities appearing in the text. They consider noun phrases (NP) as entities, and employ a coreference resolver to detect mentions of the same entity (e.g., Obama , the president ). Each entry G i,j in the entity grid represents the syntactic role that entity e j plays in sentence s i , which can be one of: subject ( S ), object ( O ), or other ( X ). In addition, entities not appearing in a sentence are marked by a special symbol ( -). If an entity appears more than once with different grammatical roles in the same sentence, the role with the highest rank (S ≻ O ≻ X) is considered.\n\nTo represent an entity grid with a feature vector, Barzilay and Lapata (2008) compute probability for each local entity transition of length k (i.e., { S, O, X, -} k ), and represent each grid by a vector of 4 k transitions probabilities. To distinguish between transitions of important entities from unimportant ones, they consider the salience of the entities, which they quantify by their occurrence frequency in the document. Assessment of text coherence is then formulated as a ranking problem in an SVM preference ranking framework (Joachims, 2002).\n\nSubsequent studies proposed to extend the basic entity grid model. Filippova and Strube (2007) attempted to improve the model by grouping entities based on semantic relatedness, but did not get significant improvement. Elsner and Charniak (2011) proposed a number of improvements. They initially show significant improvement by includ- s 0 : Eaton Corp. said it sold its Pacific Sierra Research unit to a company formed by employees of that unit.\n\nUNIT PRODUCTS RESEARCH COMPANY PARTS CONTROLS INDUSTRY ELECTRONICS TERM CONCERN AEROSPACE EMPLOYEES SERVICES LOS ANGELES EATON 0 O - X X - - - - - - - X - - X 1 - - - - - - - - S - - - - - - 2 - O - - - - X - - - - O O X - 3 - - - - X X - X - O X - - - S\n\ns 1 : Terms were not disclosed.\n\ns 2 : Pacific Sierra, based in Los Angeles, has 200 employees and supplies professional services and advanced products to industry.\n\ns 3 : Eaton is an automotive parts, controls and aerospace electronics concern.\n\nFigure 1: Entity grid representation (top) for a document (below) from WSJ (id: 0079).\n\ning non-head nouns (i.e., nouns that do not head NPs) as entities in the grid. 1 Then, they extend the grid to distinguish between entities of different types by incorporating entity-specific features like named entity, noun class, modifiers, etc. These extensions led to the best results reported so far.\n\nEntity grid and its extensions have been successfully applied to many downstream tasks including coherence rating (Barzilay and Lapata, 2008), essay scoring (Burstein et al., 2010), story generation (McIntyre and Lapata, 2010), and readability assessment (Pitler et al., 2010; Barzilay and Lapata, 2008). They have also been critical components in state-of-the-art sentence ordering models (Soricut and Marcu, 2006; Elsner and Charniak, 2011; Lin et al., 2011)."
      },
      {
        "heading": "2.1 Limitations of Entity Grid Models",
        "text": "Despite its success, existing entity grid models are limited in several ways.\n\n· Existing models use discrete representation for grammatical roles and features, which leads to the so-called curse of dimensionality problem (Bengio et al., 2003). In particular, to model transitions of length k with R different grammatical roles, the basic entity grid model needs to compute R k transition probabilities from a grid. One can imagine that the estimated distribution becomes sparse as k increases. This limits the model to consider longer transitions - existing models use k ≤ 3 .\n\n1 They match the nouns to detect coreferent entities.\n\nThis problem is exacerbated when we want to include entity-specific features, as the number of parameters grows exponentially with the number of features (Elsner and Charniak, 2011).\n\n· Existing models compute feature representations from entity grids in a task-agnostic way. In other words, feature extraction is decoupled from the target downstream tasks. This can limit the models to learn task-specific features. Therefore, models that can be trained in an end-to-end fashion on different target tasks are desirable.\n\nIn the following section, we present a neural architecture that allows us to capture long range entity transitions along with arbitrary entity-specific features without loosing generalization. We also present an end-to-end training method to learn task-specific features automatically."
      },
      {
        "heading": "3 The Neural Coherence Model",
        "text": "Figure 2 summarizes our neural architecture for modeling local coherence, and how it can be trained in a pairwise fashion. The architecture takes a document as input, and first extracts its entity grid. 2 The first layer of the neural network transforms each grammatical role in the grid into a distributed representation, a real-valued vector. The second layer computes high-level features by going over each column (transitions) of the grid. The following layer selects the most important high-level features, which are in turn used for coherence scoring. The features computed at different layers of the network are automatically trained by backpropagation to be relevant to the task. In the following, we elaborate on the layers of the neural network model.\n\n(I) Transforming grammatical roles into feature vectors: Grammatical roles are fed to our model as indices taken from a finite vocabulary V . In the simplest scenario, V contains { S, O, X, -} . However, we will see in Section 3.1 that as we include more entity-specific features, V can contain more symbols. The first layer of our network maps each of these indices into a distributed representation R d by looking up a shared embedding matrix E ∈ R | V |× d . We consider E a model parameter to be learned by backpropagation on a given task. We can initialize E randomly or using pretrained vectors trained on a general coherence task.\n\n2 For clarification, pairwise input as shown in the figure is required only to train the model.\n\nGiven an entity grid G with columns representing entity transitions over sentences in a document, the lookup layer extracts a d -dimensional vector for each entry G i,j from E . More formally,\n\nbeginarray c c · In & L ( G ) = langle E ( G _ 1 , 1 ) ··· E ( G _ i , j ) ··· E ( G _ m , n ) rangle \\ from & quad the endarray ( 1 )\n\nwhere E ( G i,j ) refers to the row in E that corresponds to the grammatical role G i,j ∈ V ; m is the total number of sentences and n is the total number of entities in the document. The output L ( G ) is a tensor in R m × n × d , which is fed to the next layer of the network as we describe below.\n\n(II) Modeling entity transitions: The vectors produced by the lookup layer are combined by subsequent layers of the network to generate a coherence score for the document. To compose higher-level features from the embedding vectors, we make the following modeling assumptions:\n\nSimilar to existing entity grid models, we assume there is no spatio-temporal relation between the entities in a document. In other words, columns in a grid are treated independently.\n\nWe are interested in modeling entity transitions of arbitrary lengths in a location-invariant way. This means, we aim to compose local patches of entity transitions into higher-level representations, while treating the patches independently of their position in the entity grid.\n\nUnder these assumptions, the natural choice to tackle this problem is to use a convolutional approach, used previously to solve other NLP tasks (Collobert et al., 2011; Kim, 2014).\n\nConvolution layer: A convolution operation involves applying a fi lter w ∈ R k.d (i.e., a vector of weight parameters) to each entity transition of length k to produce a new abstract feature\n\nh _ t = f ( mathbf w ^ T L _ t colon t + k - 1 , j + b _ t )\n\nwhere L t : t + k -1 ,j denotes the concatenation of k vectors in the lookup layer representing a transition of length k for entity e j in the grid, b t is a bias term, and f is a nonlinear activation function, e.g., ReLU (Nair and Hinton, 2010) in our model.\n\nWe apply this filter to each possible k -length transitions of different entities in the grid to generate a feature map , h i = [ h 1 , · · · , h m.n + k -1 ] . We repeat this process N times with N different filters to get N different feature maps (Figure 2). Notice that we use a wide convolution (Kalchbrenner et al., 2014), as opposed to narrow , to ensure that the filters reach entire columns of a grid, including the boundary entities. This is done by performing zero-padding , where out-of-range (i.e., for t < 0 or t > { m,n } ) vectors are assumed to be zero.\n\nFigure 2: Neural architecture for modeling local coherence and the pairwise training method.\n\nConvolutional filters learn to compose local transition features of a grid into higher-level representations automatically. Since it operates over the distributed representation of grid entries, compared to traditional grid models, the transition length k can be sufficiently large (e.g., 5 -8 in our experiments) to capture long-range transitional dependencies without overfitting on the training data. Moreover, unlike existing grid models that compute transition probabilities from a single document, embedding vectors and convolutional filters are learned from all training documents, which helps the neural framework to obtain better generalization and robustness.\n\nPooling layer: After the convolution, we apply a max-pooling operation to each feature map.\n\nm = [ μ _ p ( mathbf h ^ 1 ) , ··· , μ _ p ( mathbf h ^ N ) ] quad ( 3 ) mod\n\nwhere µ p ( h i ) refers to the max operation applied to each non-overlapping 3 window of p features in the feature map h i . Max-pooling reduces the output dimensionality by a factor of p , and it drives the model to capture the most salient local features from each feature map in the convolutional layer.\n\n3 Weset the stride size to be the same as the pooling length p to get non-overlapping regions.\n\nCoherence scoring: Finally, the max-pooled features are used in the output layer of the network to produce a coherence score y ∈ R .\n\ny = v ^ T mathbf m + b\n\nwhere v is the weight vector and b is a bias term.\n\nWhy it works: Intuitively, each filter detects a specific transition pattern (e.g., 'SS-O-X' for a coherent text), and if this pattern occurs somewhere in the grid, the resulting feature map will have a large value for that particular region and small values for other regions. By applying max pooling on this feature map, the network then discovers that the transition appeared in the grid."
      },
      {
        "heading": "3.1 Incorporating Entity-Specific Features",
        "text": "Our model as described above neuralizes the basic entity grid model that considers only entity transitions without distinguishing between types of the entities. However, as Elsner and Charniak (2011) pointed out entity-specific features could be crucial for modeling local coherence. One simple way to incorporate entity-specific features into our model is to attach the feature value (e.g., named entity type) with the grammatical role in the grid. For example, if an entity e j of type PERSON appears as a subject (S) in sentence s i , the grid entry G i,j can be encoded as PERSON-S."
      },
      {
        "heading": "3.2 Training",
        "text": "Our neural model assigns a coherence score to an input document d based on the degree of lo- cal coherence observed in its entity grid G . Let y = φ ( G | θ ) define our model that transforms an input grid G to a coherence score y through a sequence of lookup, convolutional, pooling, and linear projection layers with parameter set θ . The parameter set θ includes the embedding matrix E , the filter matrix W , the weight vector v , and the biases. We use a pairwise ranking approach (Collobert et al., 2011) to learn θ .\n\nThe training set comprises ordered pairs ( d i , d j ) , where document d i exhibits a higher degree of coherence than document d j . As we will see in Section 4 such orderings can be obtained automatically or through manual annotation. In training, we seek to find θ that assigns a higher coherence score to d i than to d j . We minimize the following ranking objective with respect to θ :\n\nJ ( θ ) = max \\ 0 , 1 - φ ( G _ i | θ ) + φ ( G _ j | θ ) \\ quad ( 5 ) quad diffe\n\nwhere G i and G j are the entity grids corresponding to documents d i and d j , respectively. Notice that (also shown in Figure 2) the network shares its layers (and hence θ ) to obtain φ ( G i | θ ) and φ ( G j | θ ) from a pair of input grids ( G i , G j ) .\n\nBarzilay and Lapata (2008) adopted a similar ranking criterion using an SVM preference kernel learner as they argue coherence assessment is best seen as a ranking problem as opposed to classification ( coherent vs. incoherent ). Also, the ranker gives a scoring function φ that a text generation system can use to compare alternative hypotheses."
      },
      {
        "heading": "4 Evaluation Tasks",
        "text": "We evaluate the effectiveness of our coherence models on two different evaluation tasks: sentence ordering and summary coherence rating."
      },
      {
        "heading": "4.1 Sentence Ordering",
        "text": "Following (Elsner and Charniak, 2011), we evaluate our models on two sentence ordering tasks: discrimination and insertion.\n\nIn the discrimination task (Barzilay and Lapata, 2008), a document is compared to a random permutation of its sentences, and the model is considered correct if it scores the original document higher than the permuted one. We use 20 permutations of each document in the test set in accordance with previous work.\n\nIn the insertion task (Elsner and Charniak, 2011), we evaluate models based on their ability to locate the original position of a sentence previously removed from a document. To measure this, each sentence in the document is removed in turn, and an insertion place is located for which the model gives the highest coherence score to the document. The insertion score is then computed as the average fraction of sentences per document reinserted in their actual position.\n\nSections # Doc. # Pairs Avg. # Sen. TRAIN 00-13 1,378 26,422 21.5 TEST 14-24 1,053 20,411 22.3\n\nTable 1: Statistics on WSJ dataset.\n\nDiscrimination can be easier for longer documents, since a random permutation is likely to be different than the original one. Insertion is a much more difficult task since the candidate documents differ only by the position of one sentence.\n\nDataset: For sentence ordering tasks, we use the Wall Street Journal (WSJ) portion of Penn Treebank, as used by (Elsner and Charniak, 2008, 2011; Lin et al., 2011; Feng et al., 2014). Table 1 gives basic statistics about the dataset. Following previous works, we use 20 random permutations of each article, and we exclude permutations that match the original document. 4 The fourth column (# Pairs) in Table 1 shows the resulting number of ( original , permuted ) pairs used for training our model and for testing in the discrimination task.\n\nSome previous studies (Barzilay and Lapata, 2008; Li and Hovy, 2014) used the AIRPLANES and the EARTHQUAKES corpora, which contain reports on airplane crashes and earthquakes, respectively. Each of these corpora contains 100 articles for training and 100 articles for testing. The average number of sentences per article in these two corpora is 10 . 4 and 11 . 5 , respectively.\n\nWe preferred WSJ corpus for several reasons. First and most importantly, WSJ corpus is larger than other corpora (see Table 1). Large training set is crucial for learning effective deep learning models (Collobert et al., 2011), and a large enough test set is necessary to make a general comment about model performance. Secondly, as Elsner and Charniak (2011) pointed out, texts in AIRPLANES and EARTHQUAKES are constrained in style, whereas WSJ documents are more like normal informative articles. Thirdly, we could re- produce results on this dataset for the competing systems (e.g., entity grid and its extensions) using publicly available Brown coherence toolkit. 5\n\n4 Short articles may produce many matches."
      },
      {
        "heading": "4.2 Summary Coherence Rating",
        "text": "We further evaluate our models on the summary coherence rating task proposed by Barzilay and Lapata (2008), where we compare rankings given by a model to a pair of summaries against rankings elicited from human judges.\n\nDataset: The summary dataset was extracted from the Document Understanding Conference (DUC'03), which contains 6 clusters of multidocument summaries produced by human experts and 5 automatic summarization systems. Each cluster has 16 summaries of a document with pairwise coherence rankings given by humans judges; see (Barzilay and Lapata, 2008) for details on the annotation method. There are 144 pairs of summaries for training and 80 pairs for testing."
      },
      {
        "heading": "5 Experiments",
        "text": "In this section, we present our experiments - the models we compare, their settings, and the results."
      },
      {
        "heading": "5.1 Models Compared",
        "text": "We compare our coherence model against a random baseline and several existing models.\n\nRandom: The Random baseline makes a random decision for the evaluation tasks.\n\nGraph-based Model: This is the graph-based unsupervised model proposed by Guinaudeau and Strube (2013). We use the implementation from the cohere 6 toolkit (Smith et al., 2016), and run it on the test set with syntactic projection (command line option 'projection=3') for graph construction. This setting yielded best scores for this model.\n\nGrid-all nouns (E&C): This is the simple extension of the original entity grid model, where all nouns are considered as entities. Elsner and Charniak (2011) report significant gains by considering all nouns as opposed to only head-nouns. Results for this model were obtained by training the baseline entity grid model (command line option '-n') in Brown coherence toolkit on our dataset.\n\n5 https://bitbucket.org/melsner/browncoherence\n\n6 https://github.com/karins/CoherenceFramework\n\nExtended grid (E&C): This represents the extended entity grid model of Elsner and Charniak (2011) that uses 9 entity-specific features; 4 of them were computed from external corpora. This model considers all nouns as entities. For this system, we train the extended grid model (command line option '-f') in Brown coherence toolkit.\n\nGrid-CNN: This is our proposed neural extension of the basic entity grid (all nouns), where we only consider entity transitions as input.\n\nExtended Grid-CNN: This corresponds to our neural model that incorporates entity-specific features following the method described in Section 3.1. To keep the model simple, we include only three entity-specific features from (Elsner and Charniak, 2011) that are easy to compute and do not require any external corpus. The features are: ( i ) named entity type, ( ii ) salience as determined by occurrence frequency of the entity, and ( iii ) whether the entity has a proper mention.\n\nA remark: We also experimented with the distributed sentence model proposed recently by Li and Hovy (2014). We trained it on our WSJ corpus using their code 7 with the same setting that produced the best results on the discrimination task. However, the results on our dataset were very disappointing - accuracy of only 19 . 34% in discrimination. We were not sure what could go wrong, therefore, we excluded it from our table of results."
      },
      {
        "heading": "5.2 Settings for Neural Models",
        "text": "We held out 10% of the training documents to form a development set (DEV) on which we tune the hyper-parameters of our neural models. For discrimination and insertion tasks, the resulting DEV set contains 138 articles and 2,678 pairs after removing the permutations that match the original documents. For the summary rating task, DEV contains 14 pairs of summaries.\n\nWe implement our models in Theano (Theano Development Team, 2016). We use rectified linear units (ReLU) as activations ( f ). The embedding matrix is initialized with samples from uniform distribution U ( -0 . 01 , 0 . 01) , and the weight matrices are initialized with samples from glorotuniform distribution (Glorot and Bengio, 2010).\n\nWe train the models by optimizing the pairwise ranking loss in Equation 5 using the gradientbased online learning algorithm RMSprop with parameters ( ρ and ϵ ) set to the values suggested by Tieleman and Hinton (2012). 8 We use up to 25 epochs. To avoid overfitting, we use dropout (Srivastava et al., 2014) of hidden units, and do early stopping by observing accuracy on the DEV set - if the accuracy does not increase for 10 consecutive epochs, we exit with the best model recorded so far. We search for optimal mini batch size in { 16 , 32 , 64 , 128 } , emb edding size in { 80 , 100 , 200 } , dropout rate in { 0 . 2 , 0 . 3 , 0 . 5 } , filter number in { 100 , 150 , 200 , 300 } , win dow size in { 2 , 3 , 4 , 5 , 6 , 7 , 8 } , and pool ing length in { 3 , 4 , 5 , 6 , 7 } . Table 2 shows the optimal hyperparameter setting for our models. The best model on DEV is then used for the final evaluation on the TEST set. We run each experiment five times, each time with a different random seed, and we report the average of the runs to avoid any randomness in results. Statistical significance tests are done using an approximate randomization test based on the accuracy. We used SIGF V.2 (Pad´ o, 2006) with 10,000 iterations.\n\n7 http://cs.stanford.edu/ bdlijiwei/code/\n\nBatch Emb. Dropout Filter Win. Pool Grid-CNN 128 100 0.5 150 6 6 Ext. Grid-CNN 32 100 0.5 150 5 6\n\nTable 2: Optimal hyper-parameter setting for our neural models based on development set accuracy.\n\nDiscr..Acc Discr..F 1 Ins.. Random 50.0 50.0 12.60 Graph-based (G&S) 64.23 65.01 11.93 Grid-all nouns (E&C) 81.58 81.60 22.13 Extended Grid (E&C) 84.95 84.95 23.28 Grid-CNN 85.57 † 85.57 † 23.12 Extended Grid-CNN 88.69 † 88.69 † 25.95 †\n\nTable 3: Results on Discr imination and Ins ertion tasks. † indicates a neural model is significantly superior to its nonneural counterpart with p-value < 0 . 01 ."
      },
      {
        "heading": "5.3 Results on Sentence Ordering",
        "text": "Table 3 shows the results on discrimination and insertion tasks. Among the existing models, the graph-based model gets the lowest scores, where the extended grid gets the highest scores on both tasks. By neuralizing the basic grid model (Grid- all nouns), our Grid-CNN model delivers absolute improvements of about 4% in discrimination and 1% in insertion. When we compare our Extended Grid-CNN with its non-neural counterpart Extended Grid, we observe similar gains in discrimination and more gains ( 2 . 5% ) in insertion. Note that the Extended Grid-CNN yields these improvements considering only a subset of the Extended Grid features. This demonstrates the effectiveness of distributed representation and convolutional feature learning method.\n\n8 Other adaptive algorithms, e.g., ADAM (Kingma and Ba, 2014), ADADELTA (Zeiler, 2012) gave similar results.\n\nAcc F 1 Random 50.0 50.0 Graph-based (G&S) 80.0 81.5 Grid (B&L) 83.8 - Grid-CNN 85.0 85.0 Extended Grid-CNN 86.3 86.3 Pre-trained Grid-CNN 86.3 86.3 Pre-trained Ext. Grid-CNN 87.5 87.5\n\nTable 4: Results on Summary Coherence Rating tasks.\n\nCompared to discrimination, gain in insertion is less verbose. There could be two reasons. First, as mentioned before, insertion is a harder task than discrimination. Second, our models were not trained specifically on the insertion task. The model that is trained to distinguish an original document from its random permutation may learn features that are not specific enough to distinguish documents when only one sentence differs. It will be interesting to see how the model performs when we train it on the insertion task directly."
      },
      {
        "heading": "5.4 Results on Summary Coherence Rating",
        "text": "Table 4 presents the results on the summary coherence rating task, where we compare our models with the graph-based method and the reported results of Barzilay and Lapata (2008) on the same experimental setting. 9 Since there are not many training instances, our neural models may not learn well for this task. Therefore, we also present versions of our model, where we use pre-trained models from discrimination task on WSJ corpus (last two rows in the table ). The pre-trained models are then fi ne-tuned on the summary rating task.\n\nWe can observe that even without pre-training our models outperform existing models, and pretraining gives further improvements. Specifically, Pre-trained Grid-CNN gives an improvement of 2 . 5% over the Grid model, and including entity features pushes the improvement further to 3 . 7% .\n\n9 The extended grid model does not use pairwise training, therefore could not be trained on the summarization dataset."
      },
      {
        "heading": "6 Related Work",
        "text": "Barzilay and Lapata (2005, 2008) introduced the entity grid representation of discourse to model local coherence that captures the distribution of discourse entities across sentences in a text. They also introduced three tasks to evaluate the performance of coherence models: discrimination, summary coherence rating, and readability.\n\nA number of extensions of the basic entity grid model has been proposed. Elsner and Charniak (2011) included entity-specific features to distinguish between entities. Feng and Hirst (2012) used the basic grid representation, but improved its learning to rank scheme. Their model learns not only from original document and its permutations but also from ranking preferences among the permutations themselves. Guinaudeau and Strube (2013) convert a standard entity grid into a bipartite graph representing entity occurrences in sentences. To model local entity transition, the method constructs a directed projection graph representing the connection between adjacent sentences. Two sentences have a connected edge if they share at least one entity in common. The coherence score of the document is then computed as the average out-degree of sentence nodes.\n\nIn addition, there are some approaches that model text coherence based on coreferences and discourse relations. Elsner and Charniak (2008) proposed the discourse-new model by taking into account mentions of all referring expression (i.e., NPs) whether they are first mention ( discoursenew ) or subsequent ( discourse-old ) mentions. Given a document, they run a maximum-entropy classifier to detect each NP as a label L np ∈ { new,old } . The coherence score of the document is then estimated by ∏ np : NPs P ( L np | np ) . In this work, they also estimate text coherence through pronoun coreference modeling. Lin et al. (2011) assume that a coherent text has certain discourse relation patterns. Instead of modeling entity transitions, they model discourse role transitions between sentences. In a follow up work, Feng et al. (2014) trained the same model but using features derived from deep discourse struc- tures annotated with Rhetorical Structure Theory or RST (Mann and Thompson, 1988b) relations. Louis and Nenkova (2012) introduced a coherence model based on syntactic patterns in text by assuming that sentences in a coherent discourse should share the same structural syntactic patterns.\n\nIn recent years, there has been a growing interest in neuralizing traditional NLP approaches language modeling (Bengio et al., 2003), sequence tagging (Collobert et al., 2011), syntactic parsing (Socher et al., 2013), and discourse parsing (Li et al., 2014), etc. Following this tradition, in this paper we propose to neuralize the popular entity grid models. Li and Hovy (2014) also proposed a neural framework to compute coherence score of a document by estimating coherence probability for every window of L sentences (in their experiments, L = 3 ). First, they use a recurrent or a recursive neural network to compute the representation for each sentence in L from its words and their pre-trained embeddings. Then the concatenated vector is passed through a non-linear hidden layer, and finally the output layer decides if the window of sentences is a coherent text or not. Our approach is fundamentally different from their approach; our model operates over entity grids, and we use convolutional architecture to model sufficiently long entity transitions."
      },
      {
        "heading": "7 Conclusion and Future Work",
        "text": "We presented a local coherence model based on a convolutional neural network that operates over the distributed representation of entity transitions in the grid representation of a text. Our architecture can model sufficiently long entity transitions, and can incorporate entity-specific features without loosing generalization power. We described a pairwise ranking approach to train the model on a target task and learn task-specific features. Our evaluation on discrimination, insertion and summary coherence rating tasks demonstrates the effectiveness of our approach yielding the best results reported so far on these tasks.\n\nIn future, we would like to include other sources of information in our model. Our initial plan is to include rhetorical relations, which has been shown to benefit existing grid models (Feng et al., 2014). We would also like to extend our model to other forms of discourse, especially, asynchronous conversations, where participants communicate with each other at different times (e.g., forum, email)."
      },
      {
        "heading": "A Supplemental Material",
        "text": ""
      }
    ],
    "references": [
      {
        "title": "Logics of Conversation , Cambridge University Press",
        "author": [
          "N. Asher",
          "A. Lascarides."
        ],
        "venue": "",
        "citeRegEx": "Asher and Lascarides.,? 2003",
        "shortCiteRegEx": "Asher and Lascarides.",
        "year": 2003
      },
      {
        "title": "Modeling local coherence: An entity-based approach",
        "author": [
          "Regina Barzilay",
          "Mirella Lapata."
        ],
        "venue": "In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics . Association for Computational Linguistics, Ann Arbor, Michigan, ACL '05, pages 141-148.",
        "citeRegEx": "Barzilay and Lapata.,? 2005",
        "shortCiteRegEx": "Barzilay and Lapata.",
        "year": 2005
      },
      {
        "title": "Modeling local coherence: An entity-based approach",
        "author": [
          "Regina Barzilay",
          "Mirella Lapata."
        ],
        "venue": "Computational Linguistics 34(1):1-34. http://www.aclweb.org/anthology/J08-1001.",
        "citeRegEx": "Barzilay and Lapata.,? 2008",
        "shortCiteRegEx": "Barzilay and Lapata.",
        "year": 2008
      },
      {
        "title": "A neural probabilistic language model",
        "author": [
          "Yoshua Bengio",
          "R´ ejean Ducharme",
          "Pascal Vincent",
          "Christian Janvin."
        ],
        "venue": "J. Mach. Learn. Res. 3. http://dl.acm.org/citation.cfm?id=944919.944966.",
        "citeRegEx": "Bengio et al\\.,? 2003",
        "shortCiteRegEx": "Bengio et al\\.",
        "year": 2003
      },
      {
        "title": "Using entity-based features to model coherence in student essays",
        "author": [
          "Jill Burstein",
          "Joel Tetreault",
          "Slava Andreyev."
        ],
        "venue": "In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics . Association for Computational Linguistics, Los Angeles, California, HLT '10, pages 681-684.",
        "citeRegEx": "Burstein et al\\.,? 2010",
        "shortCiteRegEx": "Burstein et al\\.",
        "year": 2010
      },
      {
        "title": "Natural language processing (almost) from scratch",
        "author": [
          "Ronan Collobert",
          "Jason Weston",
          "L´ eon Bottou",
          "Michael Karlen",
          "Koray Kavukcuoglu",
          "Pavel Kuksa."
        ],
        "venue": "The Journal of Machine Learning Research 12:2493-2537.",
        "citeRegEx": "Collobert et al\\.,? 2011",
        "shortCiteRegEx": "Collobert et al\\.",
        "year": 2011
      },
      {
        "title": "Coreference-inspired coherence modeling",
        "author": [
          "Micha Elsner",
          "Eugene Charniak."
        ],
        "venue": "In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers . Association for Computational Linguistics, Columbus, Ohio, HLT-Short '08, pages 41-44.",
        "citeRegEx": "Elsner and Charniak.,? 2008",
        "shortCiteRegEx": "Elsner and Charniak.",
        "year": 2008
      },
      {
        "title": "Extending the entity grid with entity-specific features",
        "author": [
          "Micha Elsner",
          "Eugene Charniak."
        ],
        "venue": "In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2 . Association for Computational Linguistics, Portland, Oregon, HLT '11, pages 125-129.",
        "citeRegEx": "Elsner and Charniak.,? 2011",
        "shortCiteRegEx": "Elsner and Charniak.",
        "year": 2011
      },
      {
        "title": "Extending the entity-based coherence model with multiple ranks",
        "author": [
          "Vanessa Wei Feng",
          "Graeme Hirst."
        ],
        "venue": "In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics . Association for Computational Linguistics, Avignon, France, EACL '12, pages 315-324.",
        "citeRegEx": "Feng and Hirst.,? 2012",
        "shortCiteRegEx": "Feng and Hirst.",
        "year": 2012
      },
      {
        "title": "The impact of deep hierarchical discourse structures in the evaluation of text coherence",
        "author": [
          "Vanessa Wei Feng",
          "Ziheng Lin",
          "Graeme Hirst."
        ],
        "venue": "In COLING .",
        "citeRegEx": "Feng et al\\.,? 2014",
        "shortCiteRegEx": "Feng et al\\.",
        "year": 2014
      },
      {
        "title": "Extending the entity-grid coherence model to semantically related entities",
        "author": [
          "Katja Filippova",
          "Michael Strube."
        ],
        "venue": "In Proceedings of the Eleventh European Workshop on Natural Language Generation . Association for Computational Linguistics, Germany, ENLG '07, pages 139-142.",
        "citeRegEx": "Filippova and Strube.,? 2007",
        "shortCiteRegEx": "Filippova and Strube.",
        "year": 2007
      },
      {
        "title": "Understanding the difficulty of training deep feedforward neural networks",
        "author": [
          "Xavier Glorot",
          "Yoshua Bengio."
        ],
        "venue": "In JMLR W&CP: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2010) . Sardinia, Italy, volume 9, pages 249-256.",
        "citeRegEx": "Glorot and Bengio.,? 2010",
        "shortCiteRegEx": "Glorot and Bengio.",
        "year": 2010
      },
      {
        "title": "Centering: A framework for modeling the local coherence of discourse",
        "author": [
          "Barbara J. Grosz",
          "Scott Weinstein",
          "Aravind K. Joshi."
        ],
        "venue": "Comput. Linguist. 21(2):203-225.",
        "citeRegEx": "Grosz et al\\.,? 1995",
        "shortCiteRegEx": "Grosz et al\\.",
        "year": 1995
      },
      {
        "title": "Graph-based local coherence modeling",
        "author": [
          "Camille Guinaudeau",
          "Michael Strube."
        ],
        "venue": "In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers . pages 93-103.",
        "citeRegEx": "Guinaudeau and Strube.,? 2013",
        "shortCiteRegEx": "Guinaudeau and Strube.",
        "year": 2013
      },
      {
        "title": "Optimizing search engines using clickthrough data",
        "author": [
          "Thorsten Joachims."
        ],
        "venue": "In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . ACM, Edmonton, Alberta, Canada, KDD '02, pages 133-142.",
        "citeRegEx": "Joachims.,? 2002",
        "shortCiteRegEx": "Joachims.",
        "year": 2002
      },
      {
        "title": "A convolutional neural network for modelling sentences",
        "author": [
          "Nal Kalchbrenner",
          "Edward Grefenstette",
          "Phil Blunsom."
        ],
        "venue": "In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, Baltimore, Maryland, pages 655-665.",
        "citeRegEx": "Kalchbrenner et al\\.,? 2014",
        "shortCiteRegEx": "Kalchbrenner et al\\.",
        "year": 2014
      },
      {
        "title": "Convolutional neural networks for sentence classification",
        "author": [
          "Yoon Kim."
        ],
        "venue": "In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, Doha, Qatar, pages 17461751.",
        "citeRegEx": "Kim.,? 2014",
        "shortCiteRegEx": "Kim.",
        "year": 2014
      },
      {
        "title": "Adam: A method for stochastic optimization",
        "author": [
          "Diederik P. Kingma",
          "Jimmy Ba."
        ],
        "venue": "CoRR abs/1412.6980. http://arxiv.org/abs/1412.6980.",
        "citeRegEx": "Kingma and Ba.,? 2014",
        "shortCiteRegEx": "Kingma and Ba.",
        "year": 2014
      },
      {
        "title": "A model of coherence based on distributed sentence representation",
        "author": [
          "Jiwei Li",
          "Eduard Hovy."
        ],
        "venue": "In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, Doha, Qatar, pages 2039-2048. http://www.aclweb.org/anthology/D14-1218.",
        "citeRegEx": "Li and Hovy.,? 2014",
        "shortCiteRegEx": "Li and Hovy.",
        "year": 2014
      },
      {
        "title": "Recursive deep models for discourse parsing",
        "author": [
          "Jiwei Li",
          "Rumeng Li",
          "Eduard H Hovy."
        ],
        "venue": "In EMNLP . pages 2061-2069.",
        "citeRegEx": "Li et al\\.,? 2014",
        "shortCiteRegEx": "Li et al\\.",
        "year": 2014
      },
      {
        "title": "Automatically evaluating text coherence using discourse relations",
        "author": [
          "Ziheng Lin",
          "Hwee Tou Ng",
          "Min-Yen Kan."
        ],
        "venue": "In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1 . Association for Computational Linguistics, Portland, Oregon, HLT '11, pages 997-1006.",
        "citeRegEx": "Lin et al\\.,? 2011",
        "shortCiteRegEx": "Lin et al\\.",
        "year": 2011
      },
      {
        "title": "A coherence model based on syntactic patterns",
        "author": [
          "Annie Louis",
          "Ani Nenkova."
        ],
        "venue": "In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning . Association for Computational Linguistics, Stroudsburg, PA, USA, EMNLP-CoNLL '12, pages 1157-1168. http://dl.acm.org/citation.cfm?id=2390948.2391078.",
        "citeRegEx": "Louis and Nenkova.,? 2012",
        "shortCiteRegEx": "Louis and Nenkova.",
        "year": 2012
      },
      {
        "title": "Rhetorical Structure Theory: Toward a Functional Theory of Text Organization",
        "author": [
          "W. Mann",
          "S. Thompson."
        ],
        "venue": "Text 8(3):243-281.",
        "citeRegEx": "Mann and Thompson.,? 1988",
        "shortCiteRegEx": "Mann and Thompson.",
        "year": 1988
      },
      {
        "title": "Rhetorical structure theory: Toward a functional theory of text organization",
        "author": [
          "William C Mann",
          "Sandra A Thompson."
        ],
        "venue": "Text 8(3):243-281.",
        "citeRegEx": "Mann and Thompson.,? 1988",
        "shortCiteRegEx": "Mann and Thompson.",
        "year": 1988
      },
      {
        "title": "Plot induction and evolutionary search for story generation",
        "author": [
          "Neil McIntyre",
          "Mirella Lapata."
        ],
        "venue": "In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Uppsala, Sweden, ACL '10, pages 1562-1572.",
        "citeRegEx": "McIntyre and Lapata.,? 2010",
        "shortCiteRegEx": "McIntyre and Lapata.",
        "year": 2010
      },
      {
        "title": "Rectified linear units improve restricted boltzmann machines",
        "author": [
          "Vinod Nair",
          "Geoffrey E. Hinton."
        ],
        "venue": "In Johannes Frnkranz and Thorsten Joachims, editors, Proceedings of the 27th International Conference on Machine Learning (ICML-10) . Omnipress, pages 807-814. http://www.icml2010.org/papers/432.pdf.",
        "citeRegEx": "Nair and Hinton.,? 2010",
        "shortCiteRegEx": "Nair and Hinton.",
        "year": 2010
      },
      {
        "title": "User's guide to sigf : Significance testing by approximate randomisation ",
        "author": [
          "Sebastian Pad´ o."
        ],
        "venue": "",
        "citeRegEx": "o.,? 2006",
        "shortCiteRegEx": "o.",
        "year": 2006
      },
      {
        "title": "Automatic evaluation of linguistic quality in multidocument summarization",
        "author": [
          "Emily Pitler",
          "Annie Louis",
          "Ani Nenkova."
        ],
        "venue": "In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Uppsala, Sweden, ACL '10, pages 544554.",
        "citeRegEx": "Pitler et al\\.,? 2010",
        "shortCiteRegEx": "Pitler et al\\.",
        "year": 2010
      },
      {
        "title": "Cohere: A toolkit for local coherence",
        "author": [
          "Karin Sim Smith",
          "Wilker Aziz",
          "Lucia Specia."
        ],
        "venue": "In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016) . European Language Resources Association (ELRA), Portoroz, Slovenia.",
        "citeRegEx": "Smith et al\\.,? 2016",
        "shortCiteRegEx": "Smith et al\\.",
        "year": 2016
      },
      {
        "title": "Parsing with compositional vector grammars",
        "author": [
          "Richard Socher",
          "John Bauer",
          "Christopher D. Manning",
          "Ng Andrew Y."
        ],
        "venue": "In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, Sofia, Bulgaria, pages 455-465. http://www.aclweb.org/anthology/P13-1045.",
        "citeRegEx": "Socher et al\\.,? 2013",
        "shortCiteRegEx": "Socher et al\\.",
        "year": 2013
      },
      {
        "title": "Discourse generation using utility-trained coherence models",
        "author": [
          "Radu Soricut",
          "Daniel Marcu."
        ],
        "venue": "In Proceedings of the COLING/ACL on Main Conference Poster Sessions . Association for Computational Linguistics, Sydney, Australia, COLING-ACL '06, pages 803-810.",
        "citeRegEx": "Soricut and Marcu.,? 2006",
        "shortCiteRegEx": "Soricut and Marcu.",
        "year": 2006
      },
      {
        "title": "Dropout: A simple way to prevent neural networks from overfitting",
        "author": [
          "Nitish Srivastava",
          "Geoffrey Hinton",
          "Alex Krizhevsky",
          "Ilya Sutskever",
          "Ruslan Salakhutdinov."
        ],
        "venue": "Journal of Machine Learning Research 15:1929-1958.",
        "citeRegEx": "Srivastava et al\\.,? 2014",
        "shortCiteRegEx": "Srivastava et al\\.",
        "year": 2014
      },
      {
        "title": "Theano: A Python framework for fast computation of mathematical expressions",
        "author": [
          "Theano Development Team."
        ],
        "venue": "arXiv e-prints abs/1605.02688. http://arxiv.org/abs/1605.02688.",
        "citeRegEx": "Team.,? 2016",
        "shortCiteRegEx": "Team.",
        "year": 2016
      },
      {
        "title": "RMSprop , COURSERA: Neural Networks",
        "author": [
          "T. Tieleman",
          "G Hinton."
        ],
        "venue": "",
        "citeRegEx": "Tieleman and Hinton.,? 2012",
        "shortCiteRegEx": "Tieleman and Hinton.",
        "year": 2012
      },
      {
        "title": "ADADELTA: an adaptive learning rate method",
        "author": [
          "Matthew D. Zeiler."
        ],
        "venue": "CoRR abs/1212.5701. http://arxiv.org/abs/1212.5701.",
        "citeRegEx": "Zeiler.,? 2012",
        "shortCiteRegEx": "Zeiler.",
        "year": 2012
      }
    ],
    "referenceMentions": [
      {
        "referenceID": 22,
        "context": "Several formal theories of coherence have been proposed (Mann and Thompson, 1988a; Grosz et al., 1995; Asher and Lascarides, 2003), and their principles have inspired development of existing coherence models (Barzilay and Lapata, 2008; Lin et al., 2011; Li and Hovy, 2014).",
        "startOffset": 57,
        "endOffset": 80
      },
      {
        "referenceID": 23,
        "context": "Several formal theories of coherence have been proposed (Mann and Thompson, 1988a; Grosz et al., 1995; Asher and Lascarides, 2003), and their principles have inspired development of existing coherence models (Barzilay and Lapata, 2008; Lin et al., 2011; Li and Hovy, 2014).",
        "startOffset": 57,
        "endOffset": 80
      },
      {
        "referenceID": 12,
        "context": "Several formal theories of coherence have been proposed (Mann and Thompson, 1988a; Grosz et al., 1995; Asher and Lascarides, 2003), and their principles have inspired development of existing coherence models (Barzilay and Lapata, 2008; Lin et al., 2011; Li and Hovy, 2014).",
        "startOffset": 83,
        "endOffset": 101
      },
      {
        "referenceID": 0,
        "context": "Several formal theories of coherence have been proposed (Mann and Thompson, 1988a; Grosz et al., 1995; Asher and Lascarides, 2003), and their principles have inspired development of existing coherence models (Barzilay and Lapata, 2008; Lin et al., 2011; Li and Hovy, 2014).",
        "startOffset": 103,
        "endOffset": 129
      },
      {
        "referenceID": 2,
        "context": "Several formal theories of coherence have been proposed (Mann and Thompson, 1988a; Grosz et al., 1995; Asher and Lascarides, 2003), and their principles have inspired development of existing coherence models (Barzilay and Lapata, 2008; Lin et al., 2011; Li and Hovy, 2014).",
        "startOffset": 209,
        "endOffset": 234
      },
      {
        "referenceID": 20,
        "context": "Several formal theories of coherence have been proposed (Mann and Thompson, 1988a; Grosz et al., 1995; Asher and Lascarides, 2003), and their principles have inspired development of existing coherence models (Barzilay and Lapata, 2008; Lin et al., 2011; Li and Hovy, 2014).",
        "startOffset": 236,
        "endOffset": 252
      },
      {
        "referenceID": 18,
        "context": "Several formal theories of coherence have been proposed (Mann and Thompson, 1988a; Grosz et al., 1995; Asher and Lascarides, 2003), and their principles have inspired development of existing coherence models (Barzilay and Lapata, 2008; Lin et al., 2011; Li and Hovy, 2014).",
        "startOffset": 254,
        "endOffset": 271
      },
      {
        "referenceID": 2,
        "context": "Among these models, the entity grid (Barzilay and Lapata, 2008), which is based on Centering Theory (Grosz et al., 1995), is arguably the most popular, and has seen a number of improvements over the years.",
        "startOffset": 37,
        "endOffset": 62
      },
      {
        "referenceID": 12,
        "context": "Among these models, the entity grid (Barzilay and Lapata, 2008), which is based on Centering Theory (Grosz et al., 1995), is arguably the most popular, and has seen a number of improvements over the years.",
        "startOffset": 101,
        "endOffset": 119
      },
      {
        "referenceID": 7,
        "context": "Extensions of this basic grid model incorporate entity-specific features (Elsner and Charniak, 2011), multiple ranks (Feng and Hirst, 2012), and coherence relations (Feng et al., 2014).",
        "startOffset": 74,
        "endOffset": 99
      },
      {
        "referenceID": 8,
        "context": "Extensions of this basic grid model incorporate entity-specific features (Elsner and Charniak, 2011), multiple ranks (Feng and Hirst, 2012), and coherence relations (Feng et al., 2014).",
        "startOffset": 118,
        "endOffset": 138
      },
      {
        "referenceID": 9,
        "context": "Extensions of this basic grid model incorporate entity-specific features (Elsner and Charniak, 2011), multiple ranks (Feng and Hirst, 2012), and coherence relations (Feng et al., 2014).",
        "startOffset": 166,
        "endOffset": 183
      },
      {
        "referenceID": 3,
        "context": "Firstly, they use discrete representation for grammatical roles and features, which limits the model to consider sufficiently long transitions (Bengio et al., 2003).",
        "startOffset": 144,
        "endOffset": 163
      },
      {
        "referenceID": 2,
        "context": "We evaluate our approach on three different evaluation tasks: discrimination, insertion, and summary coherence rating, proposed previously for evaluating coherence models (Barzilay and Lapata, 2008; Elsner and Charniak, 2011).",
        "startOffset": 172,
        "endOffset": 197
      },
      {
        "referenceID": 7,
        "context": "We evaluate our approach on three different evaluation tasks: discrimination, insertion, and summary coherence rating, proposed previously for evaluating coherence models (Barzilay and Lapata, 2008; Elsner and Charniak, 2011).",
        "startOffset": 199,
        "endOffset": 224
      },
      {
        "referenceID": 12,
        "context": "Motivated by Centering Theory (Grosz et al., 1995), Barzilay and Lapata (2008) proposed an entity-based model for representing and assessing text coherence.",
        "startOffset": 31,
        "endOffset": 49
      },
      {
        "referenceID": 14,
        "context": "Assessment of text coherence is then formulated as a ranking problem in an SVM preference ranking framework (Joachims, 2002).",
        "startOffset": 109,
        "endOffset": 123
      },
      {
        "referenceID": 2,
        "context": "Entity grid and its extensions have been successfully applied to many downstream tasks including coherence rating (Barzilay and Lapata, 2008), essay scoring (Burstein et al., 2010), story generation (McIntyre and Lapata, 2010), and readability assessment (Pitler et al., 2010; Barzilay and Lapata, 2008).",
        "startOffset": 115,
        "endOffset": 140
      },
      {
        "referenceID": 4,
        "context": "Entity grid and its extensions have been successfully applied to many downstream tasks including coherence rating (Barzilay and Lapata, 2008), essay scoring (Burstein et al., 2010), story generation (McIntyre and Lapata, 2010), and readability assessment (Pitler et al., 2010; Barzilay and Lapata, 2008).",
        "startOffset": 158,
        "endOffset": 179
      },
      {
        "referenceID": 24,
        "context": "Entity grid and its extensions have been successfully applied to many downstream tasks including coherence rating (Barzilay and Lapata, 2008), essay scoring (Burstein et al., 2010), story generation (McIntyre and Lapata, 2010), and readability assessment (Pitler et al., 2010; Barzilay and Lapata, 2008).",
        "startOffset": 200,
        "endOffset": 225
      },
      {
        "referenceID": 27,
        "context": "Entity grid and its extensions have been successfully applied to many downstream tasks including coherence rating (Barzilay and Lapata, 2008), essay scoring (Burstein et al., 2010), story generation (McIntyre and Lapata, 2010), and readability assessment (Pitler et al., 2010; Barzilay and Lapata, 2008).",
        "startOffset": 256,
        "endOffset": 275
      },
      {
        "referenceID": 2,
        "context": "Entity grid and its extensions have been successfully applied to many downstream tasks including coherence rating (Barzilay and Lapata, 2008), essay scoring (Burstein et al., 2010), story generation (McIntyre and Lapata, 2010), and readability assessment (Pitler et al., 2010; Barzilay and Lapata, 2008).",
        "startOffset": 277,
        "endOffset": 302
      },
      {
        "referenceID": 30,
        "context": "They have also been critical components in state-of-the-art sentence ordering models (Soricut and Marcu, 2006; Elsner and Charniak, 2011; Lin et al., 2011).",
        "startOffset": 86,
        "endOffset": 109
      },
      {
        "referenceID": 7,
        "context": "They have also been critical components in state-of-the-art sentence ordering models (Soricut and Marcu, 2006; Elsner and Charniak, 2011; Lin et al., 2011).",
        "startOffset": 111,
        "endOffset": 136
      },
      {
        "referenceID": 20,
        "context": "They have also been critical components in state-of-the-art sentence ordering models (Soricut and Marcu, 2006; Elsner and Charniak, 2011; Lin et al., 2011).",
        "startOffset": 138,
        "endOffset": 154
      },
      {
        "referenceID": 3,
        "context": "· Existing models use discrete representation for grammatical roles and features, which leads to the so-called curse of dimensionality problem (Bengio et al., 2003).",
        "startOffset": 144,
        "endOffset": 163
      },
      {
        "referenceID": 7,
        "context": "This problem is exacerbated when we want to include entity-specific features, as the number of parameters grows exponentially with the number of features (Elsner and Charniak, 2011).",
        "startOffset": 155,
        "endOffset": 180
      },
      {
        "referenceID": 5,
        "context": "Under these assumptions, the natural choice to tackle this problem is to use a convolutional approach, used previously to solve other NLP tasks (Collobert et al., 2011; Kim, 2014).",
        "startOffset": 145,
        "endOffset": 167
      },
      {
        "referenceID": 16,
        "context": "Under these assumptions, the natural choice to tackle this problem is to use a convolutional approach, used previously to solve other NLP tasks (Collobert et al., 2011; Kim, 2014).",
        "startOffset": 169,
        "endOffset": 178
      },
      {
        "referenceID": 25,
        "context": "Convolution layer: A convolution operation involves applying a fi lter w ∈ R k.d (i.e., a vector of weight parameters) to each entity transition of length k to produce a new abstract feature\n\nh _ t = f ( mathbf w ^ T L _ t colon t + k - 1 , j + b _ t )\n\nwhere L t : t + k -1 ,j denotes the concatenation of k vectors in the lookup layer representing a transition of length k for entity e j in the grid, b t is a bias term, and f is a nonlinear activation function, e.g., ReLU (Nair and Hinton, 2010) in our model.",
        "startOffset": 477,
        "endOffset": 498
      },
      {
        "referenceID": 15,
        "context": "Notice that we use a wide convolution (Kalchbrenner et al., 2014), as opposed to narrow , to ensure that the filters reach entire columns of a grid, including the boundary entities.",
        "startOffset": 39,
        "endOffset": 64
      },
      {
        "referenceID": 5,
        "context": "We use a pairwise ranking approach (Collobert et al., 2011) to learn θ .",
        "startOffset": 36,
        "endOffset": 58
      },
      {
        "referenceID": 7,
        "context": "Following (Elsner and Charniak, 2011), we evaluate our models on two sentence ordering tasks: discrimination and insertion.",
        "startOffset": 11,
        "endOffset": 36
      },
      {
        "referenceID": 2,
        "context": "In the discrimination task (Barzilay and Lapata, 2008), a document is compared to a random permutation of its sentences, and the model is considered correct if it scores the original document higher than the permuted one.",
        "startOffset": 28,
        "endOffset": 53
      },
      {
        "referenceID": 7,
        "context": "In the insertion task (Elsner and Charniak, 2011), we evaluate models based on their ability to locate the original position of a sentence previously removed from a document.",
        "startOffset": 23,
        "endOffset": 48
      },
      {
        "referenceID": 6,
        "context": "Dataset: For sentence ordering tasks, we use the Wall Street Journal (WSJ) portion of Penn Treebank, as used by (Elsner and Charniak, 2008, 2011; Lin et al., 2011; Feng et al., 2014).",
        "startOffset": 113,
        "endOffset": 138
      },
      {
        "referenceID": 20,
        "context": "Dataset: For sentence ordering tasks, we use the Wall Street Journal (WSJ) portion of Penn Treebank, as used by (Elsner and Charniak, 2008, 2011; Lin et al., 2011; Feng et al., 2014).",
        "startOffset": 146,
        "endOffset": 162
      },
      {
        "referenceID": 9,
        "context": "Dataset: For sentence ordering tasks, we use the Wall Street Journal (WSJ) portion of Penn Treebank, as used by (Elsner and Charniak, 2008, 2011; Lin et al., 2011; Feng et al., 2014).",
        "startOffset": 164,
        "endOffset": 181
      },
      {
        "referenceID": 2,
        "context": "Some previous studies (Barzilay and Lapata, 2008; Li and Hovy, 2014) used the AIRPLANES and the EARTHQUAKES corpora, which contain reports on airplane crashes and earthquakes, respectively.",
        "startOffset": 23,
        "endOffset": 48
      },
      {
        "referenceID": 18,
        "context": "Some previous studies (Barzilay and Lapata, 2008; Li and Hovy, 2014) used the AIRPLANES and the EARTHQUAKES corpora, which contain reports on airplane crashes and earthquakes, respectively.",
        "startOffset": 50,
        "endOffset": 67
      },
      {
        "referenceID": 5,
        "context": "Large training set is crucial for learning effective deep learning models (Collobert et al., 2011), and a large enough test set is necessary to make a general comment about model performance.",
        "startOffset": 75,
        "endOffset": 97
      },
      {
        "referenceID": 2,
        "context": "Each cluster has 16 summaries of a document with pairwise coherence rankings given by humans judges; see (Barzilay and Lapata, 2008) for details on the annotation method.",
        "startOffset": 106,
        "endOffset": 131
      },
      {
        "referenceID": 28,
        "context": "We use the implementation from the cohere 6 toolkit (Smith et al., 2016), and run it on the test set with syntactic projection (command line option 'projection=3') for graph construction.",
        "startOffset": 53,
        "endOffset": 71
      },
      {
        "referenceID": 7,
        "context": "To keep the model simple, we include only three entity-specific features from (Elsner and Charniak, 2011) that are easy to compute and do not require any external corpus.",
        "startOffset": 79,
        "endOffset": 104
      },
      {
        "referenceID": 32,
        "context": "We implement our models in Theano (Theano Development Team, 2016).",
        "startOffset": 54,
        "endOffset": 64
      },
      {
        "referenceID": 11,
        "context": "01) , and the weight matrices are initialized with samples from glorotuniform distribution (Glorot and Bengio, 2010).",
        "startOffset": 92,
        "endOffset": 115
      },
      {
        "referenceID": 31,
        "context": "To avoid overfitting, we use dropout (Srivastava et al., 2014) of hidden units, and do early stopping by observing accuracy on the DEV set - if the accuracy does not increase for 10 consecutive epochs, we exit with the best model recorded so far.",
        "startOffset": 38,
        "endOffset": 61
      },
      {
        "referenceID": 26,
        "context": "We used SIGF V.2 (Pad´ o, 2006) with 10,000 iterations.",
        "startOffset": 23,
        "endOffset": 30
      },
      {
        "referenceID": 17,
        "context": "8 Other adaptive algorithms, e.g., ADAM (Kingma and Ba, 2014), ADADELTA (Zeiler, 2012) gave similar results.",
        "startOffset": 41,
        "endOffset": 60
      },
      {
        "referenceID": 34,
        "context": "8 Other adaptive algorithms, e.g., ADAM (Kingma and Ba, 2014), ADADELTA (Zeiler, 2012) gave similar results.",
        "startOffset": 73,
        "endOffset": 85
      },
      {
        "referenceID": 22,
        "context": "(2014) trained the same model but using features derived from deep discourse struc- tures annotated with Rhetorical Structure Theory or RST (Mann and Thompson, 1988b) relations.",
        "startOffset": 141,
        "endOffset": 164
      },
      {
        "referenceID": 23,
        "context": "(2014) trained the same model but using features derived from deep discourse struc- tures annotated with Rhetorical Structure Theory or RST (Mann and Thompson, 1988b) relations.",
        "startOffset": 141,
        "endOffset": 164
      },
      {
        "referenceID": 3,
        "context": "In recent years, there has been a growing interest in neuralizing traditional NLP approaches language modeling (Bengio et al., 2003), sequence tagging (Collobert et al., 2011), syntactic parsing (Socher et al., 2013), and discourse parsing (Li et al., 2014), etc.",
        "startOffset": 112,
        "endOffset": 131
      },
      {
        "referenceID": 5,
        "context": "In recent years, there has been a growing interest in neuralizing traditional NLP approaches language modeling (Bengio et al., 2003), sequence tagging (Collobert et al., 2011), syntactic parsing (Socher et al., 2013), and discourse parsing (Li et al., 2014), etc.",
        "startOffset": 152,
        "endOffset": 174
      },
      {
        "referenceID": 29,
        "context": "In recent years, there has been a growing interest in neuralizing traditional NLP approaches language modeling (Bengio et al., 2003), sequence tagging (Collobert et al., 2011), syntactic parsing (Socher et al., 2013), and discourse parsing (Li et al., 2014), etc.",
        "startOffset": 196,
        "endOffset": 215
      },
      {
        "referenceID": 19,
        "context": "In recent years, there has been a growing interest in neuralizing traditional NLP approaches language modeling (Bengio et al., 2003), sequence tagging (Collobert et al., 2011), syntactic parsing (Socher et al., 2013), and discourse parsing (Li et al., 2014), etc.",
        "startOffset": 241,
        "endOffset": 256
      },
      {
        "referenceID": 9,
        "context": "Our initial plan is to include rhetorical relations, which has been shown to benefit existing grid models (Feng et al., 2014).",
        "startOffset": 107,
        "endOffset": 124
      }
    ],
    "year": null,
    "abstractText": "Wepropose a local coherence model based on a convolutional neural network that operates over the entity grid representation of a text. The model captures long range entity transitions along with entity-specific features without loosing generalization, thanks to the power of distributed representation. We present a pairwise ranking method to train the model in an end-to-end fashion on a task and learn task-specific high level features. Our evaluation on three different coherence assessment tasks demonstrates that our model achieves state of the art results outperforming existing models by a good margin.",
    "creator": "Docling-granite"
  }
}