[
  {
    "sample_id": 0,
    "title": "4 and 7-bit Labeling for Projective and Non-Projective Dependency Trees",
    "abstract": "We introduce an encoding for syntactic parsing as sequence labeling that can represent any projective dependency tree as a sequence of 4-bit labels, one per word. The bits in each word’s label represent (1) whether it is a right or left dependent, (2) whether it is the outermost (left/right) dependent of its parent, (3) whether it has any left children and (4) whether it has any right children. We show that this provides an injective mapping from trees to labels that can be encoded and decoded in linear time. We then define a 7-bit extension that represents an extra plane of arcs, extending the coverage to almost full non-projectivity (over 99.9% empirical arc coverage). Results on a set of diverse treebanks show that our 7-bit encoding obtains substantial accuracy gains over the previously best-performing sequence labeling encodings.",
    "introduction": "Approaches that cast parsing as sequence labeling have gathered interest as they are simple, fast (Anderson and Gómez-Rodríguez, 2021), highly parallelizable (Amini and Cotterell, 2022) and produce outputs that are easy to feed to other tasks (Wang et al., 2019). Their main ingredient are the encodings that map trees into sequences of one discrete label per word. Thus, various such encodings have been proposed both for constituency (GómezRodríguez and Vilares, 2018; Amini and Cotterell, 2022) and dependency parsing (Strzyz et al., 2019; Lacroix, 2019; Gómez-Rodríguez et al., 2020).\nMost such encodings have an unbounded label set, whose cardinality grows with sentence length. An exception for constituent parsing is tetratagging (Kitaev and Klein, 2020). For dependency parsing, to our knowledge, no bounded encodings were known. Simultaneously to this work, Amini et al. (2023) have just proposed one: hexatagging, where projective dependency trees are represented by tagging each word with one of a set of 8 tags.1\n0 1 2 3 4 5 6 7 -ROOT- It should continue to be defanged .\n<* < \\>*/ <* < \\> >* 0100 0000 1111 0100 0000 1010 1100\nFigure 1: A dependency tree and its 4-bit encoding.\nContribution We present a bounded sequencelabeling encoding that represents any projective dependency tree with 4 bits (i.e., 16 distinct labels) per word. While this requires one more bit than hexatagging, it is arguably more straightforward, as the bits directly reflect properties of each node in the dependency tree without an intermediate constituent structure, as hexatagging requires. Also, it has a clear relation to existing bracketing encodings, and has a straightforward non-projective extension using 7 bits with almost full non-projective coverage. Empirical results show that our encoding provides more accurate parsers than the existing unbounded bracketing encodings, which had the best previous results among sequence-labeling encodings, although it underperforms hexatagging.",
    "conclusion": "We have presented two new bracketing encodings for dependency parsing as sequence labeling, which use a bounded number of labels. The 4-bit encoding, designed for projective trees, excels in projective treebanks and low-resource setups. The 7-bit encoding, designed to accommodate non-projectivity, clearly outperforms the best prior sequence-labeling encodings across a diverse set of treebanks. The source code is available at https://github.com/Polifack/ CoDeLin/releases/tag/1.25."
  },
  {
    "sample_id": 1,
    "title": "A2N: Attending to Neighbors for Knowledge Graph Inference",
    "abstract": "State-of-the-art models for knowledge graph completion aim at learning a fixed embedding representation of entities in a multirelational graph which can generalize to infer unseen entity relationships at test time. This can be sub-optimal as it requires memorizing and generalizing to all possible entity relationships using these fixed representations. We thus propose a novel attentionbased method to learn query-dependent representation of entities which adaptively combines the relevant graph neighborhood of an entity leading to more accurate KG completion. The proposed method is evaluated on two benchmark datasets for knowledge graph completion, and experimental results show that the proposed model performs competitively or better than existing state-of-the-art, including recent methods for explicit multi-hop reasoning. Qualitative probing offers insight into how the model can reason about facts involving multiple hops in the knowledge graph, through the use of neighborhood attention.",
    "introduction": "Knowledge graphs, such as Freebase (Bollacker et al., 2008), contain a wealth of structured knowledge in the form of relationships between entities and are useful for numerous end applications. However, knowledge graphs (KG)—whether automatically constructed or human curated—are incomplete (Banko et al., 2007) and thus automatic methods for KG completion have been an important area of research (Nickel et al., 2016). The task of KG completion requires inferring missing entity relationships from the observed graph and is often formulated as predicting a target entity for a given query of source entity e and relation r, that is, to complete the tuple (e, r, ?).\nMost state-of-the-art methods for KG completion learn vector embeddings of entities and relations (Bordes et al., 2013; Toutanova et al., 2015; Dettmers et al., 2017; Trouillon et al., 2016) which are used in conjunction with a (potentially parameterized) scoring function that scores every tuple in the graph. These embeddings are optimized such that the score for observed graph tuples is higher than a random tuple. While these models achieve good performance, they learn a fixeddimensional embedding for every entity, which necessitates that this embedding must memorize and then be able to generalize to infer all possible relationships for the entity, which may require multiple-hops of reasoning in the KG (Neelakantan et al., 2015; Das et al., 2017).\nIn contrast, it can be beneficial to compose embeddings from a query-relevant subset of the graph neighborhood of the entity. As a motivating example, consider answering the query (e, nationality, ?) for some entity e. Observing the KG neighbor (e, lived in, Maui), can allow us to project e into the Maui region of the embedding space which can lead to a high score for predicting the target USA (through an appropriate scoring function), as Maui and USA were close in embedding space due to other relations between them in KG. Note that here e can have a type that is very different than the type of Maui, for example e can be Oprah Winfrey in which case it’s type would be Actor but using the neighborhood we can still project it to be close to USA for the query.\nThus, we propose A2N, an effective model (Section 2) which, conditioned on the query, uses a bi-linear attention on the graph neighborhood of an entity to generate an embedding representation of the entity. This query-specific and neighborhood-informed representation is then used to score target entities for the query. Intuitively, for the example described above, the model\nFigure 1: An actual example of how the A2N model generates the answer for two different queries for the same entity Oprah Winfrey on the FB15k-237 graph. We show a subset of the top neighbors. Each neighbor is assigned a probability based on the query and the neighbor representations are pooled based on these probabilities to obtain the entity embedding for the source entity. Top 3 neighbors are in bold face.\ncan score neighbors connected via the lived in relations higher so that the resulting embedding of the entity would be in the US region of the embedding space which when scored in conjunction with the query relation nationality can yield a high score for the target entity US. Fig. 1 shows an actual example how the model scores the graph neighborhood for two different queries on the same node, attending to a different relevant subset of the neighborhood for each query.\nOn two standard benchmark datasets for KG completion (Dettmers et al., 2017) – FB15k-237 (Toutanova et al., 2015) and WN18RR – we show (Section 3.2) that the model performs competitively or better than existing state-of-the-art models. Qualitative analysis (Fig. 1, Section 3.2) shows that the model indeed assigns higher scores to relevant neighbors based on the query and provides insight into how the model answers queries requiring multiple hops.",
    "conclusion": "We proposed A2N, an attention-based model for learning query-dependent entity embeddings based on graph neighborhood. The model performs favorably when compared with state-of-theart models for KG completion. The model has attractive properties as it is interpretable and its number of parameters do not depend on the size of entity neighborhoods. Future research will look into applying such methods to reason jointly about text and KG, by attending to textual mentions of entities in addition to graph (Verga et al., 2016).\nFigure 2: Example of queries, their top prediction and the set of top 5 attention neighbors as well as their attention probabilities for the A2N model."
  },
  {
    "sample_id": 2,
    "title": "Abstractive Summarization Guided by Latent Hierarchical Document Structure",
    "abstract": "Sequential abstractive neural summarizers often do not use the underlying structure in the input article or dependencies between the input sentences. This structure is essential to integrate and consolidate information from different parts of the text. To address this shortcoming, we propose a hierarchy-aware graph neural network (HierGNN) which captures such dependencies through three main steps: 1) learning a hierarchical document structure through a latent structure tree learned by a sparse matrixtree computation; 2) propagating sentence information over this structure using a novel message-passing node propagation mechanism to identify salient information; 3) using graphlevel attention to concentrate the decoder on salient information. Experiments confirm HierGNN improves strong sequence models such as BART, with a 0.55 and 0.75 margin in average ROUGE-1/2/L for CNN/DM and XSum. Further human evaluation demonstrates that summaries produced by our model are more relevant and less redundant than the baselines, into which HierGNN is incorporated. We also find HierGNN synthesizes summaries by fusing multiple source sentences more, rather than compressing a single source sentence, and that it processes long inputs more effectively.1",
    "introduction": "Sequential neural network architectures in their various forms have become the mainstay in abstractive summarization (See et al., 2017; Lewis et al., 2020). However, the quality of machine-produced summaries still lags far behind the quality of human summaries (Huang et al., 2020a; Xie et al., 2021; Cao et al., 2022; Lebanoff et al., 2019). Due to their sequential nature, a challenge with neural summarizers is to capture hierarchical and inter-sentential dependencies in the summmarized document.\ntrained more than 100 Olympic medal-winning rowers. - 2 sentences are abbreviated here.\n4. The Royal Mail has painted more than 50 postboxes gold following Team GB’s gold medal haul at London 2012.\n5. Originally it said it was only painting them in winners home towns, or towns with which they are closely associated.\n6. Town mayor Elizabeth Hodgkin said: “ We are the home of rowing ... I feel very excited about it.\"\n- 5 sentences are abbreviated here.\nmaries given by human-written reference, BART (Lewis et al., 2020) and our HierGNN equipped with BART. BART’s summary fails to capture all information pieces as the reference (as highlighted in various colors), while HierGNN has advantages in combining the information from multiple locations in the source side.\nProgress in cognitive science suggests that humans construct and reason over a latent hierarchical structure of a document when reading the text in it (Graesser et al., 1994; Goldman et al., 1999). Such reasoning behavior includes uncovering the salient contents and effectively aggregating all related clues spreading across the documents to understand the document. Lebanoff et al. (2019) found that human editors usually prefer writing a summary by fusing information from multiple article sentences and reorganizing the information in summaries (sentence fusion), rather than dropping non-essential elements in an original sentence such as prepositional phrases and adjectives (sentence compression). Different summarization benchmarks show there are between 60-85% summary sentences that are generated by sentence fusing. These recent findings support our motivation to make use of hierarchical document structure when summarizing a document.\nWe present a document hierarchy-aware graph neural network (HierGNN), a neural encoder with a reasoning functionality that can be effectively incorporated into any sequence-to-sequence (seq2seq) neural summarizer. Our HierGNN first learns a latent hierarchical graph via a sparse variant of the matrix-tree computation (Koo et al., 2007; Liu et al., 2019a). It then formulates sentence-level reasoning as a graph propagation problem via a novel message passing mechanism. During decoding, a graph-selection attention mechanism serves as a source sentence selector, hierarchically indicating the attention module which tokens in the input sentences to focus on.\nOur experiments with HierGNN, incorporated into both pointer-generator networks (See et al., 2017) and BART (Lewis et al., 2020), confirm that HierGNN substantially improves both the nonpretrained and pretrained seq2seq baselines in producing high-quality summaries. Specifically, our best HierGNN-BART achieves an average improvement of 0.55 and 0.75 points in ROUGE-1/2/L on CNN/DM and XSum. Compared with a plain seq2seq model, HierGNN encourages the summarizers to favor sentence fusion more than sentence compression when generating summaries. Modeling the hierarchical document structure via our sparse matrix-tree computation also enables HierGNN to treat long sequences more effectively. In addition, our sparse adaptive variant of the matrixtree computation demonstrates a more powerful expressive ability over the original one (Koo et al., 2007; Liu et al., 2019a). We summarize our contributions as follows,\n• We present a novel encoder architecture for improving seq2seq summarizers. This architecture captures the hierarchical document structure via an adaptive sparse matrix-tree computation, with a new propagation rule for achieving intersentence reasoning.\n• We design a graph-selection attention mechanism to fully leverage the learned structural information during decoding in advantages over only using it in encoding.\n• Results on CNN/DM and XSum demonstrates the effectiveness of HierGNN in improving the quality of summaries for both non-pretrained and pretrained baselines. An in-depth analysis confirms our module improves the integration of information from multiple sites in the input article and that it is more effective in processing long sequence inputs.",
    "conclusion": "We propose HierGNN that can be used in tandem with existing generation models. The module learns the document hierarchical structure while being able to integrate information from different parts of the text as a form of reasoning. Our experiments verify that HierGNN is effective in improving the plain sequential summarization models."
  },
  {
    "sample_id": 3,
    "title": "Accelerating Sparse Matrix Operations in Neural Networks on Graphics Processing Units",
    "abstract": "Graphics Processing Units (GPUs) are commonly used to train and evaluate neural networks efficiently. While previous work in deep learning has focused on accelerating operations on dense matrices/tensors on GPUs, efforts have concentrated on operations involving sparse data structures. Operations using sparse structures are common in natural language models at the input and output layers, because these models operate on sequences over discrete alphabets. We present two new GPU algorithms: one at the input layer, for multiplying a matrix by a few-hot vector (generalizing the more common operation of multiplication by a one-hot vector) and one at the output layer, for a fused softmax and top-N selection (commonly used in beam search). Our methods achieve speedups over state-of-theart parallel GPU baselines of up to 7× and 50×, respectively. We also illustrate how our methods scale on different GPU architectures.",
    "introduction": "The speedups introduced by parallel architectures inspired the development of accelerators tailored towards specialized functions. Graphics Processing Units (GPUs) are now a standard platform for deep learning. GPUs provide faster model training and inference times compared to serial processors, because they can parallelize the linear algebra operations used so heavily in neural networks (Raina et al., 2009).\nCurrently, major open source toolkits (Abadi et al., 2016) provide additional layers of abstraction to support one or more parallel GPU architectures. The seamless compatibility with multiple GPUs allows researchers to train a single model on multiple hardware platforms with no significant changes to their code base and no specialized knowledge about the targeted architectures. The disadvantage of hardware agnostic APIs is the lack of optimizations for a set of task-specific functions.\nAdapting parallel neural operations to a specific hardware platform is required to obtain optimal speed. Since matrix operations are used heavily in deep learning, much research has been done on optimizing them on GPUs (Chetlur et al., 2014; Gupta et al., 2015). Recently, some efforts have been made to other kinds of operations: serial operations running on the GPU (Povey et al., 2016), operations not involving matrix multiplications (Bogoychev et al., 2018), and models using sparse structures (Zhang et al., 2016). In this paper, we focus on sparse operations running exclusively on the GPU architecture.\nMuch recent work in High Performance Computing (HPC) and Natural Language Processing (NLP) focuses on an expensive step of a model or models and optimizes it for a specific architecture. The lookup operation used in the input layer and the softmax function used in the output are two examples seen in machine translation, language modeling, and other tasks. Previous work has accelerated the softmax step by skipping it entirely (Devlin et al., 2014), or approximating it (Shim et al., 2017; Grave et al., 2017).\nAnother strategy is to fuse multiple tasks into a single step. This approach increases the room for parallelism. Recent efforts have fused the softmax and top-N operations to accelerate beam search on the GPU using similar approaches (Hoang et al., 2018; Milakov and Gimelshein, 2018). Our approach differs from former methods in the following aspects: We deliver a novel method tailored towards scenarios seen in Neural Machine Translation (NMT), we introduce a new GPU-specific method to obtain the top-N elements from a list of hypotheses using a different sorting mechanism, and we introduce a sparse lookup method for GPUs.\nNMT uses beam search during inference to limit the full set of potential output translations explored during decoding (Cho et al., 2014; Graves, 2012). This algorithm is widely used to obtain state-of-the-art results during test time. At each decoding time-step t, the top-N hypotheses are chosen for further expansion and the rest are discarded. The top-N selection part of the search has been accelerated using hashing methods to avoid a full sort (Shi et al., 2018; Pagh and Rodler, 2004). The aim of this paper is to both combine softmax and top-N operations seen in the last layer of a neural network and optimize the top-N selection operation used by several NMT models.\nOur work uses ideas from previous work to accelerate two different operations. We focus on operations that manipulate sparse structures (Saad, 1990). By sparse, we mean operations that only require a small fraction of the elements in a tensor to output the correct result. We propose two different optimizations for sparse scenarios in deep learning: The first operation involves the first layer of a neural network. We accelerate the first matrix multiplication using batched sparse vectors as input. The second operation is the computation of the softmax used for beam search. We combine the softmax and the top-N selection into one operation obtaining a speedup over a parallel stateof-the-art baseline. We show that our fused topN selection and sparse lookups achieve speedups of 7× and 50× relative to other parallel NVIDIA baselines.",
    "conclusion": "In this work, we introduce two parallel methods for sparse computations found in NMT. The first operation is the sparse multiplication found in the input layer, and the second one is a fused softmax and top-N. Both implementations outperform different parallel baselines. We obtained speedups of up to 7× for the sparse affine transformation, and 50× for the fused softmax and top-N task.3\nFuture work includes the fusion of additional operations in neural models. Matrix operations form the largest bottleneck in deep learning. The last affine transformation in deep neural models can be fused with our softmax and top-N methods. The fusion of these three operations requires a different implementation of the matrix multiplication, and shared memory usage."
  },
  {
    "sample_id": 4,
    "title": "Accented Speech Recognition With Accent-specific Codebooks",
    "abstract": "Speech accents pose a significant challenge to state-of-the-art automatic speech recognition (ASR) systems. Degradation in performance across underrepresented accents is a severe deterrent to the inclusive adoption of ASR. In this work, we propose a novel accent adaptation approach for end-to-end ASR systems using cross-attention with a trainable set of codebooks. These learnable codebooks capture accent-specific information and are integrated within the ASR encoder layers. The model is trained on accented English speech, while the test data also contained accents which were not seen during training. On the Mozilla Common Voice multi-accented dataset, we show that our proposed approach yields significant performance gains not only on the seen English accents (up to 37% relative improvement in word error rate) but also on the unseen accents (up to 5% relative improvement in WER). Further, we illustrate benefits for a zero-shot transfer setup on the L2Artic dataset. We also compare the performance with other approaches based on accent adversarial training.",
    "introduction": "Accents in speech typically refer to the distinctive way in which the words are pronounced by diverse speakers. While a speaker’s accent may be primarily derived from their native language, speech accents are also influenced by various other factors related to the geographic location, educational background, socio-economic and socio-linguistic factors like race, gender and cultural diversity (Benzeghiba et al., 2007). It is therefore infeasible to build automatic speech recognition (ASR) systems which comprehensively cover speech accents during training. In such scenarios, novel speech accents continue to have an adverse effect on ASR performance (Beringer et al., 1998; Aksënova et al., 2022). While humans effectively recognize speech from new and unseen accents (Clarke and Garrett, 2004), ASR systems show substantial degradation in performance when dealing with new accents that are unseen during training (Chu et al., 2021).\nPrior works attempting to address accent-related challenges for ASR can be categorized into three groups: i) multi-accent training (Huang et al., 2014; Elfeky et al., 2016), ii) accent-aware training using accent embeddings (Jain et al., 2018) or adversarial learning (Sun et al., 2018), and iii) accent adaptation using supervised (Rao and Sak, 2017; Winata et al., 2020) or unsupervised techniques (Turan et al., 2020). While partial success has been achieved using most of these approaches, the development of robust speech recognition systems that are invariant to accent differences in training and test remains a challenging problem.\nIn this work, we propose a new codebook based technique for accent adaptation of state-of-theart Conformer-based end-to-end (E2E) ASR models (Gulati et al., 2020). For each of the accents observed in the training data, we define a codebook with a predefined number of randomly-initialized vectors. These accent codes are integrated with the self-attended representations in each encoder layer via the cross-attention mechanism, similar to the perceiver framework (Jaegle et al., 2021). The ASR model is trained on multi-accented data with standard end-to-end (E2E) ASR objectives. The codes capture accent-specific information as the training progresses. During inference, we propose a beam-search decoding algorithm that searches over a combined set of hypotheses obtained by using each set of accent-specific codes (once for each seen accent) with the trained ASR model. On the Mozilla Common Voice (MCV) corpus, we observe significant improvements on both seen and new accents at test-time compared to the baseline and existing supervised accent-adaptation techniques.\nOur main contributions are:\n• We propose a new accent adaptation technique for Conformer-based end-to-end ASR models using cross-attention over a set of learnable codebooks. Our technique comprises learning accent-specific codes during training and a new beam-search decoding algorithm to perform an optimized combination of the codes from the seen accents. We demonstrate significant performance improvements on both seen and unseen accents over competitive baselines on the MCV dataset.\n• Even on a zero-shot setting involving a new accented evaluation set, L2-Arctic (Zhao et al., 2018), we show significant improvements using our codebooks trained using MCV.\n• We publicly release our train/development/test splits spanning different seen and unseen accents in the MCV corpus. Reproducible splits on MCV have been entirely missing in prior work and we hope this will facilitate fair comparisons across existing and new accentadaptation techniques.1",
    "conclusion": "In this work, we propose a new end-to-end technique for accented ASR that uses accent-specific codebooks and cross-attention to achieve significant performance improvements on seen and unseen accents at test time. We experiment with the Mozilla Common Voice corpus and show detailed ablations over our design choices. We also empirically analyze whether our codebooks encode information relevant to accents. The effective use of codebooks for accents opens up future avenues to encode non-semantic cues in speech that affect ASR performance, such as types of noise, dialects, emotion styles of speech, etc."
  },
  {
    "sample_id": 5,
    "title": "Acquiring language from speech by learning to remember and predict",
    "abstract": "Classical accounts of child language learning invoke memory limits as a pressure to discover sparse, language-like representations of speech, while more recent proposals stress the importance of prediction for language learning. In this study, we propose a broadcoverage unsupervised neural network model to test memory and prediction as sources of signal by which children might acquire language directly from the perceptual stream. Our model embodies several likely properties of real-time human cognition: it is strictly incremental, it encodes speech into hierarchically organized labeled segments, it allows interactive top-down and bottom-up information flow, it attempts to model its own sequence of latent representations, and its objective function only recruits local signals that are plausibly supported by human working memory capacity. We show that much phonemic structure is learnable from unlabeled speech on the basis of these local signals. We further show that remembering the past and predicting the future both contribute to the linguistic content of acquired representations, and that these contributions are at least partially complementary.",
    "introduction": "How children acquire language from the environment is one of the fundamental mysteries of cognitive science. Much theoretical, experimental, and computational research into this question has focused on acquiring abstractions over lowerorder symbols, such acquiring morphemes from phoneme sequences or syntactic structures from word sequences (Chomsky, 1965; Gold, 1967; Elman, 1991; Saffran et al., 1996; Albright, 2002; Klein and Manning, 2004; Goldwater et al., 2009; Christodoulopoulos et al., 2012, inter alia). Children, however, do not get symbolic input; symbolic representations at any level of granularity constitute abstractions inferred from highly variable, noisy, and information-rich perceptual signals like audition and vision. This work joins a growing computational literature exploring the kinds of architectures and learning objectives that best support acquisition of linguistic representations directly from the speech signal without supervision (Versteegh et al., 2015; Dunbar et al., 2017). Such models can be used to test questions about language acquisition under more realistic assumptions about the input signal, especially to the extent that they reflect known constraints on human cognition (Shain and Elsner, 2019; Beguˇ s, 2020).\nThis study uses computational modeling to examine two influential and possibly complementary ideas about how people learn abstract representations, including language, from data: learning to remember, and learning to predict. Both hypotheses have been advocated by prior work in language acquisition, cognitive neuroscience, and computational modeling, yet their relative contributions to language learning are not yet clear. Our model permits precise manipulation of memory and prediction pressures during acquisition, allowing direct comparison of these hypotheses.\nIn so doing, we implement several constraints on real-time language processing that have not been simultaneously present in prior modeling of this domain: (1) we jointly segment and label the speech signal without supervision; (2) the learning objective is applied incrementally during real-time processing using only locally available feedback; (3) the encoded signal is segmental, sparse, and hierarchically organized; (4) segments are represented featurally as patterns of activation, rather than discrete category symbols; and (5) the system is optimized by modeling its own state at multiple timescales, rather than by modeling the data alone.\nResults show a systematic improvement along multiple measures of phoneme induction quality from both learning to remember and learning to predict, suggesting that these two kinds of signals may play complementary roles during child language acquisition. The contributions of this work are as follows:\n• We propose a novel deep neural encoderdecoder for unsupervised speech processing that is incremental, segmental, and useful for testing hypothesized cognitive constraints.\n• We show empirically that memory-based and prediction-based signals contribute separately to the acquisition of linguistic regularities, simultaneously supporting two existing classes of theories about the learning pressures that underlie human language acquisition.",
    "conclusion": "We proposed an unsupervised deep neural model of speech processing that is incremental, segmental, and optimized by local feedback. We manipulated the model’s objective function in order to investigate prior hypotheses about the role in human language acquisition of memory constraints on the one hand and predictive processing on the other. Results support a role for both memory and prediction pressures for acquiring phonemes from speech. Both objectives inform the model’s segmentation behavior and the content of its segment encodings. In addition, results suggest that these two mechanisms coordinate to support phoneme discovery by introducing countervailing pressures toward retention of previously encountered signals (memory) and consultation of top-down signals (prediction)."
  },
  {
    "sample_id": 6,
    "title": "ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation",
    "abstract": "Label aggregation such as majority voting is commonly used to resolve annotator disagreement in dataset creation. However, this may disregard minority values and opinions. Recent studies indicate that learning from individual annotations outperforms learning from aggregated labels, though they require a considerable amount of annotation. Active learning, as an annotation cost-saving strategy, has not been fully explored in the context of learning from disagreement. We show that in the active learning setting, a multi-head model performs significantly better than a single-head model in terms of uncertainty estimation. By designing and evaluating acquisition functions with annotator-specific heads on two datasets, we show that group-level entropy works generally well on both datasets. Importantly, it achieves performance in terms of both prediction and uncertainty estimation comparable to full-scale training from disagreement, while saving 70% of the annotation budget.",
    "introduction": "An important aspect of creating a dataset is asking for multiple annotations and aggregating them in order to derive a single ground truth label. Aggregating annotations, however, implies a single golden ground truth, which is not applicable to many subjective tasks such as hate speech detection (Ovesdotter Alm, 2011). A human’s judgement on subjective tasks can be influenced by their perspective and beliefs or cultural background (Waseem et al., 2021; Sap et al., 2022). When addressing disagreement in annotation, aggregating them by majority vote could result in the viewpoints of the minority being overlooked (Suresh and Guttag, 2019).\nIn order to address this issue, many works have been proposed to directly learn from the annotation disagreements in subjective tasks. There are two major approaches to achieving that: learning from the soft label (Peterson et al., 2019; Uma et al.,\n! \"\nWho to ask?\nMulti-head Model\nFigure 1: For each sample that needs to be labelled, our model actively selects specific annotators for annotations to learn from the label variation.\n2020; Fornaciari et al., 2021) and learning from the hard label of individual annotators (Cohn and Specia, 2013; Rodrigues and Pereira, 2018; Davani et al., 2022).\nIn a recent work, Davani et al. (2022) shows that modelling the individual annotators by adding annotator-specific classification heads in a multitask setup outperforms the traditional approach that learns from a majority vote. However, training such a model needs a huge amount of data with multiple annotations to model the opinions and beliefs of the individual annotators.\nOn another line, Active Learning (AL) is a framework that allows learning from limited labelled data by querying the data to be annotated. In this paper, we propose to take the best of both worlds: active learning and human label variation, to mitigate the high cost of the annotation budget needed for training the model. In particular, we propose a novel active learning setting, where the multi-head model actively selects the annotator and the sample to be labelled. Our results show this effectively reduces annotation costs while at the same time allowing for modelling individual perspectives.\nKey Findings We made several key observations:\n• The multi-head model works significantly better than the single-head model on uncertainty estimation.\n• The use of group-level entropy is generally recommended. Individual-level entropy methods perform differently depending on the dataset properties.\n• The multi-head model achieves a performance comparable to full-scale training with only around 30% annotation budget.",
    "conclusion": "We presented an active learning framework that embraces human label variation by modelling the annotator with annotator-specific classification heads, which are used to estimate the uncertainty at the individual annotator level and the group level. We first showed that a multi-head model is a better choice over a single-head model in the active learning setting, especially for uncertainty estimation. We then designed and tested five acquisition functions for the annotator-heads model on two datasets. We found that group-level entropy works generally well on both datasets and is recommended. Depending on the dataset properties, the individual-level entropy method performs differently."
  },
  {
    "sample_id": 7,
    "title": "AdaLoGN: Adaptive Logic Graph Network for Reasoning-Based Machine Reading Comprehension",
    "abstract": "Recent machine reading comprehension datasets such as ReClor and LogiQA require performing logical reasoning over text. Conventional neural models are insufficient for logical reasoning, while symbolic reasoners cannot directly apply to text. To meet the challenge, we present a neural-symbolic approach which, to predict an answer, passes messages over a graph representing logical relations between text units. It incorporates an adaptive logic graph network (AdaLoGN) which adaptively infers logical relations to extend the graph and, essentially, realizes mutual and iterative reinforcement between neural and symbolic reasoning. We also implement a novel subgraph-to-node message passing mechanism to enhance context-option interaction for answering multiple-choice questions. Our approach shows promising results on ReClor and LogiQA.",
    "introduction": "Machine reading comprehension (MRC) has drawn much research attention. Early MRC datasets are not difficult for state-of-the-art neural methods. Indeed, BERT (Devlin et al., 2019) has outperformed humans on SQuAD (Rajpurkar et al., 2016). Recent datasets become more challenging. For example, ReClor (Yu et al., 2020) and LogiQA (Liu et al., 2020) require understanding and reasoning over logical relations described in text, where neural methods showed unsatisfactory performance.\nFor instance, consider the MRC task in Figure 1. The context consists of a set of textual propositions describing logical relations between elementary discourse units (EDUs) (Mann and Thompson, 1988). For example, the first sentence describes an implication between two EDUs: “the company gets project A” implies that “product B can be put on the market on schedule”. With the help of propositional calculus, humans can formalize propositions and then apply inference rules in proposi-\nContext: If the company gets project A, product B can be\nput on the market on schedule. Product B is put on\nschedule if and only if the company’s fund can be normally\nturned over. If the company’s fund cannot be turned over normally, the development of product C cannot be carried\nout as scheduled. The fact is that the development of product C is carried out as scheduled.\nQuestion: This shows: Options:\nA. The company gets project A and product B is put on the market on schedule.\nB. The company does not get project A and product B is not put on the market on schedule.\nC. Product B is put on the market on schedule and the company’s fund is turned over normally.\nD. Product B is not put on the market on schedule, and the company’s fund turnover is extremely abnormal.\nFigure 1: An example MRC task (adapted from a task in LogiQA). Logical connectives are highlighted in italics. marks the correct answer.\ntional logic to prove the proposition in option C. However, how can machines solve such a task?\nExisting Methods and Limitations To solve it, conventional neural models are insufficient for providing the required reasoning capabilities, while symbolic reasoners cannot directly apply to unstructured text. One promising direction is to consider a neural-symbolic solution, such as the recent DAGN method (Huang et al., 2021a). It breaks down the context and each option into a set of EDUs and connects them with discourse relations as a graph. Then it performs graph neural network (GNN) based reasoning to predict an answer.\nHowever, we identify two limitations in this method. L1: Despite the graph representation, it is predominantly a neural method over discourse relations. It is debatable whether the required symbolic reasoning over logical relations (e.g., implication, negation) can be properly approximated. L2: The graph is often loosely connected and composed of long paths. Node-to-node message passing implemented in existing GNN models (Kipf and Welling, 2017; Schlichtkrull et al., 2018; Velickovic et al., 2018) is prone to provide insufficient interaction be-\n(a) Raw TLG.\n(b) Extended TLG. Dashed nodes and edges represent adaptively inferred EDUs and logical relations, respectively. Double edges represent subgraph-to-node message passing.\nFigure 2: Two TLGs for exemplifying our approach. For readability, we omit rev edges.\ntween the context and the option, which is critical to answering a multiple-choice question.\nOur Approach. While we follow the general framework of DAGN, i.e., graph construction and then graph-based reasoning, we overcome its two limitations with a novel neural-symbolic approach.\nTo address L1, Figure 3 sketches out our idea. Specifically, we propose to construct a text logic graph (TLG) representing EDUs and their logical relations as opposed to discourse relations, so we can explicitly perform symbolic reasoning to extend the TLG with inferred logical relations, as illustrated in Figure 2. The inferred relations may provide crucial connections to be used in the subsequent graph-based message passing, i.e., symbolic reasoning reinforces neural reasoning.\nFurther, while trivially computing and admitting the deductive closure may extend the TLG with irrelevant connections which would mislead message passing, we leverage signals from neural reasoning to adaptively admit relevant extensions, i.e., neural reasoning reinforces symbolic reasoning.\nMoreover, we iterate the above mutual reinforcement by restarting inference in each iteration with signals from the previous iteration to accommodate corrections to the reasoning process and allow sufficient neural-symbolic interaction.\nTo address L2, we aggregate the information in the context subgraph of TLG and employ a novel subgraph-to-node message passing mechanism to enhance the interaction from the holistic context\nFigure 3: Our main idea: mutual and iterative reinforcement between symbolic and neural reasoning.\nsubgraph to each node in the option subgraph, and vice versa, as illustrated in Figure 2b.\nWe incorporate the above two ideas into our new Adaptive Logic Graph Network (AdaLoGN). To\nsummarize, our technical contributions include\n• a novel neural-symbolic approach where neural and symbolic reasoning mutually and iteratively reinforce each other, and\n• a novel aggregation-based enhancement of message passing in graph-based neural reasoning.\nOutline. We elaborate our approach in Section 2, present experiments in Section 3, discuss related work in Section 4, and conclude in Section 5.\nOur code is available on GitHub: https:// github.com/nju-websoft/AdaLoGN.",
    "conclusion": "To meet the challenge of reasoning-based MRC, we presented a neural-symbolic approach where neural and symbolic reasoning mutually and iteratively reinforce each other via our new AdaLoGN model. We also enhanced graph-based neural reasoning with a novel subgraph-to-node message passing mechanism. Since these ideas are quite general, we believe they have great potential for a variety of applications beyond MRC, e.g., link prediction.\nError analysis has revealed some shortcomings of our approach. Currently we rely on syntactic tools to extract a raw TLG from text. We will explore other extraction methods to achieve a higher quality. We also plan to apply more inference rules and incorporate quantifiers to improve the expressivity of our symbolic reasoning."
  },
  {
    "sample_id": 8,
    "title": "Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine Translation",
    "abstract": "Neural Machine Translation (NMT) is notorious for its need for large amounts of bilingual data. An effective approach to compensate for this requirement is MultiTask Learning (MTL) to leverage different linguistic resources as a source of inductive bias. Current MTL architectures are based on the SEQ2SEQ transduction, and (partially) share different components of the models among the tasks. However, this MTL approach often suffers from task interference, and is not able to fully capture commonalities among subsets of tasks. We address this issue by extending the recurrent units with multiple blocks along with a trainable routing network. The routing network enables adaptive collaboration by dynamic sharing of blocks conditioned on the task at hand, input, and model state. Empirical evaluation of two low-resource translation tasks, English to Vietnamese and Farsi, show +1 BLEU score improvements compared to strong baselines.",
    "introduction": "Neural Machine Translation (NMT) has shown remarkable progress in recent years. However, it requires large amounts of bilingual data to learn a translation model with reasonable quality (Koehn and Knowles, 2017). This requirement can be compensated by leveraging curated monolingual linguistic resources in a multi-task learning framework. Essentially, learned knowledge from auxiliary linguistic tasks serves as inductive bias for the translation task to lead to better generalizations.\nMulti-Task Learning (MTL) is an effective approach for leveraging commonalities of related tasks to improve performance. Various recent works have attempted to improve NMT by scaffolding translation task on a single auxiliary task (Domhan and Hieber, 2017; Zhang and Zong, 2016; Dalvi et al., 2017). Recently, (Niehues and Cho, 2017) have made use of several linguistic tasks to improve NMT. Their method shares components of the SEQ2SEQ model among the tasks, e.g. encoder, decoder or the attention mechanism. However, this approach has two limitations: (i) it fully shares the components, and (ii) the shared component(s) are shared among all of the tasks. The first limitation can be addressed using deep stacked layers in encoder/decoder, and sharing the layers partially (Zaremoodi and Haffari, 2018). The second limitation causes this MTL approach to suffer from task interference or inability to leverages commonalities among a subset of tasks. Recently, (Ruder et al., 2017) tried to address this issue; however, their method is restrictive for SEQ2SEQ scenarios and does not consider the input at each time step to modulate parameter sharing.\nIn this paper, we address the task interference problem by learning how to dynamically control the amount of sharing among all tasks. We extended the recurrent units with multiple blocks along with a routing network to dynamically control sharing of blocks conditioning on the task at hand, the input, and model state. Empirical results on two low-resource translation scenarios, English to Farsi and Vietnamese, show the effectiveness of the proposed model by achieving +1 BLEU score improvement compared to strong baselines.",
    "conclusion": "We have presented an effective MTL approach to improve NMT for low-resource languages, by leveraging curated linguistic resources on the source side. We address the task interference issue in previous MTL models by extending the recurrent units with multiple blocks along with a trainable routing network. Our experimental results on low-resource English to Farsi and Vietnamese datasets, show +1 BLEU score improvements compared to strong baselines."
  },
  {
    "sample_id": 9,
    "title": "AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification",
    "abstract": "Recent work has found that few-shot sentence classification based on pre-trained Sentence Encoders (SEs) is efficient, robust, and effective. In this work, we investigate strategies for domain-specialization in the context of fewshot sentence classification with SEs. We first establish that unsupervised Domain-Adaptive Pre-Training (DAPT) of a base Pre-trained Language Model (PLM) (i.e., not an SE) substantially improves the accuracy of few-shot sentence classification by up to 8.4 points. However, applying DAPT on SEs, on the one hand, disrupts the effects of their (general-domain) Sentence Embedding Pre-Training (SEPT). On the other hand, applying general-domain SEPT on top of a domain-adapted base PLM (i.e., after DAPT) is effective but inefficient, since the computationally expensive SEPT needs to be executed on top of a DAPT-ed PLM of each domain. As a solution, we propose AdaSent, which decouples SEPT from DAPT by training a SEPT adapter on the base PLM. The adapter can be inserted into DAPT-ed PLMs from any domain. We demonstrate AdaSent’s effectiveness in extensive experiments on 17 different few-shot sentence classification datasets. AdaSent matches or surpasses the performance of full SEPT on DAPT-ed PLM, while substantially reducing the training costs. The code for AdaSent is available1.",
    "introduction": "Few-shot learning aims at training an effective model with a few labeled examples, reducing the cost of developing models for new domains and tasks. In recent work, SetFit (Tunstall et al., 2022) achieves strong performance in few-shot classification by contrastively fine-tuning (Koch et al., 2015)\n09.05.2023 Computer Science Department | UKP Lab –Iryna Gurevych | Yongxin\non paraphrase data in general domain\non task-specific unlabeled data\non few-shot labeled data\nPLM Adapter\nPLM Adapter\nFigure 1: Training diagram of AdaSent. Trainable parameters are marked in green. After Domain-Adaptive Pre-training (DAPT) on the Pre-Trained Language Model (PLM) and Sentence-Embedding Pre-Training (SEPT) with an adapter, the two parts are assembled together to perform SetFit for few-shot classification.\npre-trained sentence embeddings. Being promptfree and effective on relative small models, SetFit is much more efficient than popular promptbased methods including In-Context Learning (ICL, Brown et al., 2020) and Pattern Exploit Training (PET, Schick and Schütze, 2021), which require careful prompt engineering and large model size.\nDespite its success, SetFit fine-tunes a sentence encoder with only a few labeled samples without leveraging unlabeled data from the target-task domain, which are easy to obtain. It is well-known that Domain-Adaptive Pre-Training (DAPT)2 on a vanilla PLM with unlabeled in-domain data can significantly improve its downstream performance (Han and Eisenstein, 2019; Gururangan et al., 2020). However, it is ineffective to apply DAPT on sentence encoders, i.e. vanilla PLMs that have undergone Sentence Embedding Pre-Training (SEPT, Reimers and Gurevych, 2019) in general domain, as DAPT messes up the effects of SEPT and disrupts the model’s ability to semantically accurately embed sentences. Though DAPT before SEPT is effective in contrast (Wang et al., 2021), it is computationally inefficient as the generaldomain SEPT has to be done all over again on every domain-adapted PLM if we have more than one domain.\nTo create a domain-specialized sentence encoder for few-shot sentence classification both efficiently and effectively, we propose AdaSent, which combines DAPT and SEPT in a modular fashion. Specifically, it stores the sentence-specialization abilities – obtained via a single SEPT procedure in the general domain – into an adapter. This sentenceencoding adapter is trained once regardless of the number of domains, and can be plugged into domain-adapted PLMs from various domains to make them domain-specialized sentence encoders, on which SetFit is carried out to do downstream classification training (Figure 1). Our experiments show that AdaSent can match or surpass the inefficient \"full SEPT after DAPT\" approach’s performance on 17 sentence classification tasks from various domains. The contribution of AdaSent is two-fold:\n• AdaSent significantly improves SetFit, the previous state-of-the-art few-shot classification approach, by leveraging unlabeled taskspecific data through DAPT.\n• AdaSent resolves the conflict between DAPT and SEPT and the efficiency issue of the sequential execution of both training procedures, by combining them in a modular fashion without sacrificing the performance.",
    "conclusion": "We introduce an efficient method to obtain domainadapted sentence embeddings for few-shot classification. We found that SetFit, the previous state-of-the-art approach, can be significantly improved by introducing a simple Domain-Adaptive Pre-Training (DAPT) stage before its SentenceEmbedding Pre-Training (SEPT). However, this DAPT →SEPT approach requires the same SEPT\nprocedure to be done on each DAPT-ed PLM for every domain, resulting in computational inefficiency. We propose a novel approach, AdaSent, to address this issue by storing the SEPT knowledge in an adapter that is trained on an unadapted PLM and insertable into any DAPT-ed PLM. AdaSent matches or surpasses the performance of DAPT →SEPT,\nwhile significantly reducing the training cost of SEPT. We attribute the success of AdaSent to the generalization ability of the SEPT adapter to work with PLM parameters trained on data from different domains with a consistent MLM objective."
  },
  {
    "sample_id": 10,
    "title": "Addressing Semantic Drift in Generative Question Answering with Auxiliary Extraction",
    "abstract": "Recently, question answering (QA) based on machine reading comprehension has become popular. This work focuses on generative QA which aims to generate an abstractive answer to a given question instead of extracting an answer span from a provided passage. Generative QA often suffers from two critical problems: (1) summarizing content irrelevant to a given question, (2) drifting away from a correct answer during generation.\nIn this paper, we address these problems by a novel Rationale-Enriched Answer Generator (REAG), which incorporates an extractive mechanism into a generative model. Specifically, we add an extraction task on the encoder to obtain the rationale for an answer, which is the most relevant piece of text in an input document to a given question. Based on the extracted rationale and original input, the decoder is expected to generate an answer with high confidence. We jointly train REAG on the MS MARCO QA+NLG task and the experimental results show that REAG improves the quality and semantic accuracy of answers over baseline models.",
    "introduction": "Question Answering (QA) has come a long way from answer sentence selection, relationship QA to machine reading comprehension (MRC). Recently, QA has become an essential problem in natural language understanding and a major milestone towards human-level machine intelligence. Current mainstream approaches (Chen et al., 2017; Wang et al., 2018; Yan et al., 2018) treat MRC as a process of extracting a consecutive piece of text from a document to a given question.\nDespite the great success in extractive MRC (Wang et al., 2018; Chen et al., 2020), in real-world applications, correct answers may span different\ngenerative reading comprehension from the MARCO dataset (Nguyen et al., 2016). The text span of words in blue is the rationale extracted by REAG.\npassages or even not be literally present in the passages. Directly extracting a consecutive answer span is often inadequate. Therefore, the ability of generating an abstractive answer is needed, which requires a QA model to summarize the main content in a paragraph that is relevant to a given question.\nAnswering questions in natural language can be beneficial to a variety of QA applications, and has led to the development of smart devices such as Siri, Cortana and Alexa. However, compared with answer extraction, answer generation for reading comprehension is more challenging, and has been less explored. A major challenge in generative reading comprehension comes from out-of-control generation of abstractive answers. Although much work has been done in neural language generation (NLG), e.g., KIGN(Li et al., 2018) for summarization, out-of-control generation remains an open question for generative QA which aims to produce correct and coherent answers. Specifically, we observed that generative models often generate answers semantically drifting away from the given passage and question, known as the “semantic drift” problem. As shown in Table 1, the baseline generative model PALM (Bi et al., 2020) generates an answer that has almost contrary semantics with the gold answer. In general, a generative model often suffers from two critical problems: (1) summarizing content irrelevant to a given question, and (2) drifting away from a correct answer during generation.\nIn this paper, we address these problems by a novel Rationale-Enriched Answer Generator (REAG), which incorporates an extractive mechanism into a generative model in order to leverage relevant information to a given question in the contextual passage. Specifically, we add an extraction task on the encoder to obtain the rationale for an answer, which is the most relevant piece of text in an input document to the given question. On one hand, the introduction of the supervised extraction task enables the encoder to learn the relevance between a question and a passage; On the other hand, the extracted rationale can be further used to guide the answer generation. Based on the extracted rationale and original input, the decoder is expected to summarize content relevant to a given question and generates an answer with high confidence. Finally, we jointly train REAG on the MS MARCO QA+NLG task based on the common bottom layers. The experimental results show that REAG improves the semantic accuracy of answers over the other state-of-the-art models.",
    "conclusion": "This paper presents a novel model REAG that is designed to incorporate an extractive mechanism into a generative QA model. REAG introduces a new task on the encoder to extract rationales. Based on these rationales and original input, a rationaleenriched decoder is proposed to generate an answer with high confidence. The experimental results show that REAG significantly improves the quality and semantic accuracy of generated answers over state-of-the-art models."
  },
  {
    "sample_id": 11,
    "title": "AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples",
    "abstract": "We consider the problem of learning textual entailment models with limited supervision (5K-10K training examples), and present two complementary approaches for it. First, we propose knowledge-guided adversarial example generators for incorporating large lexical resources in entailment models via only a handful of rule templates. Second, to make the entailment model—a discriminator—more robust, we propose the first GAN-style approach for training it using a natural language example generator that iteratively adjusts based on the discriminator’s performance. We demonstrate effectiveness using two entailment datasets, where the proposed methods increase accuracy by 4.7% on SciTail and by 2.8% on a 1% training sub-sample of SNLI. Notably, even a single hand-written rule, negate, improves the accuracy on the negation examples in SNLI by 6.1%.",
    "introduction": "The impressive success of machine learning models on large natural language datasets often does not carry over to moderate training data regimes, where models often struggle with infrequently observed patterns and simple adversarial variations. A prominent example of this phenomenon is textual entailment, the fundamental task of deciding whether a premise text entails () a hypothesis text. On certain datasets, recent deep learning entailment systems (Parikh et al., 2016; Wang et al., 2017; Gong et al., 2018) have achieved close to human level performance. Nevertheless, the problem is far from solved, as evidenced by how easy it is to generate minor adversarial ex-\nTable 1: Failure examples from the SNLI dataset: negation (Top) and re-ordering (Bottom). P is premise, H is hypothesis, and S is prediction made\namples that break even the best systems. As Table 1 illustrates, a state-of-the-art neural system for this task, namely the Decomposable Attention Model (Parikh et al., 2016), fails when faced with simple linguistic phenomena such as negation, or a re-ordering of words. This is not unique to a particular model or task. Minor adversarial examples have also been found to easily break neural systems on other linguistic tasks such as reading comprehension (Jia and Liang, 2017).\nA key contributor to this brittleness is the use of specific datasets such as SNLI (Bowman et al., 2015) and SQuAD (Rajpurkar et al., 2016) to drive model development. While large and challenging, these datasets also tend to be homogeneous. E.g., SNLI was created by asking crowd-source workers to generate entailing sentences, which then tend to have limited linguistic variations and annotation artifacts (Gururangan et al., 2018). Consequently, models overfit to sufficiently repetitive patterns—and sometimes idiosyncrasies—in the datasets they are trained on. They fail to cover long-tail and rare patterns in the training distribution, or linguistic phenomena such as negation that would be obvious to a layperson.\nTo address this challenge, we propose to train textual entailment models more robustly using adversarial examples generated in two ways: (a) by incorporating knowledge from large linguistic resources, and (b) using a sequence-to-sequence neural model in a GAN-style framework.\nThe motivation stems from the following observation. While deep-learning based textual entailment models lead the pack, they generally do not incorporate intuitive rules such as negation, and ignore large-scale linguistic resources such as PPDB (Ganitkevitch et al., 2013) and WordNet (Miller, 1995). These resources could help them generalize beyond specific words observed during training. For instance, while the SNLI dataset contains the pattern two men people, it\ndoes not contain the analogous pattern two dogs animals found easily in WordNet.\nEffectively integrating simple rules or linguistic resources in a deep learning model, however, is challenging. Doing so directly by substantially adapting the model architecture (Sha et al., 2016; Chen et al., 2018) can be cumbersome and limiting. Incorporating such knowledge indirectly via modified word embeddings (Faruqui et al., 2015; Mrkˇ si´ c et al., 2016), as we show, can have little positive impact and can even be detrimental.\nOur proposed method, which is task-specific but model-independent, is inspired by dataaugmentation techniques. We generate new training examples by applying knowledge-guided rules, via only a handful of rule templates, to the original training examples. Simultaneously, we also use a sequence-to-sequence or seq2seq model for each entailment class to generate new hypotheses from a given premise, adaptively creating new adversarial examples. These can be used with any entailment model without constraining model architecture.\nWe also introduce the first approach to train a robust entailment model using a Generative Adversarial Network or GAN (Goodfellow et al., 2014) style framework. We iteratively improve both the entailment system (the discriminator) and the differentiable part of the data-augmenter (specifically the neural generator), by training the generator based on the discriminator’s performance on the generated examples. Importantly, unlike the typical use of GANs to create a strong generator, we use it as a mechanism to create a strong and robust discriminator.\nOur new entailment system, called AdvEntuRe, demonstrates that in the moderate data regime, adversarial iterative data-augmentation via only a handful of linguistic rule templates can be surprisingly powerful. Specifically, we observe 4.7% accuracy improvement on the challenging SciTail dataset (Khot et al., 2018) and a 2.8% improvement on 10K-50K training subsets of SNLI. An evaluation of our algorithm on the negation examples in the test set of SNLI reveals a 6.1% improvement from just a single rule.",
    "conclusion": "We introduced an adversarial training architecture for textual entailment. Our seq2seq and knowledge-guided example generators, trained in an end-to-end fashion, can be used to make any base entailment model more robust. The effectiveness of this approach is demonstrated by the significant improvement it achieves on both SNLI and SciTail, especially in the low to medium data regimes. Our rule-based generators can be expanded to cover more patterns and phenomena, and the seq2seq generator extended to incorporate per-example loss for adversarial training."
  },
  {
    "sample_id": 12,
    "title": "Adversarial Learning for Discourse Rhetorical Structure Parsing",
    "abstract": "Text-level discourse rhetorical structure (DRS) parsing is known to be challenging due to the notorious lack of training data. Although recent top-down DRS parsers can better leverage global document context and have achieved certain success, the performance is still far from perfect. To our knowledge, all previous DRS parsers make local decisions for either bottomup node composition or top-down split point ranking at each time step, and largely ignore DRS parsing from the global view point. Obviously, it is not sufficient to build an entire DRS tree only through these local decisions. In this work, we present our insight on evaluating the pros and cons of the entire DRS tree for global optimization. Specifically, based on recent well-performing top-down frameworks, we introduce a novel method to transform both gold standard and predicted constituency trees into tree diagrams with two color channels. After that, we learn an adversarial bot between gold and fake tree diagrams to estimate the generated DRS trees from a global perspective. We perform experiments on both RST-DT and CDTB corpora and use the original Parseval for performance evaluation. The experimental results show that our parser can substantially improve the performance when compared with previous state-of-the-art parsers.",
    "introduction": "As the main linguistic theory on discourse rhetorical structure (DRS), Rhetorical Structure Theory (RST) (Mann and Thompson, 1988) describes an article as a discourse tree (DT). As illustrated in Figure 1, each leaf node of the tree corresponds to an Elementary Discourse Unit (EDU), and relevant leaf nodes are connected by relation and nuclearity (nucleus (N) or satellite (S)) tags to form high-layer discourse units (DUs), where the\n[e 1: In fact,] [e 2: Budget indicated] [e 3: it saw some benefit] [e 4: to staying involved in these programs,] [e 5: in which renters earn frequent-flier miles] [e 6: and fliers\ncan get car-rental discounts.] wsj_2394\nAttribution (NS) List (NN)\nElaboration (NS)\nFigure 1: An example RST-style discourse tree.\nnucleus is considered more important than the satellite. Since the RST structure can well describe the organization of an article, it has been playing a central role in various down-stream tasks like summarization (Xu et al., 2020), text categorization (Ji and Smith, 2017), and so on.\nWith the release of various discourse corpora, text-level DSR parsing has been drawing more and more attention in the last decade. However, since the corpus annotation is usually time-consuming, existing DRS corpora are much limited in size. For example, the English RST-DT (Carlson et al., 2001) corpus only contains 385 WSJ articles, and the Chinese CDTB (Li et al., 2014b) corpus only contains 500 newswire articles. In this situation, previous studies usually rely on multifarious handengineered features (Hernault et al., 2010; Feng and Hirst, 2014; Ji and Eisenstein, 2014; Li et al., 2014a, 2016; Braud et al., 2017). And all these systems perform DRS parsing in a bottom-up fashion. Until recently, some researchers turn to top-down DRS parsing (Lin et al., 2019; Zhang et al., 2020; Kobayashi et al., 2020) to explore the potential capabilities of data-driven models. Nevertheless, text-level DRS parsing is still challenging and worthy of in-depth exploration.\nTheoretically, in supervised learning, annotated\n(a) (b)\nGlobal OPT\nFigure 2: Local and global optimization of DRS trees.\ndata corpora can provide neural models with specific learning objectives, and the corpus size limitation will weaken the learning of these goals. To mitigate this problem, we researchers need (i) an efficient model to better learn from the limited data and (ii) more high-quality training objectives to enhance the model learning. Existing studies on\ntext-level DRS parsing show that\n• Compared with bottom-up DRS parsers, recent top-down frameworks can better leverage global document context and have achieved promising results in text-level DRS parsing (Zhang et al., 2020; Kobayashi et al., 2020).\n• All previous studies produce their DRS parsers with local decisions made at each time step for either bottom-up node composition or top-down split point selection (Figure 2 (a)), and no global decisions are made for the entire DRS structure (Figure 2 (b)). Therefore, it is difficult for them to achieve global optimization. Although some studies (Braud et al., 2017; Mabona et al., 2019) leverage “beam-search” to traverse the solution space to find the optimal parsing route, the algorithms are time-consuming to some extent.\nConsidering the above-mentioned status quo, in this work, we study a global optimization method based on the well-performing top-down parsers. For model structure, we take the top-down parser of Zhang et al. (2020) as our baseline system and make some improvements to it. For global optimization, we first utilize a novel strategy to transform both gold standard and predicted DRS trees into tree diagrams with two color channels. After that, an LSGAN-based adversarial bot is structured between gold and fake tree diagrams as an examiner for global estimation and optimization. Experimental results on the RST-DT and CDTB corpora show that our approaches are effective.",
    "conclusion": "In this research, we explored a global optimization method based on recent top-down frameworks. Particularly, we proposed a novel strategy to transform both gold standard and predicted DRS trees into tree diagrams with two color channels. On this basis, we produced an LSGAN-based adversarial bot between gold and fake trees for global optimization. Experimental results on two popular corpora showed that our proposed adversarial approach is effective in DRS parsing and has established new state-of-the-art results for both corpora."
  },
  {
    "sample_id": 13,
    "title": "AdvPicker: Effectively Leveraging Unlabeled Data via Adversarial Discriminator for Cross-Lingual NER",
    "abstract": "Neural methods have been shown to achieve high performance in Named Entity Recognition (NER), but rely on costly high-quality labeled data for training, which is not always available across languages. While previous works have shown that unlabeled data in a target language can be used to improve crosslingual model performance, we propose a novel adversarial approach (AdvPicker) to better leverage such data and further improve results. We design an adversarial learning framework in which an encoder learns entity domain knowledge from labeled source-language data and better shared features are captured via adversarial training - where a discriminator selects less language-dependent target-language data via similarity to the source language. Experimental results on standard benchmark datasets well demonstrate that the proposed method benefits strongly from this data selection process and outperforms existing state-ofthe-art methods; without requiring any additional external resources (e.g., gazetteers or via machine translation). 1",
    "introduction": "Named entity recognition (NER) is a fundamental information extraction task, which seeks to identify named entities in text and classify them into predefined entity types (such as person, organization, location, etc.) and it is key in various downstream tasks, e.g., question answering (Moll´ a et al., 2006). Neural NER models are highly successful for languages with a large amount of quality annotated data. However, most languages don’t have enough labeled data to train a fully supervised model. This motivates research on cross-lingual transfer, which leverages labeled data from a source language (e.g., English) to address the lack of training data problem in a target language. In this paper, following Wu and Dredze (2019) and Wu et al. (2020a), we focus on zero-shot cross-lingual NER, where labeled data is not available in the target language.\nThe state-of-the-art methods for zero-shot cross-lingual NER are mainly divided into three categories: i) feature-based methods (Wu and Dredze, 2019; Wu et al., 2020b; Pfeiffer et al., 2020), which train a NER model to capture language-independent features of the labeled source-language data and then apply it to the target language; ii) translation-based methods (Mayhew et al., 2017; Xie et al., 2018), which build pseudo target-language dataset via translating from labeled source-language data and mapping entity labels; and iii) pseudo-labeling methods, which generate pseudo-labeled data for training a target-language NER model via a source-language model (Wu et al., 2020a) or annotation projection (Ni et al., 2017).\nHowever, each method has its own disadvantages. Feature-based methods only learn the knowledge in the source language, but cannot leverage any target-language information. Translation-based methods require high-quality translation resources, which are expensive to obtain. And pseudo-labeled methods assume that all pseudo-labeled data is beneficial for cross-lingual transfer learning, which is not always the case.\nTherefore, here we propose a novel approach – AdvPicker – which combines feature-based and pseudo-labeling methods, while not requiring any extra costly resources (e.g., translation models or parallel data). Furthermore, to address the described problems, we enhance the source-language NER model with unlabeled target language data via adversarial training. Unlike other pseudolabeling methods, we only leverage the languageindependent pseudo-labeled data selected by an adversarial discriminator, to alleviate overfitting the model in language-specific features of the sourcelanguage.\nSpecifically, we first train an encoder and a NER classifier on labeled source-language data to learn entity domain knowledge. Meanwhile, a language discriminator and the encoder are trained on a token-level adversarial task which enhances the ability of the encoder to capture shared features. We then apply the encoder and the NER classifier on unlabeled target-language data to generate pseudolabels and use an adversarial discriminator to select less language-specific data samples. Finally, we utilize knowledge distillation to train a targetlanguage NER model on this selected dataset.\nWe evaluate our proposed AdvPicker over 3 target languages on standard benchmark datasets. Our experimental results show that the proposed method benefits strongly from this data selection process and outperforms existing SOTA methods; without requiring any additional external resources (e.g., gazetteers or machine translation).\nOur major contributions are as follows:\n• We propose a novel approach to combine feature-based and pseudo-labeling methods via language adversarial learning for crosslingual NER;\n• We adopt an adversarial discriminator to select what language-independent data to leverage in training a cross-lingual NER model to improved performance. To the best of our knowledge, this is the first successful attempt in selecting data by adversarial discriminator for XL-NER;\n• Experiments on standard multi-lingual datasets showcase AdvPicker achieves new state-of-the-art results in cross-lingual NER.",
    "conclusion": "In this paper, we propose a novel approach to combine the feature-based method and pseudo labeling via language adversarial learning for cross-lingual NER. AdvPicker is the first successful attempt in selecting language-independent data by adversarial discriminator to cross-lingual NER. Our experimental results show that the proposed system benefits strongly from this new data selection process and outperforms existing state-of-the-art methods, even without requiring additional extra resources."
  },
  {
    "sample_id": 14,
    "title": "ALANNO: An Active Learning Annotation System for Mortals",
    "abstract": "Supervised machine learning has become the cornerstone of today’s data-driven society, increasing the need for labeled data. However, the process of acquiring labels is often expensive and tedious. One possible remedy is to use active learning (AL) – a special family of machine learning algorithms designed to reduce labeling costs. Although AL has been successful in practice, a number of practical challenges hinder its effectiveness and are often overlooked in existing AL annotation tools. To address these challenges, we developed ALANNO, an open-source annotation system for NLP tasks equipped with features to make AL effective in real-world annotation projects. ALANNO facilitates annotation management in a multi-annotator setup and supports a variety of AL methods and underlying models, which are easily configurable and extensible.",
    "introduction": "We are witnessing an ever-growing demand for data along with the rapid development of machine learning and deep learning algorithms. In particular, we need an abundance of labeled data to develop well-performing models, which is not easy to obtain. For many natural language processing (NLP) tasks, the labeling process, i.e., annotation, is often the most expensive and time-consuming part of developing machine learning models. The cognitive exertion of human annotators can affect their judgment, which further affects label validity. Consequently, this manifests in poor agreement – a proxy for label reliability, which is a prerequisite for validity (Artstein and Poesio, 2008; Paun et al., 2022). Poor label reliability and validity negatively affect the machine learning algorithm, as it is only as good as the data it consumes.\nDesigned to alleviate labeling issues and reduce annotation cost, active learning (AL; Settles, 2009) is a special family of machine learning algorithms. In contrast to the standard random selection of instances for labeling, a typical AL method iteratively queries the most informative instances for the underlying model to achieve the best possible performance with the fewest possible labels. AL has been shown to reduce annotation effort across machine learning applications, e.g., (Beluch et al., 2018; Zhang and Chen, 2002), especially in NLP, e.g., (Chen et al., 2012; Settles and Craven, 2008; Ein-Dor et al., 2020).\nDespite the demonstrated successes of AL, many challenges are involved in deploying AL in realworld scenarios (Lowell et al., 2019; Attenberg and Provost, 2011). Unfortunately, these challenges are often overlooked in both research and practice. In particular, annotation tools that support AL rarely address the problems of unbiased evaluation of AL, imbalanced data, and stopping criteria for AL. The lack of concrete solutions for these problems hinders the effectiveness of AL. Aside from the practical challenges in AL, managing annotation campaigns is often very cumbersome, especially in multi-annotator setups (when multiple annotators are assigned to a single instance). Specifically, assigning instances to multiple annotators can be painstaking, particularly if one aims to achieve balanced combinations of annotators across instances. While there are many serviceable frameworks for simulating AL in idealized scenarios, e.g., (Danka and Horvath, 2018; Tang et al., 2019; Schröder et al., 2021), there are only a few tools for running real-world AL annotation campaigns with multiple annotators, none of them explicitly addressing the practical AL challenges.\nTo facilitate the creation of high-quality NLP datasets at reduced annotation costs, we developed ALANNO (Active Learning Annotation), an opensource annotation system with AL strategies for data sampling. ALANNO’s is specifically designed to address the practical challenges of AL and facilitate the management of multi-annotator annotation projects. In particular, ALANNO guides toward more quality labels with a novel method for the balanced assignment of unlabeled instances to annotators in a multi-annotator setup. We support building gold labels by monitoring the interannotator agreement with task-specific metrics and agreement-aware weighted aggregation of labels. Equally important, ALANNO incorporates many features to address the major challenges of using AL in practice. Namely, we support guided learning (Attenberg and Provost, 2010) for mitigating data imbalance, and we ensure trustworthy evaluation of the underlying model on an unbiased test set and a stopping criterion to maximize the effectiveness of AL. As an essential practical solution, we enable a project-specific stopping criterion with a novel performance forecasting method based on Bayesian regression. By estimating the performance of the underlying model with hypothetically enlarged labeled sets, we enable practitioners to determine on the spot whether further annotation will only have diminishing returns. Lastly, ALANNO supports a wide range of state-of-the-art AL methods from the literature, allowing seamless inclusion of new models or methods.\nIn summary, our main contribution is ALANNO, an open-source AL annotation system for NLP tasks, which features (1) practical strategies for applying AL to real-world problems with a range of AL methods and (2) annotation management facilitation in a multi-annotator setup with quality control. ALANNO enables non-experts in AL to reap its benefits by accounting for key practical issues in annotation management and AL. In two case studies, we demonstrate ALANNO’s two key features – balanced data assignment and AL performance forecasting. We also provide a short video1 demonstration and release the code2 under the Apache 2.0 license. While ALANNO has been born out of several years of experience with NLP annotations for various tasks and has evolved with each new project, it remains highly configurable, allowing easy customization and extension.",
    "conclusion": "ALANNO is an open-source annotation system for natural language processing tasks powered by active learning. The system addresses the critical practical challenges of active learning in real-world annotation projects that have previously been overlooked. ALANNO enables non-experts in active learning to conduct effective annotation campaigns by supporting solutions for unbiased evaluation, stopping criterion for active learning, and class balancing. Additionally, the system facilitates annotation management in a multi-annotator setup, emphasizing label quality through agreement monitoring, agreement-aware label aggregation, and a novel method for the balanced assignment of unlabeled instances to annotators."
  },
  {
    "sample_id": 15,
    "title": "ALDi: Quantifying the Arabic Level of Dialectness of Text",
    "abstract": "Transcribed speech and user-generated text in Arabic typically contain a mixture of Modern Standard Arabic (MSA), the standardized language taught in schools, and Dialectal Arabic (DA), used in daily communications. To handle this variation, previous work in Arabic NLP has focused on Dialect Identification (DI) on the sentence or the token level. However, DI treats the task as binary, whereas we argue that Arabic speakers perceive a spectrum of dialectness, which we operationalize at the sentence level as the Arabic Level of Dialectness (ALDi), a continuous linguistic variable. We introduce the AOC-ALDi dataset (derived from the AOC dataset), containing 127,835 sentences (17% from news articles and 83% from user comments on those articles) which are manually labeled with their level of dialectness. We provide a detailed analysis of AOC-ALDi and show that a model trained on it can effectively identify levels of dialectness on a range of other corpora (including dialects and genres not included in AOC-ALDi), providing a more nuanced picture than traditional DI systems. Through case studies, we illustrate how ALDi can reveal Arabic speakers’ stylistic choices in different situations, a useful property for sociolinguistic analyses.",
    "introduction": "Arabic is spoken by more than 420 million people all over the world (Bergman and Diab, 2022), and exists in a state of Diglossia, in which two variants of the language co-exist in Arabic-speaking communities (Ferguson, 1959). Modern Standard Arabic (MSA) is the standardized variant, which is taught in schools and used in formal communications and as a common language across all Arab countries. However, many local variants of Dialectal Arabic (DA) are used for daily communication— mainly in speech and speech-like text such as social media. They can diverge from MSA and each other in phonology, morphology, syntax, and semantics\nus written with different levels of dialectness in two Arabic dialects. Words with DA features are underlined. The dialectal sentences use their preferred SVO word order, contrasted by VOS order for MSA. The low dialectness example also shows a lexical dialectal feature for the word the man (MSA r): the Egyptian word\n(r) differs from MSA in a single character, while\nthe equivalent Levantine word (Tmz) has a different origin. Both dialects allow different variants for the verb: one variant (AnWs), used in both dialects, shares\na root with the MSA variant, while the more dialectal\nvariants (AnOyhJ in Egyptian and Ann in Levantine)\ndo not.\n(Habash, 2010)—sometimes even being mutually unintelligible (Abu Farha and Magdy, 2022)—and they do not have a standard orthography.\nThese differences between MSA and DA, and the fact that speakers commonly code-switch between the two, are a major challenge for Arabic NLP systems. As a result, many systems have been designed to perform Dialect Identification (DI), often on the sentence level (Zaidan and CallisonBurch, 2011; Elfardy and Diab, 2013; Salameh et al., 2018), but also on the token level as a way of detecting code-switching points (Solorio et al., 2014; Molina et al., 2016). Both formulations take a binary view of the problem (a sentence or token is either MSA or DA), and assume all the features of DA have the same impact on the perceived “dialectness” of a sentence. We argue, however, that the level of dialectness of a sentence is a spectrum, as illustrated in Table 1. Earlier initiatives recognized the presence of such a spectrum (Habash et al., 2008; Zaidan and Callison-Burch, 2011), however, the datasets that were developed are either skewed toward more standardized documents with limited code-switching or lack information about the distribution and the quality of the levels of dialectness labels. Consequently, the Level of Dialectness has not yet been adopted as a linguistic variable that is formally recognized in analyzing Arabic text, despite being potentially useful for NLP applications.\nWe argue that the level of dialectness is an important but overlooked aspect of Arabic text which is complementary to, and more nuanced than, dialect identification. To support this claim and promote further research in the area, we:\n1. Define the Arabic Level of Dialectness (ALDi) as a continuous linguistic variable that quantifies the dialectness of a sentence (or sentence-like unit) and can enrich the analysis of Arabic text.\n2. Release AOC-ALDi1, a dataset of 127,835 Arabic comments with their ALDi labels, which is derived from the Arabic Online Commentary dataset (Zaidan and Callison-Burch, 2011). We provide the first detailed analysis of the level of dialectness labels and form canonical splits for the AOC-ALDi dataset.\n3. Propose an effective method for estimating the ALDi of sentences, that can generalize to corpora of other genres and dialects 2.\n4. Demonstrate via case studies that ALDi estimation of transcribed political speeches can highlight interesting insights that existing DI systems fail to detect.\nWe hope that our work on the Level of Dialectness variable can motivate research in this direction applied to other languages such as Swiss German where a Standard variant co-exists with nonstandardized ones.",
    "conclusion": "We presented ALDi, a linguistic variable that quantifies the level of dialectness of an Arabic sentence. We release AOC-ALDi, a dataset of Arabic comments annotated with their ALDi scores. A BERTbased regression model fine-tuned on AOC-ALDi showed superior performance compared to existing baselines that are based on lexicons and DI models. Our analysis shows that the model generalizes to various Arabic dialects. In addition, the model provides a nuanced distinction of dialectal features, which token and sentence DI models can not perform. Lastly, we presented multiple case studies demonstrating the effectiveness of ALDi in revealing new insights in Arabic text. For future work, we aim to explore the possible applications of ALDi for text analysis, especially for sociolinguistics and computational social science studies. Moreover, we aim to apply the level of dialectness work to other languages that have the same phenomena of Arabic, such as Swiss-German."
  },
  {
    "sample_id": 16,
    "title": "Algorithms for Weighted Pushdown Automata",
    "abstract": "Weighted pushdown automata (WPDAs) are at the core of many natural language processing tasks, like syntax-based statistical machine translation and transition-based dependency parsing. As most existing dynamic programming algorithms are designed for context-free grammars (CFGs), algorithms for PDAs often resort to a PDA-to-CFG conversion. In this paper, we develop novel algorithms that operate directly on WPDAs. Our algorithms are inspired by Lang’s algorithm, but use a more general definition of pushdown automaton and either reduce the space requirements by a factor of |Γ | (the size of the stack alphabet) or reduce\nthe runtime by a factor of more than |𝑄\n|\n(the\nnumber of states). When run on the same class of PDAs as Lang’s algorithm, our algorithm is\nboth more space-efficient by a factor of |Γ\n|\nand\nmore time-efficient by a factor of |𝑄\n| · |Γ |.\nhttps://github.com/rycolab/wpda",
    "introduction": "Weighted pushdown automata (WPDAs) are widespread in natural language processing (NLP), primarily in syntactic analysis. For instance, WPDAs have found use in syntax-based statistical machine translation (Allauzen et al., 2014), and many transition-based dependency parsers (Nivre, 2004; Chen and Manning, 2014; Weiss et al., 2015; Dyer et al., 2015; Andor et al., 2016; Shi et al., 2017; Ma et al., 2018; Fernández-González and Gómez-Rodríguez, 2019) are special cases of WPDAs. In addition, PDAs have been used in computational psycholinguistics as models of human sentence processing (Resnik, 1992). Despite their ubiquity, there has been relatively little research on the theory of WPDAs themselves. In some ways, WPDAs are treated as second-class citizens compared to their equivalent cousins, weighted contextfree grammars (WCFGs), for which a variety of dy-\nExtended\nPDA\nTop-down\nPDA\nBottom-up\nPDA\nTop-down NF PDA\nSimple PDA\nBottom-up NF PDA\nStringsum\n⊂\nFigure 1: Roadmap of the paper. Solid lines are new results in this paper; dashed lines are old results. We are aware of two existing methods for PDA stringsums, via CFG and via Lang’s algorithm; our algorithms are faster and/or more general than both.\nnamic programming algorithms exists (Bar-Hillel et al., 1961; Earley, 1970; Stolcke, 1995). To help fill this gap, this paper offers several new and improved algorithms for computing with WPDAs.\nFigure 1 gives an overview of most of our results. We start by defining a weighted version of the extended PDAs of Aho and Ullman (1972, p. 173) and two special cases: the standard definition (Hopcroft et al., 2006), which we call topdown, and its mirror image, which we call bottomup. Both top-down and bottom-up WPDAs have been used in NLP. Roark’s (2001) generative parser is a top-down PDA as is Dyer et al.’s (2016). Most transition-based dependency parsers, both arc-standard (Nivre, 2004; Huang et al., 2009) and arc-eager (Nivre, 2003; Zhang and Clark, 2008), are bottom-up WPDAs.\nNext, we give a normal form for WPDAs analogous to Chomsky normal form, and we derive new dynamic programming algorithms to compute the weight of a string under top-down and bottom-up WPDAs in normal form. We are only aware of one previous recognition algorithm for PDAs, that of Lang (1974), which we generalize to the weighted case and improve in the following ways:\n• On PDAs more general than those Lang considers, our algorithm is more space-efficient by a factor of |Γ | (the stack alphabet size);\n• We can speed up our algorithm to be more timeefficient by a factor of more than |𝑄\n|\n(the number\nof states), but without the space-complexity improvement;\n• On the same PDAs that Lang considers, which we call simple, our sped-up algorithm is more efficient by a factor of |Γ\n|\nin space and |𝑄 | · |Γ |\nin time.\nCompared with the pipeline of standard procedures for converting a top-down PDA to a CFG, converting to Chomsky normal form, and parsing with CKY, our top-down algorithm is faster by a factor of more than 𝑂 (|𝑄 |3 ).\nFinally, we present iterative algorithms for computing the total weight of all runs of a WPDA.",
    "conclusion": "Our study has contributed several results and algorithms whose weighted CFG analogues have long been known, but have previously been missing for weighted PDAs—a normal form analogous to Chomsky normal form and a stringsum algorithm analogous to weighted CKY. But it has also revealed some important differences, confirming that the study of weighted PDAs is of interest in its own right. Most notably, we identified two different normal forms and two corresponding stringsum algorithms (and two allsum algorithms). Since the only existing PDA stringsum algorithm we are aware of, Lang’s algorithm, is better suited to bottomup PDAs, whereas the more standard definition of PDAs is of top-down PDAs, our algorithm for top-down WPDAs fills a significant gap."
  },
  {
    "sample_id": 17,
    "title": "Aligned Dual Channel Graph Convolutional Network for Visual Question Answering",
    "abstract": "Visual question answering aims to answer the natural language question about a given image. Existing graph-based methods only focus on the relations between objects in an image and neglect the importance of the syntactic dependency relations between words in a question. To simultaneously capture the relations between objects in an image and the syntactic dependency relations between words in a question, we propose a novel dual channel graph convolutional network (DC-GCN) for better combining visual and textual advantages. The DC-GCN model consists of three parts: an I-GCN module to capture the relations between objects in an image, a Q-GCN module to capture the syntactic dependency relations between words in a question, and an attention alignment module to align image representations and question representations. Experimental results show that our model achieves comparable performance with the state-of-theart approaches.",
    "introduction": "As a form of visual Turing test, visual question answering (VQA) has drawn much attention. The goal of VQA (Antol et al., 2015; Goyal et al., 2017) is to answer a natural language question related to the contents of a given image. Attention mechanisms are served as the backbone of the previous mainstream approaches (Lu et al., 2016; Yang et al., 2016; Yu et al., 2017), however, they tend to catch only the most discriminative information, ignoring other rich complementary clues (Liu et al., 2019).\nRecent VQA studies have been exploring higher level semantic representation of images, notably using graph-based structures for better image understanding, such as scene graph generation (Xu et al., 2017; Yang et al., 2018), visual relationship detection (Yao et al., 2018), object counting (Zhang et al.,\nFigure 1: (a) The question and the ground true answer. (b) The wrong answer is predicted by a state-of-the-art model, which focuses on the highlighted region in the image. The depth of the color indicates the weights of the words in the question, where deeper color represents higher weight. The question is performed by syntactic dependency parsing. (c) The dependency parsing of the question is obtained by the universal Standford Dependencies tool (De Marneffe et al., 2014).\n2018a), and relation reasoning (Cao et al., 2018; Li et al., 2019; Cadene et al., 2019a). Representing images as graphs allows one to explicitly model interactions between two objects in an image, so as to seamlessly transfer information between graph nodes (e.g., objects in an image).\nVery recent research methods (Li et al., 2019; Cadene et al., 2019a; Yu et al., 2019) have achieved remarkable performances, but there is still a big gap between them and human. As shown in Figure 1(a), given an image of a group of persons and the corresponding question, a VQA system needs to not only recognize the objects in an image (e.g., batter, umpire and catcher), but also grasp the textual information in the question “what color is the umpire’s shirt”. However, even many competitive VQA models struggle to process them accurately, and as a result predict the incorrect answer (black) rather than the correct answer (blue), including the state-of-the-art methods.\nAlthough the relations between two objects in an image have been considered, the attention-based VQA models lack building blocks to explicitly capture the syntactic dependency relations between words in a question. As shown in Figure 1(c), these dependency relations can reflect which object is being asked (e.g., the word umpire’s modifies the word shirt) and which aspect of the object is being asked (e.g., the word color is the direct object of the word is). If a VQA model only knows the word shirt rather than the relation between words umpire’s and shirt in a question, it is difficult to distinguish which object is being asked. In fact, we do need the modified relations to discriminate the correct object from multiple similar objects. Therefore, we consider that it is necessary to explore the relations between words at linguistic level in addition to constructing the relations between objects at visual level.\nMotivated by this, we propose a dual channel graph convolutional network (DC-GCN) to simultaneously capture the relations between objects in an image and the syntactic dependency relations between words in a question. Our proposed DCGCN model consists of an Image-GCN (I-GCN) module, a Question GCN (Q-GCN) module, and an attention alignment module. The I-GCN module captures the relations between objects in an image, the Q-GCN module captures the syntactic dependency relations between words in a question, and the attention alignment module is used to align two representations of image and question. The contributions of this work are summarized as follows:\n1) We propose a dual channel graph convolutional network (DC-GCN) to simultaneously capture the visual and textual relations, and design the attention alignment module to align the multimodal representations, thus reducing the semantic gaps between vision and language.\n2) We explore how to construct the syntactic dependency relations between words at linguistic level via graph convolutional networks as well as the relations between objects at visual level.\n3) We conduct extensive experiments and ablation studies on VQA-v2 and VQA-CP-v2 datasets to examine the effectiveness of our DC-GCN model. Experimental results show that the DC-GCN model achieves competitive performance with the state-of-the-art approaches.",
    "conclusion": "In this paper, we propose a dual channel graph convolutional network to explore the relations between objects in an image and the syntactic dependency relations between words in a question. Furthermore, we explicitly construct the relations between words by dependency tree and align the image and question representations by an attention alignment module to reduce the gaps between vision and language. Extensive experiments on the VQA-v2 and VQA-CP-v2 datasets demonstrate that our model achieves comparable performance with the stateof-the-art approaches. We will explore more complicated object relation modeling in future work."
  },
  {
    "sample_id": 18,
    "title": "Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Models",
    "abstract": "Existing dialog system models require extensive human annotations and are difficult to generalize to different tasks. The recent success of large pre-trained language models has suggested the effectiveness of incorporating language priors in down-stream NLP tasks. However, how much pre-trained language models can help dialog response generation is still under exploration. In this paper, we propose a simple, general, and effective framework: Alternating Recurrent Dialog Model (ARDM)1. ARDM models each speaker separately and takes advantage of large pre-trained language models. It requires no supervision from human annotations such as belief states or dialog acts to achieve effective conversations. ARDM outperforms or is on par with the state-of-theart methods on two popular task-oriented dialog datasets: CamRest676 and MultiWOZ. Moreover, we can generalize ARDM to more challenging, non-collaborative tasks such as persuasion. In the PersuasionForGood task, ARDM is capable of generating human-like responses to persuade people to donate to a charity.",
    "introduction": "It has been a long-standing ambition for artificial intelligence researchers to create an intelligent conversational agent that can generate human-like responses. Recently, data-driven dialog models are more and more popular. However, most current state-of-the-art approaches still heavily rely on extensive human annotations such as belief states and dialog acts (Lei et al., 2018). However, dialog content can vary considerably in different dialog tasks. Having a different intent or dialog act annotation scheme for each task is costly and even impossible for tasks such as open-domain social chat. Thus, it is difficult to utilize these methods on challenging dialog tasks where dialog states and acts are difficult to annotate such as persuasion and negotiation.\nEric and Manning (2017) proposed a simple sequence-to-sequence architecture that requires no explicit annotations. The model learns to extract information from dialog history with attention and copy mechanism. However, due to the limited language modeling capability in the previous model, Sequicity (Lei et al., 2018), which uses belief states as inputs for supervision, outperforms Eric and Manning (2017)’s method significantly in the recent dialog datasets. But with the success of large pre-trained language models such as BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019), we investigate how large-scale pre-trained language models can help dialog tasks.\nPrevious sequence-to-sequence models are used to tackle documents with only one narrator. However, in dialogs, two speakers have different roles; therefore, their language model distributions are very different from each other. To address this issue, we propose ARDM, a dialog model that encodes and decodes different speaker utterances in alternating order. This structure makes the model more flexible and efficient than traditional sequence-to-sequence models in processing various dialogs. We evaluate our model on three different task-oriented dialog datasets: CamRes676, MultiWOZ, and PersuasionForGood. The first two datasets are traditional information request dialog datasets with well-defined automatic evaluation metrics on task completion. By contrast, PersuasionForGood is a new dataset that focuses on persuading people to donate to a charity. Due to the complexity of dialog content, there is no explicit dialog state defined in this task.\nWe observe that ARDM is capable of improving task-oriented dialog tasks performance over the previous state-of-the-art methods without incorporating any explicit supervision from belief states or dialog acts. Also, because of ARDM’s simplicity and generality, one can rapidly build a dialog prototype on different types of applications using only conversations without additional human annotations. We also found that ARDM works well on complex dialogs, such as persuasion. The model generates dialog responses that successfully persuade people to donate to a charity, suggesting the potential of ARDM being used in wide-scale real-world settings.",
    "conclusion": "We propose to build Alternating Recurrent Dialog Model (ARDM), a simple, general, and effective dialog method that models user and system separately with large-scale pre-trained language models. Since ARDM does not require any annotations, it generalizes to different dialog applications. Experimental results on CamRest676 and MultiWOZ suggest that ARDM outperforms or is on-par with the current state-of-the-art methods that use manual annotation information, such as belief states and dialog acts. Furthermore, we find our model’s excellent performance generalizes to more complex non-collaborative dialog settings. It can generate high-quality responses to persuade people to donate to charity. However, the easiness of training ARDM raises concerns about the misuse of the model in scenarios such as sales, harassment, or scam on a mass scale. We caution the public in deploying such systems in the real world."
  },
  {
    "sample_id": 19,
    "title": "AMR Parsing as Graph Prediction with Latent Alignment",
    "abstract": "Abstract meaning representations (AMRs) are broad-coverage sentence-level semantic representations. AMRs represent sentences as rooted labeled directed acyclic graphs. AMR parsing is challenging partly due to the lack of annotated alignments between nodes in the graphs and words in the corresponding sentences. We introduce a neural parser which treats alignments as latent variables within a joint probabilistic model of concepts, relations and alignments. As exact inference requires marginalizing over alignments and is infeasible, we use the variational autoencoding framework and a continuous relaxation of the discrete alignments. We show that joint modeling is preferable to using a pipeline of align and parse. The parser achieves the best reported results on the standard benchmark (74.4% on LDC2016E25).",
    "introduction": "Abstract meaning representations (AMRs) (Banarescu et al., 2013) are broad-coverage sentencelevel semantic representations. AMR encodes, among others, information about semantic relations, named entities, co-reference, negation and modality. The semantic representations can be regarded as rooted labeled directed acyclic graphs (see Figure 1). As AMR abstracts away from details of surface realization, it is potentially beneficial in many semantic related NLP tasks, including text summarization (Liu et al., 2015; Dohare and Karnick, 2017), machine translation (Jones et al., 2012) and question answering (Mitra and Baral, 2016).\nThe boys must not go\nboy go-02\n3 2 4\nFigure 1: An example of AMR, the dashed lines denote latent alignments, obligate-01 is the root. Numbers indicate depth-first traversal order.\nAMR parsing has recently received a lot of attention (e.g., (Flanigan et al., 2014; Artzi et al., 2015; Konstas et al., 2017)). One distinctive aspect of AMR annotation is the lack of explicit alignments between nodes in the graph (concepts) and words in the sentences. Though this arguably simplified the annotation process (Banarescu et al., 2013), it is not straightforward to produce an effective parser without relying on an alignment. Most AMR parsers (Damonte et al., 2017; Flanigan et al., 2016; Werling et al., 2015; Wang and Xue, 2017; Foland and Martin, 2017) use a pipeline where the aligner training stage precedes training a parser. The aligners are not directly informed by the AMR parsing objective and may produce alignments suboptimal for this task.\nIn this work, we demonstrate that the alignments can be treated as latent variables in a joint probabilistic model and induced in such a way as to be beneficial for AMR parsing. Intuitively, in our probabilistic model, every node in a graph is assumed to be aligned to a word in a sentence: each concept is predicted based on the corresponding RNN state. Similarly, graph edges (i.e. relations) are predicted based on representations of concepts and aligned words (see Figure 2). As alignments are latent, exact inference requires marginalizing over latent alignments, which is infeasible. Instead we use variational inference, specifically the variational autoencoding framework of Kingma and Welling (2014). Using discrete latent variables in deep learning has proven to be challenging (Mnih and Gregor, 2014; Bornschein and Bengio, 2015). We use a continuous relaxation of the alignment problem, relying on the recently introduced Gumbel-Sinkhorn construction (Mena et al., 2018). This yields a computationally-efficient approximate method for estimating our joint probabilistic model of concepts, relations and alignments.\nWe assume injective alignments from concepts to words: every node in the graph is aligned to a single word in the sentence and every word is aligned to at most one node in the graph. This is necessary for two reasons. First, it lets us treat concept identification as sequence tagging at test time. For every word we would simply predict the corresponding concept or predict NULL to signify that no concept should be generated at this position. Secondly, Gumbel-Sinkhorn can only work under this assumption. This constraint, though often appropriate, is problematic for certain AMR constructions (e.g., named entities). In order to deal with these cases, we re-categorized AMR concepts. Similar recategorization strategies have been used in previous work (Foland and Martin, 2017; Peng et al., 2017).\nThe resulting parser achieves 74.4% Smatch score on the standard test set when using LDC2016E25 training set,1 an improvement of 3.4% over the previous best result (van Noord and Bos, 2017). We also demonstrate that inducing alignments within the joint model is indeed beneficial. When, instead of inducing alignments, we follow the standard approach and produce them on preprocessing, the performance drops by 0.9% Smatch. Our main contributions can be summarized as follows:\n• we introduce a joint probabilistic model for alignment, concept and relation identification;\n• we demonstrate that a continuous relaxation can be used to effectively estimate the model;\n• the model achieves the best reported results.2",
    "conclusion": "We introduced a neural AMR parser trained by jointly modeling alignments, concepts and relations. We make such joint modeling computationally feasible by using the variational autoencoding framework and continuous relaxations. The parser achieves state-of-the-art results and ablation tests show that joint modeling is indeed beneficial.\nWe believe that the proposed approach may be extended to other parsing tasks where alignments are latent (e.g., parsing to logical form (Liang, 2016)). Another promising direction is integrating character seq2seq to substitute the copy function. This should also improve the handling of negation and rare words. Though our parsing model does not use any linearization of the graph, we relied on LSTMs and somewhat arbitrary linearization (depth-first traversal) to encode the AMR graph in our alignment model. A better alternative would be to use graph convolutional networks (Marcheggiani and Titov, 2017; Kipf and Welling, 2017): neighborhoods in the graph are likely to be more informative for predicting alignments than the neighborhoods in the graph traversal."
  },
  {
    "sample_id": 20,
    "title": "AMR Quality Rating with a Lightweight CNN",
    "abstract": "Structured semantic sentence representations such as Abstract Meaning Representations (AMRs) are potentially useful in various NLP tasks. However, the quality of automatic parses can vary greatly and jeopardizes their usefulness. This can be mitigated by models that can accurately rate AMR quality in the absence of costly gold data, allowing us to inform downstream systems about an incorporated parse’s trustworthiness or select among different candidate parses.\nIn this work, we propose to transfer the AMR graph to the domain of images. This allows us to create a simple convolutional neural network (CNN) that imitates a human judge tasked with rating graph quality. Our experiments show that the method can rate quality more accurately than strong baselines, in several quality dimensions. Moreover, the method proves to be efficient and reduces the incurred energy consumption.",
    "introduction": "The goal of sentence meaning representations is to capture the meaning of sentences in a well-defined format. One of the most prominent frameworks for achieving this is Abstract Meaning Representation (AMR) (Banarescu et al., 2013). In AMR, sentences are represented as directed acyclic and rooted graphs. An example is displayed in Figure 1, where we see three equivalent displays of an AMR that represents the meaning of the sentence “The baby is sleeping well”. In AMR, nodes are variables or concepts, while (labeled) edges express their relations. Among other phenomena, this allows AMR to capture coreference (via re-entrant structures) and semantic roles (via :arg n relation). Furthermore, AMR links sentences to KBs: e.g., predicates are mapped to PropBank (Palmer et al., 2005; Kingsbury and Palmer, 2002), while named\n:arg0 (b / baby ) :mod (w / well ))\n{ <s, instance, sleep>,\n<b, instance, baby>, <w, instance, well>,\n<s, mod, w> }\nFigure 1: Equivalent representations of the AMR for “The baby is sleeping well”.\n(p4 / possible-01\n:arg1 (d5 / destabilize-01\n:arg0 [:arg1] (c3 / country\n:quant (w2 / whole)))\n:condition (e1 / economy\n[:poss c3]\n:arg0-of (f0 / function-01\n[:pol -] )))\nFigure 2: Parse of Without a functioning economy, the whole country may destabilize with errors outlined.\nentities are linked to Wikipedia. From a logical perspective, AMR is closely related to first-order logic (FOL, see Bos (2016, 2019) for translation mechanisms).\nCurrently, AMRs are leveraged to enhance a variety of natural language understanding tasks. E.g., they have enhanced commonsense reasoning and question answering (Mitra and Baral, 2016), machine translation (Song et al., 2019), text summarization (Liao et al., 2018; Dohare et al., 2017) and paraphrasing (Issa et al., 2018). However, there is a critical issue with automatically generated AMRs (parses): they are often deficient.\nThese deficiencies can be quite severe, even when high-performance parsers are used. For example, in Figure 2, a neural parser (Lyu and Titov, 2018) conducts several errors when parsing Without a functioning economy the whole country may destabilize. E.g., it misses a negative polarity and classifies a patient argument as the agent by failing\ntheir accessibility with respect to human or computer (: ‘okay’, : ‘perhaps possible, but difficult’).\nto see that destabilize here functions as an ergative verb (parser: the country is the causer of destabilize; correct: the country is the object that is destabilized). In sum, the parse has misrepresented the sentence’s meaning.1 However, assessing such deficiencies via comparison against a gold reference (as in classical parser evaluation) is often infeasible in practice: it takes a trained annotator and appr. 10 minutes to manually create one AMR (Banarescu et al., 2013).\nTo mitigate these issues, we would like to automatically rate the quality of AMRs without the costly gold graphs. This would allow us to signal downstream task systems the incorporated graphs’ trustworthiness or select among different candidate graphs from different parsing systems. To achieve this, we propose a method that imitates a human rater, who is inspecting the graphs. We show that the method can efficiently rate the quality of the AMRs in the absence of gold graphs.\nThe remainder of the paper is structured as follows: in Section 2, we outline our idea to exploit the textual multi-line string representation of AMRs, allowing for efficient and simple AMR processing while preserving vital graph structure. In Section 2.2, we instantiate this idea in a lightweight CNN that predicts the quality of AMR graphs along multiple dimensions of interest. In our experiments (Section 3), we show that this framework is efficient and performs better than strong baselines. Our code is available at https://github.\ncom/flipz357/amr-quality-rater.",
    "conclusion": "In this work, we have developed an approach to rate the quality of AMR graphs in the absence of costly gold data. Our model imitates a human judge that is confronted, ‘on paper’, with the AMR in its native multi-line Penman format. We saw how this setup allowed efficient AMR processing with convolutions. Our experiments indicate that the method rates AMR quality more accurately and more efficiently than previous work."
  },
  {
    "sample_id": 21,
    "title": "Analysis of Automatic Annotation Suggestions for Hard Discourse-Level Tasks in Expert Domains",
    "abstract": "Many complex discourse-level tasks can aid domain experts in their work but require costly expert annotations for data creation. To speed up and ease annotations, we investigate the viability of automatically generated annotation suggestions for such tasks. As an example, we choose a task that is particularly hard for both humans and machines: the segmentation and classification of epistemic activities in diagnostic reasoning texts. We create and publish a new dataset covering two domains and carefully analyse the suggested annotations. We find that suggestions have positive effects on annotation speed and performance, while not introducing noteworthy biases. Envisioning suggestion models that improve with newly annotated texts, we contrast methods for continuous model adjustment and suggest the most effective setup for suggestions in future expert tasks.",
    "introduction": "Current deep learning methods require large amounts of training data to achieve reasonable performance. Scalable solutions to acquire labelled data use crowdsourcing (e.g., Potthast et al., 2018), gamification (Ahn, 2006), or incidental supervision (Roth, 2017). For many complex tasks in expert domains, such as law or medicine, this is, however, not an option since crowdworkers and gamers lack the necessary expertise. Annotating data manually is therefore often the only way to train a model for tasks aiding experts with their work. But the more expertise an annotation task requires, the more time- and funding-intensive it typically is, which is why many projects suffer from small corpora and deficient models.\nIn this paper, we propose and analyse an annotation setup aiming to increase the annotation speed and ease for a discourse-level sequence labelling task requiring extensive domain expertise, without sacrificing annotation quality. For the first time, we study the effects of automatically suggesting annotations to expert annotators in a task that is hard for both humans (only moderate agreement) and machine learning models (only mediocre performance) and compare the effects across different domains and suggestion models. We furthermore investigate how the performance of the models changes if they continuously learn from expert annotations.\nAs our use case, we consider the task of annotating epistemic activities in diagnostic reasoning texts, which was recently introduced by Schulz et al. (2018, 2019). The task is theoretically grounded in the learning sciences (Fischer et al., 2014) and enables innovative applications that teach diagnostic skills to university students based on automatically generated feedback about their reasoning processes. This task is an ideal choice for our investigations, since it is novel, with limited resources and experts available, and so far neural prediction models only achieve an F1 score of 0.6, while also human agreement is in a mid range around α = 0.65.\nSchulz et al. (2018) created annotated corpora of epistemic activities for 650 texts in the medicine domain (MeD) and 550 in the school teaching domain (TeD). We extend these corpora by 457 and 394 texts, respectively. As a novel component, half of the domain expert annotators receive automatically generated annotation suggestions. That is, the annotation interface features texts with (suggested) annotations rather than raw texts. Annotators can accept or reject the suggested annotations as well as add new ones, as in the standard annotation setup.\nBased on the collected data, we investigate the effects of these suggestions in terms of inter- and intra-annotator agreement, annotation time, suggestion usefulness, annotation bias, and the type of suggestion model. As our analysis reveals positive effects, we additionally investigate training suggestion models that learn continuously as new data becomes available. Such incremental models can benefit tasks with no or little available data.\nOur work is an important step towards our vision that even hard annotation tasks in expert domains, requiring extensive training and discourselevel context, can be annotated more efficiently, thus advancing applications that aid domain experts in their work. Besides epistemic activities, discourse-level expert annotation tasks concern, for example, legal documents (Nazarenko et al., 2018), psychiatric patient–therapist interactions (Mieskes and Stiegelmayr, 2018), or transcripts of police body cameras (Voigt et al., 2017).\nThe contributions of our work are: (1) We study the effects of automatically suggesting annotations to expert annotators across two domains for a hard discourse-level sequence labelling task. (2) We learn incremental suggestion models for little data scenarios through continuous adjustments of the suggestion model and discuss suitable setups. (3) We publish new diagnostic reasoning corpora for two domains annotated with epistemic activities.1",
    "conclusion": "We presented the first study of annotation suggestions for discourse-level sequence labelling requiring expert annotators, using the hard task of epistemic activity identification as an example. Our results show that even mediocre suggestion models have a positive effect in terms of agreement between annotators and annotation speed, while annotation biases are negligible.\nBased on our experiments on training suggestion models, we propose for future annotation studies that annotation suggestions can be given after having annotated only a small amount of data (in our case 70 texts), which ensures a sufficient model performance (0.5 macro-F1). Since the exact number of texts required to reach sufficient model performance depends on the task, we suggest using continuous model adjustments from the start, ensuring flexibility as to when to start giving suggestions (namely whenever sufficient performance is achieved). If computational resources are an important factor, we propose the usage of INC training with a bundle size of 30 or higher to optimise performance and training time. If model performance is more important, we recommend CUM training using a small bundle size of 10 or 20 to improve suggestions in short intervals.\nIn our model adjustment experiments, we used gold annotations. To create them on the fly, annotation aggregation methods for sequence labelling (Simpson and Gurevych, 2018) can be used.\nWe expect our work to have a large impact on future work requiring expert annotations, in particular regarding new tasks with no or little available data, for example for legal (Nazarenko et al., 2018), chemical (Guo et al., 2014), or psychiatric (Mieskes and Stiegelmayr, 2018) text processing."
  },
  {
    "sample_id": 22,
    "title": "Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation",
    "abstract": "In Neural Machine Translation (and, more generally, conditional language modeling), the generation of a target token is influenced by two types of context: the source and the prefix of the target sequence. While many attempts to understand the internal workings of NMT models have been made, none of them explicitly evaluates relative source and target contributions to a generation decision. We argue that this relative contribution can be evaluated by adopting a variant of Layerwise Relevance Propagation (LRP). Its underlying ‘conservation principle’ makes relevance propagation unique: differently from other methods, it evaluates not an abstract quantity reflecting token importance, but the proportion of each token’s influence. We extend LRP to the Transformer and conduct an analysis of NMT models which explicitly evaluates the source and target relative contributions to the generation process. We analyze changes in these contributions when conditioning on different types of prefixes, when varying the training objective or the amount of training data, and during the training process. We find that models trained with more data tend to rely on source information more and to have more sharp token contributions; the training process is non-monotonic with several stages of different nature.1",
    "introduction": "With the success of neural approaches to natural language processing, analysis of NLP models has become an important and active topic of research. In NMT, approaches to analysis include probing for linguistic structure (Belinkov et al., 2017; Conneau et al., 2018), evaluating via contrastive translation pairs (Sennrich, 2017; Burlot and Yvon, 2017; Rios Gonzales et al., 2017; Tang et al., 2018), inspecting model components, such as attention (Ghader and Monz, 2017; Voita et al., 2018; Tang et al., 2018; Raganato and Tiedemann, 2018; Voita et al., 2019) or neurons (Dalvi et al., 2019; Bau et al., 2019), among others.\nUnfortunately, although a lot of work on model analysis has been done, a question of how the NMT predictions are formed remains largely open. Namely, the generation of a target token is defined by two types of context, source and target, but there is no method which explicitly evaluates the relative contribution of source and target to a given prediction. The ability to measure this relative contribution is important for model understanding since previous work showed that NMT models often fail to effectively control information flow from source and target contexts. For example, adding context gates to dynamically control the influence of source and target leads to improvement for both RNN (Tu et al., 2017; Wang et al., 2018) and Transfomer (Li et al., 2020) models. A more popular example is a model’s tendency to generate hallucinations (fluent but inadequate translations); it is usually attributed to the inappropriately strong influence of target context. Several works observed that, when hallucinating, a model fails to properly use source: it produces a deficient attention matrix, where almost all the probability mass is concentrated on uninformative source tokens (EOS and punctuation) (Lee et al., 2018; Berard et al., 2019).\nWe argue that a natural way to estimate how the source and target contexts contribute to generation is to apply Layerwise Relevance Propagation (LRP) (Bach et al., 2015) to NMT models. LRP redistributes the information used for a prediction between all input elements keeping the total contribution constant. This ‘conservation principle’ makes relevance propagation unique: differently from other methods estimating influence of individual tokens (Alvarez-Melis and Jaakkola, 2017; He et al., 2019a; Ma et al., 2018), LRP evaluates not an abstract quantity reflecting a token importance, but the proportion of each token’s influence.\nWe extend one of the LRP variants to the Transformer and conduct the first analysis of NMT models which explicitly evaluates the source and target relative contributions to the generation process. We analyze changes in these contributions when conditioning on different types of prefixes (reference, generated by a model or random translations), when varying training objective or the amount of training data, and during the training process. We show that models suffering from exposure bias are more prone to over-relying on target history (and hence to hallucinating) than the ones where the exposure bias is mitigated. When comparing models trained with different amount of data, we find that extra training data teaches a model to rely on source information more heavily and to be more confident in the choice of important tokens. When analyzing the training process, we find that changes in training are non-monotonic and form several distinct stages (e.g., stages changing direction from decreasing influence of source to increasing).\nOur key contributions are as follows:\n• we show how to use LRP to evaluate the relative contribution of source and target to NMT predictions;\n• we analyze how the contribution of source and target changes when conditioning on different types of prefixes: reference, generated by a model or random translations;\n• by looking at the contributions when conditioning on random prefixes, we observe that models suffering from exposure bias are more prone to over-relying on target history (and hence to hallucinating);\n• we find that (i) with more data, models rely on source information more and have more sharp token contributions, (ii) the training process is non-monotonic with several distinct stages.",
    "conclusion": "We show how to use LRP to evaluate the relative contributions of source and target to NMT predictions. We illustrate the potential of this approach by analyzing changes in these contributions when conditioning on different types of prefixes (references, model predictions or random translations), when varying training objectives or the amount of training data, and during the training process. Some of our findings are: (1) models trained with more data rely on source information more and have more sharp token contributions; (2) the training process is non-monotonic with several distinct stages. These stages agree with the ones found in previous work focused on validating the lottery ticket hypothesis, which suggests future investigation of this connection. Additionally, we show that models suffering from exposure bias are more prone to over-relying on target history (and hence to hallucinating) than the ones where the exposure bias is mitigated. In future work, our methodology can be used to measure the effects of different and novel training regimes on the balance of source and target contributions."
  },
  {
    "sample_id": 23,
    "title": "Anchor-based Bilingual Word Embeddings for Low-Resource Languages",
    "abstract": "Good quality monolingual word embeddings (MWEs) can be built for languages which have large amounts of unlabeled text. MWEs can be aligned to bilingual spaces using only a few thousand word translation pairs. For low resource languages training MWEs monolingually results in MWEs of poor quality, and thus poor bilingual word embeddings (BWEs) as well. This paper proposes a new approach for building BWEs in which the vector space of the high resource source language is used as a starting point for training an embedding space for the low resource target language. By using the source vectors as anchors the vector spaces are automatically aligned during training. We experiment on English-German, English-Hiligaynon and English-Macedonian. We show that our approach results not only in improved BWEs and bilingual lexicon induction performance, but also in improved target language MWE quality as measured using monolingual word similarity.",
    "introduction": "Bilingual Word Embeddings are useful for crosslingual tasks such as cross-lingual transfer learning or machine translation. Mapping based BWE approaches rely only on a cheap bilingual signal, in the form of a seed lexicon, and monolingual data to train monolingual word embeddings (MWEs) for each language, which makes them easily applicable in low-resource scenarios (Mikolov et al., 2013b; Xing et al., 2015; Artetxe et al., 2016). It was shown that BWEs can be built using a small seed lexicon (Artetxe et al., 2017) or without any word pairs (Lample et al., 2018a; Artetxe et al., 2018) relying on the assumption of isomorphic MWE spaces. Recent approaches showed that BWEs can be built without the mapping step. Lample et al. (2018b) built FASTTEXT embeddings (Bojanowski et al., 2017) on the concatenated source and target language corpora exploiting the shared character n-grams in them. Similarly, the shared source and target language subword tokens are used as a cheap cross-lingual signal in Devlin et al. (2019); Conneau and Lample (2019). Furthermore, the advantages of mapping and jointly training the MWEs and BWEs were combined in Wang et al. (2020) for even better BWEs.\nWhile these approaches already try to minimize the amount of bilingual signal needed for cross-lingual applications, they still require a larger amount of monolingual data to train semantically rich word embeddings (Adams et al., 2017). This becomes a problem when one of the two languages does not have sufficient monolingual data available (Artetxe et al., 2020). In this case, training a good embedding space can be infeasible which means mapping based approaches are not able to build useful BWEs (Michel et al., 2020).\nIn this paper we introduce a new approach to building BWEs when one of the languages only has limited available monolingual data. Instead of using mapping or joint approaches, this paper takes the middle ground by making use of the MWEs of a resource rich language and training the low resource language embeddings on top of it. For this, a bilingual seed lexicon is used to initialize the representation of target language words by taking the pre-trained vectors of their source pairs prior to target side training, which acts as an informed starting point to shape the vector space during the process. We randomly initialize the representations of all non-lexicon target words and run Continuous Bag-of-Words (CBOW) and skipgram (SG) training procedures to generate target embeddings with both WORD2VEC (Mikolov et al., 2013a) and FASTTEXT (Bojanowski et al., 2017). Our approach ensures that the source language MWE space is intact, so that the data deficit on the target side does not result in lowered source embedding quality. The improved monolingual word embeddings for the target language outperform embeddings trained solely on monolingual data for semantic tasks such as word-similarity prediction. We study low-resource settings for EnglishGerman and English-Hiligaynon, where previous approaches have failed (Michel et al., 2020), as well as English-Macedonian.",
    "conclusion": "We proposed a novel approach to build BWEs to improve performance on language pairs with limited monolingual data on the target side. By utilizing pre-trained MWEs of resource rich languages and a seed lexicon to fix anchor points before training, a structurally similar embedding space can be learned for the low resource language which is aligned with the source representations. We evaluated our approach on the BDI task using English-German to test varying training parameters and corpora sizes, on English-\nFigure 2: Spearman’s rho correlation on monolingual word similarity across corpora sizes for German.\nMacedonian and the extremely low resource language pair English-Hiligaynon on which previous approaches failed. We showed that the performance of existing mapping approaches degrades drastically with lower monolingual data sizes, even when there are large seed lexicons available. In contrast, our proposed system outperformed previous mapping based approaches on these setups including English-Hiligaynon. On top of improved BWEs, we showed improved MWE quality as well for the target language by outperforming standard MWEs on the monolingual word similarity task showing that it is beneficial for monolingual tasks as well. We implemented our approach for both Word2Vec and FastText which we publicly release to promote reproducibility and further research.2"
  },
  {
    "sample_id": 24,
    "title": "AnnIE: An Annotation Platform for Constructing Complete Open Information Extraction Benchmark",
    "abstract": "Open Information Extraction (OIE) is the task of extracting facts from sentences in the form of relations and their corresponding arguments in schema-free manner. Intrinsic performance of OIE systems is difficult to measure due to the incompleteness of existing OIE benchmarks: ground truth extractions do not group all acceptable surface realizations of the same fact that can be extracted from a sentence. To measure performance of OIE systems more realistically, it is necessary to manually annotate complete facts (i.e., clusters of all acceptable surface realizations of the same fact) from input sentences. We propose AnnIE: an interactive annotation platform that facilitates such challenging annotation tasks and supports creation of complete fact-oriented OIE evaluation benchmarks. AnnIE is modular and flexible in order to support different use case scenarios (i.e., benchmarks covering different types of facts) and different languages. We use AnnIE to build two complete OIE benchmarks: one with verb-mediated facts and another with facts encompassing named entities. We evaluate several OIE systems on our complete benchmarks created with AnnIE. We publicly release AnnIE under non-restrictive license.1",
    "introduction": "Open Information Extraction (OIE) is the task of extracting relations and their arguments from natural language text in schema-free manner (Banko et al., 2007). Consider the input sentence \"Edmund Barton, who was born in Australia, was a judge\". Without the use of a pre-specified schema, an OIE system should extract the triples (\"Edmund Barton\"; \"was born in\"; \"Australia\") and (\"Edmund Barton\"; \"was\"; \"judge\"). The output of OIE systems is used in many downstream tasks, including open link prediction (Broscheit et al., 2020), automated knowledge base construction (Gashteovski et al., 2020), question answering (Khot et al., 2017) and text summarization (Xu and Lapata, 2021).\nIntrinsic evaluation of OIE systems is done either manually (Mausam et al., 2012; Pal et al., 2016) or with the use of evaluation benchmarks (Stanovsky and Dagan, 2016; Bhardwaj et al., 2019). While manual evaluations are usually of higher quality, they are expensive and time consuming. Automated benchmark evaluations are faster and more economic than manual OIE evaluations (Hohenecker et al., 2020), but are less reliable than human judgments of extraction correctness (Zhan and Zhao, 2020), because they are based on approximate token-level matching of system extractions against ground truth extractions. The main shortcoming of existing OIE benchmarks is their incompleteness: they do not exhaustively list all acceptable surface realizations of the same piece of information (i.e., same fact) and, because of this, resort to unreliable scoring functions based on token-level matching between system and gold extractions (Schneider et al., 2017).\nObtaining complete manual OIE annotations is, however, very difficult and time-consuming. Annotating a complete OIE benchmark requires human annotators to write all possible combinations of extractions expressing the same fact (i.e., exhaustively list all acceptable surface realizations of the same fact; see Section 3). To facilitate and speed up this process, we introduce AnnIE, a dedicated annotation tool for constructing complete fact-oriented OIE benchmark. AnnIE facilitates the annotation process by (1) highlighting the tokens of interest (e.g., for verb-mediated extractions, it highlights the verbs, which are candidates for head words of predicates); (2) providing web-based interface for annotating triples and grouping them into fact synsets, i.e., groups of informationally equivalent extractions (Section 3). To the best of our knowledge, AnnIE is the first publicly-available annotation platform for constructing OIE benchmarks.\nWe showcase AnnIE2 by creating two complete fact-based OIE benchmarks: (1) benchmark of verb-mediated facts on English, German, Chinese, Galician, Arabic and Japanese, making this gold data the first such OIE resource on languages other than English; (2) benchmark for facts associating named entities (for English only). We then benchmark several state-of-the-art OIE systems on these fact-based benchmarks and demonstrate that they are significantly less effective than indicated by existing OIE benchmarks that use token-level scoring. We hope that AnnIE motivates the creation of many more fact-based (as opposed to token-level) OIE evaluation benchmarks.",
    "conclusion": "Exhaustively annotating all acceptable OIE triples is a tedious task, but important for realistic intrinsic evaluation of OIE systems. To support annotators, we introduced AnnIE: annotation tool for constructing comprehensive evaluation benchmarks for OIE. AnnIE allows custom specification of tokens of interests (e.g., verbs) and is designed for creating fact-oriented benchmarks in which the factequivalent–yet superficially differing–extractions are grouped into fact synset. AnnIE’s lightweight architecture, easy installation and customizable components make it a practical solution for future OIE annotation."
  },
  {
    "sample_id": 25,
    "title": "An Empirical Revisiting of Linguistic Knowledge Fusion in Language Understanding Tasks",
    "abstract": "Though linguistic knowledge emerges during large-scale language model pretraining, recent work attempt to explicitly incorporate humandefined linguistic priors into task-specific finetuning. Infusing language models with syntactic or semantic knowledge from parsers has shown improvements on many language understanding tasks. To further investigate the effectiveness of structural linguistic priors, we conduct empirical study of replacing parsed graphs or trees with trivial ones (rarely carrying linguistic knowledge e.g., balanced tree) for tasks in the GLUE benchmark. Encoding with trivial graphs achieves competitive or even better performance in fully-supervised and few-shot settings. It reveals that the gains might not be significantly attributed to explicit linguistic priors but rather to more feature interactions brought by fusion layers. Hence we call for attention to using trivial graphs as necessary baselines to design advanced knowledge fusion methods in the future.",
    "introduction": "Recently large-scale pretrained language models (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020) have shown to gain linguistic knowledge from unlabeled corpus and achieve strong performance on many downstream natural language processing (NLP) tasks. Though probing analysis indicate that, to some extent, they can implicitly capture syntactic or semantic structures (Hewitt and Manning, 2019; Goldberg, 2019; Tenney et al., 2018; Hou and Sachan, 2021), whether they can further benefit from more explicit linguistic knowledge remains an open problem. Attempts have been made to inject syntactic biases into language model pretraining (Kuncoro et al., 2020; Wang et al., 2021; Xu et al., 2021b) or infuse finetuning with semantic information (Zhang et al., 2020a; Wu et al., 2021), and positive results are reported on downstream tasks.\nHowever, the concerns about the effect or viability of linguistic knowledge have been raised. On the one hand, the performance gains highly rely on the availability of human-annotated dependency parsers (Sachan et al., 2021) or oracle semantic graphs (Prange et al., 2022), which limits the real-world applications. Developing accurate semantic graph parsers is yet challenging (Oepen et al., 2019; Bai et al., 2022). On the other hand, incorporating trees induced from pretrained language models (Wu et al., 2020) can outperform the ones fused with dependency-parsed trees for aspect-level sentiment analysis (Dai et al., 2021). This discovery is in line with the similar findings of trivial trees for tree-LSTM encoders in sequence modeling tasks (Shi et al., 2018). In this work, we push the envelop and answer the following two questions. Do knowledge fusion methods in Wu et al. (2021) benefit from trivial graphs that contain no linguistic information? If that’s the case, where might the performance gains come from?\nWith the above questions, we empirically revisit the effectiveness of linguistic knowledge fusion in language understanding tasks. Motivated by Shi et al. (2018), we compare the performance between original dependency-parsed trees and balanced trees for syntax fusion, and compare the results between parsed semantic graphs and sequential graphs for semantic fusion. To our surprise, trivial graphs outperform syntactic trees or semantic graphs in full-supervised setting and achieve competitive results in few-shot setting. All the evidence again shows that the linguistic inductive bias might not be the major contributor of consistent improvements over baselines. Additional analysis gives some clues that the possible reasons are extra model parameters and feature interactions from fusion modules. This work encourages future research to add trivial graphs as necessary baselines when designing more advanced knowledge fusion methods for downstream tasks. Our experimental code is available at https://github.com/HKUST-KnowComp/ revisit-nlu-linguistic-knowledge.",
    "conclusion": "Our study demonstrates that GLUE tasks can benefit from both trivial graphs and linguistic graphs, indicating that the performance gains of previous fusion methods should not be attributed to linguistic bias entirely. We argue that comparisons merely between methods with and without knowledge fusion may not be able to capture the whole picture. For example, without baselines considering trivial graph structures, the quality of various fused knowledge may not be accurately assessed. More careful evaluations of the effectiveness claims in existing work (Sachan et al., 2021; Liu et al., 2020; Peng et al., 2021, inter alia) may be encouraged in the same spirit. In addition, tasks and evaluation benchmarks are also crucial to investigate when linguistic structures help. Our study contributes to the broader question of how to accurately evaluate models that integrate external knowledge for downstream tasks, such as world or commonsense knowledge (Xu et al., 2021a; Zhu et al., 2022)."
  },
  {
    "sample_id": 26,
    "title": "An Exploratory Analysis of Multilingual Word-Level Quality Estimation with Cross-Lingual Transformers",
    "abstract": "Most studies on word-level Quality Estimation (QE) of machine translation focus on languagespecific models. The obvious disadvantages of these approaches are the need for labelled data for each language pair and the high cost required to maintain several language-specific models. To overcome these problems, we explore different approaches to multilingual, word-level QE. We show that multilingual QE models perform on par with the current language-specific models. In the cases of zeroshot and few-shot QE, we demonstrate that it is possible to accurately predict word-level quality for any given new language pair from models trained on other language pairs. Our findings suggest that the word-level QE models based on powerful pre-trained transformers that we propose in this paper generalise well across languages, making them more useful in real-world scenarios.",
    "introduction": "Quality Estimation (QE) is the task of assessing the quality of a translation without having access to a reference translation (Specia et al., 2009). Translation quality can be estimated at different levels of granularity: word, sentence and document level (Ive et al., 2018). So far the most popular task has been sentence-level QE (Specia et al., 2020), in which QE models provide a score for each pair of source and target sentences. A more challenging task, which is currently receiving a lot of attention from the research community, is word-level quality estimation. This task provides more fine-grained information about the quality of a translation, indicating which words from the source have been incorrectly translated in the target, and whether the words inserted between these words are correct (good vs bad gaps). This information can be useful for post-editors by indicating the parts of a sentence on which they have to focus more.\nWord-level QE is generally framed as a supervised ML problem (Kepler et al., 2019; Lee, 2020) trained on data in which the correctness of translation is labelled at word-level (i.e. good, bad, gap). The training data publicly available to build wordlevel QE models is limited to very few language pairs, which makes it difficult to build QE models for many languages. From an application perspective, even for the languages with resources, it is difficult to maintain separate QE models for each language since the state-of-the-art neural QE models are large in size (Ranasinghe et al., 2020b).\nIn our paper, we address this problem by developing multilingual word-level QE models which perform competitively in different domains, MT types and language pairs. In addition, for the first time, we propose word-level QE as a zero-shot crosslingual transfer task, enabling new avenues of research in which multilingual models can be trained once and then serve a multitude of languages and domains. The main contributions of this paper are the following:\ni We introduce a simple architecture to perform\nword-level quality estimation that predicts the quality of the words in the source sentence, target sentence and the gaps in the target sentence.\nii We explore multilingual, word-level quality es-\ntimation with the proposed architecture. We show that multilingual models are competitive with bilingual models.\niii We inspect few-shot and zero-shot word-level\nquality estimation with the bilingual and multilingual models. We report how the sourcetarget direction, domain and MT type affect the predictions for a new language pair.\niv We release the code and the pre-trained models\nas part of an open-source framework1.\nFigure 1: Model Architecture",
    "conclusion": "In this paper, we explored multilingual, word-level QE with transformers. We introduced a new architecture based on transformers to perform wordlevel QE. The implementation of the architecture, which is based on Hugging Face (Wolf et al., 2020), has been integrated into the TransQuest framework (Ranasinghe et al., 2020b) which won the WMT 2020 QE task (Specia et al., 2020) on sentencelevel direct assessment (Ranasinghe et al., 2020a)2.\nIn our experiments, we observed that multilingual QE models deliver excellent results on the language pairs they were trained on. In addition, the multilingual QE models perform well in the majority of the zero-shot scenarios where the multilingual QE model is tested on an unseen language pair. Furthermore, multilingual models perform very well with few-shot learning on an unseen language pair when compared to training from scratch for that language pair, proving that multilingual QE models are effective even with a limited number of training instances. While we centered our analysis around the F1-score of the target words, these findings are consistent with the F1-score of the target gaps and the F1-score of the source words too. This suggests that we can train a single multilingual QE model on as many languages as possible and apply it on other language pairs as well. These findings can be beneficial to perform QE in low-resource languages for which the training data is scarce and when maintaining several QE models for different language pairs is arduous."
  },
  {
    "sample_id": 27,
    "title": "An Interactive Multi-Task Learning Network for End-to-End Aspect-Based Sentiment Analysis",
    "abstract": "Aspect-based sentiment analysis produces a list of aspect terms and their corresponding sentiments for a natural language sentence. This task is usually done in a pipeline manner, with aspect term extraction performed first, followed by sentiment predictions toward the extracted aspect terms. While easier to develop, such an approach does not fully exploit joint information from the two subtasks and does not use all available sources of training information that might be helpful, such as document-level labeled sentiment corpus. In this paper, we propose an interactive multi-task learning network (IMN) which is able to jointly learn multiple related tasks simultaneously at both the token level as well as the document level. Unlike conventional multi-task learning methods that rely on learning common features for the different tasks, IMN introduces a message passing architecture where information is iteratively passed to different tasks through a shared set of latent variables. Experimental results demonstrate superior performance of the proposed method against multiple baselines on three benchmark datasets.",
    "introduction": "Aspect-based sentiment analysis (ABSA) aims to determine people’s attitude towards specific aspects in a review. This is done by extracting explicit aspect mentions, referred to as aspect term extraction (AE), and detecting the sentiment orientation towards each extracted aspect term, referred to as aspect-level sentiment classification (AS). For example, in the sentence “Great food but the service is dreadful”, the aspect terms are “food” and “service”, and the sentiment orientations towards them are positive and negative respectively.\nIn previous works, AE and AS are typically treated separately and the overall task is performed in a pipeline manner, which may not fully exploit the joint information between the two tasks. Recently, two studies (Wang et al., 2018; Li et al., 2019) have shown that integrated models can achieve comparable results to pipeline methods. Both works formulate the problem as a single sequence labeling task with a unified tagging scheme1. However, in their methods, the two tasks are only linked through unified tags, while the correlation between them is not explicitly modeled. Furthermore, the methods only learn from aspect-level instances, the size of which is usually small, and do not exploit available information from other sources such as related documentlevel labeled sentiment corpora, which contain useful sentiment-related linguistic knowledge and are much easier to obtain in practice.\nIn this work, we propose an interactive multitask learning network (IMN), which solves both tasks simultaneously, enabling the interactions between both tasks to be better exploited. Furthermore, IMN allows AE and AS to be trained together with related document-level tasks, exploiting the knowledge from larger document-level corpora. IMN introduces a novel message passing mechanism that allows informative interactions between tasks. Specifically, it sends useful information from different tasks back to a shared latent representation. The information is then combined with the shared latent representation and made available to all tasks for further processing. This operation is performed iteratively, allowing the information to be modified and propagated across multiple links as the number of iterations increases. In contrast to most multi-task learning schemes which share information through learning a common feature representation, IMN not only allows shared features, but also explicitly models the interactions between tasks through the message passing mechanism, allowing different tasks to better influence each other.\nIn addition, IMN allows fined-grained tokenlevel classification tasks to be trained together with document-level classification tasks. We incorporated two document-level classification tasks – sentiment classification (DS) and domain classification (DD) – to be jointly trained with AE and AS, allowing the aspect-level tasks to benefit from document-level information. In our experiments, we show that the proposed method is able to outperform multiple pipeline and integrated baselines on three benchmark datasets2.",
    "conclusion": "We propose an interactive multi-task learning network IMN for jointly learning aspect and opinion term co-extraction, and aspect-level sentiment classification. The proposed IMN introduces a novel message passing mechanism that allows informative interactions between tasks, enabling the correlation to be better exploited. In addition, IMN is able to learn from multiple training data sources, allowing fine-grained token-level tasks to benefit from document-level labeled corpora. The proposed architecture can potentially be applied to similar tasks such as relation extraction, semantic role labeling, etc."
  },
  {
    "sample_id": 28,
    "title": "An Interpretable Neuro-Symbolic Reasoning Framework for Task-Oriented Dialogue Generation",
    "abstract": "We study the interpretability issue of taskoriented dialogue systems in this paper. Previously, most neural-based task-oriented dialogue systems employ an implicit reasoning strategy that makes the model predictions uninterpretable to humans. To obtain a transparent reasoning process, we introduce neurosymbolic to perform explicit reasoning that justifies model decisions by reasoning chains. Since deriving reasoning chains requires multihop reasoning for task-oriented dialogues, existing neuro-symbolic approaches would induce error propagation due to the one-phase design. To overcome this, we propose a twophase approach that consists of a hypothesis generator and a reasoner. We first obtain multiple hypotheses, i.e., potential operations to perform the desired task, through the hypothesis generator. Each hypothesis is then verified by the reasoner, and the valid one is selected to conduct the final prediction. The whole system is trained by exploiting raw textual dialogues without using any reasoning chain annotations. Experimental studies on two public benchmark datasets demonstrate that the proposed approach not only achieves better results, but also introduces an interpretable decision process. Code and data: https://github. com/shiquanyang/NS-Dial.",
    "introduction": "Neural task-oriented dialogue systems have enjoyed a rapid progress recently (Peng et al., 2020; Hosseini-Asl et al., 2020; Wu et al., 2020), achieving strong empirical results on various benchmark datasets such as SMD (Eric et al., 2017) and MultiWOZ (Budzianowski et al., 2018). However, most existing approaches suffer from the lack of explainability due to the black-box nature of neural networks (Doshi-Velez and Kim, 2017; Lipton, 2018; Bommasani et al., 2021), which may hurt the trustworthiness between the users and the system. For\n[Chadstone, Located_in, Leichhardt] [Cityroom, Located_in, Leichhardt]\nVerification:\n[Cityroom, Next_to, Palm_Lawn], [Palm_Lawn, Located_in, Chadstone],\nnal KB. The context entity (i.e., Leichhardt) and answer entity (i.e., Cityroom) are marked as Red and Yellow, respectively. The triple containing the context entity and answer entity is not directly stored in KB and should be derived by a reasoning chain formed by multiple KB triplets.\ninstance, in Figure 1, a user is asking for a hotel recommendation at a given location. The system performs reasoning on a knowledge base (KB) and incorporates the correct entity in the response. However, when the system fails to provide the correct entities, it would be difficult for humans to trace back the issues and debug the errors due to its intrinsic implicit reasoning nature. As a result, such system cannot be sufficiently trusted to be deployed in real-world products.\nTo achieve trustworthy dialogue reasoning, we aim to develop an interpretable KB reasoning as it’s crucial for not only providing useful information (e.g., locations in Figure 1) to users, but also essential for communicating options and selecting target entities. Without interpretability, it’s difficult for users to readily trust the reasoning process and the returned entities.\nTo tackle this challenge, we present a novel Neuro-Symbolic Dialogue framework (NS-Dial) which combines representation capacities of neural networks and explicit reasoning nature of symbolic approaches (e.g., rule-based expert systems). Existing neuro-symbolic approaches (Vedantam et al., 2019; Chen et al., 2020) mostly employ a onephase procedure where a tree-structured program composed of pre-defined human interpretable neural modules (e.g., attention and classification modules in Neural Module Networks (Andreas et al., 2016)) is generated to execute to obtain the final predictions. However, since the KB reasoning task involves a reasoning process spanning over multiple triplets in a diverse and large-scale KB, only generating and following a single program (i.e., a reasoning chain formed by KB triplets) is prone to error propagation where a mistake in one step could lead to a failure of the subsequent reasoning process and may result in sub-optimal performances.\nTo address this, we propose a two-phase procedure to alleviate the effects of error propagation by first generating and then verifying multiple hypotheses. Here, a hypothesis is in the form of a triplet containing an entity mentioned in dialogue context and an entity within KB, and their corresponding relation. The valid (i.e., correct) hypothesis is the one that contains the entity mentioned in the ground-truth response. Once we obtain multiple hypothesis candidates during the generation phase, we employ a reasoning engine for verifying those hypotheses. For instance in Figure 1, given the user query “Can you recommend me a hotel located in Leichhardt?”, in order to find the valid hypothesis, the hypothesis generator obtains multiple candidates e.g., [Cityroom, Located_in, Leichhardt] and [Gonville_Hotel, Located_in, Leichhardt]. The reasoning engine will then construct proof trees to verify them, e.g., for the first hypothesis [Cityroom, Located_in, Leichhardt], it can be verified with the following reasoning chain in the KB: [Cityroom, Next_to, Palm_Lawn] → [Palm_Lawn, Located_in, Chadstone] → [Chadstone, Located_in, Leichhardt]. The whole framework is trained end-to-end using raw dialogues and thus does not require additional intermediate labels for either the hypothesis generation or verification modules.\nTo summarize, our contributions are as follows:\n• We introduce a novel neuro-symbolic framework for interpretable KB reasoning in taskoriented dialogue systems.\n• We propose a two-phase “generating-andverifying” approach which generates multiple hypotheses and verifies them via reasoning chains to mitigate the error-propagation issue.\n• We conduct extensive experimental studies on two benchmark datasets to verify the effectiveness of our proposed model. By analyzing the generated hypotheses and the verifications, we demonstrate our model’s interpretability.",
    "conclusion": "In this paper, we propose an explicit and interpretable Neuro-Symbolic KB reasoning framework for task-oriented dialogue generation. The hypothesis generator employs a divide-and-conquer strategy to learn to generate hypotheses, and the reasoner employs a recursive strategy to learn to generate verification for the hypotheses. We evaluate our proposed framework on two public benchmark datasets including SMD and MultiWOZ 2.1. Extensive experimental results demonstrate the effectiveness of our proposed framework, as well being more interpretable."
  },
  {
    "sample_id": 29,
    "title": "An Investigation of the (In)effectiveness of Counterfactually Augmented Data",
    "abstract": "While pretrained language models achieve excellent performance on natural language understanding benchmarks, they tend to rely on spurious correlations and generalize poorly to out-of-distribution (OOD) data. Recent work has explored using counterfactuallyaugmented data (CAD)—data generated by minimally perturbing examples to flip the ground-truth label—to identify robust features that are invariant under distribution shift. However, empirical results using CAD during training for OOD generalization have been mixed. To explain this discrepancy, through a toy theoretical example and empirical analysis on two crowdsourced CAD datasets, we show that: (a) while features perturbed in CAD are indeed robust features, it may prevent the model from learning unperturbed robust features; and (b) CAD may exacerbate existing spurious correlations in the data. Our results thus show that the lack of perturbation diversity limits CAD’s effectiveness on OOD generalization, calling for innovative crowdsourcing procedures to elicit diverse perturbation of examples.",
    "introduction": "Large-scale datasets have enabled tremendous progress in natural language understanding (NLU) (Rajpurkar et al., 2016; Wang et al., 2019) with the rise of pretrained language models (Devlin et al., 2019; Peters et al., 2018). Despite this progress, there have been numerous works showing that models rely on spurious correlations in the datasets, i.e. heuristics that are effective on a specific dataset but do not hold in general (McCoy et al., 2019; Naik et al., 2018; Wang and Culotta, 2020). For example, BERT (Devlin et al., 2019) trained on MNLI (Williams et al., 2018) learns the spurious correlation between world overlap and entailment label.\nA recent promising direction is to collect counterfactually-augmented data (CAD) by asking humans to minimally edit examples to flip their ground-truth label (Kaushik et al., 2020). Figure 1 shows example edits for Natural Language Inference (NLI). Given interventions on robust features that “cause” the label to change, the model is expected to learn to disentangle the spurious and robust features.\nDespite recent attempt to explain the efficacy of CAD by analyzing the underlying causal structure of the data (Kaushik et al., 2021), empirical results on out-of-distribution (OOD) generalization using CAD are mixed. Specifically, Huang et al. (2020) show that CAD does not improve OOD generalization for NLI; Khashabi et al. (2020) find that for question answering, CAD is helpful only when it is much cheaper to create than standard examples — but Bowman et al. (2020) report that the cost is actually similar per example.\nIn this work, we take a step towards bridging this gap between what theory suggests and what we observe in practice in regards to CAD. An intuitive example to illustrate our key observation is shown in Figure 1 (a), where the verb ‘eating’ is changed to ‘drinking’ to flip the label. While there are many other words that could have been changed to flip the label, given only these two examples, the model learns to use only the verbs (e.g. using a Naive Bayes model, all other words would have zero weights). As a result, this model would fail when evaluated on examples such as those in (b) where the quantifier ‘two’ is changed to ‘three’, while a model trained on the unaugmented data may learn to use the quantifiers.\nFirst, we use a toy theoretical setting to formalize counterfactual augmentation, and demonstrate that with CAD, the model can learn to ignore the spurious features without explicitly intervening on them. However, we find that without perturbing all robust features to generate CAD, perturbations of one robust feature can prevent the model from learning other unperturbed robust features. Motivated by this, we set up an empirical analysis on\nPremise: The lady is standing next to her two children who are eating a pizza.\nOriginal Hypothesis: The two children near the lady are eating something. (Entailment)\nRevised Hypothesis: The two children near the lady are drinking something. (Contradiction)\nPremise: The lady is standing next to her two children who are eating a pizza.\nOriginal Hypothesis: The two children near the lady are eating something. (Entailment)\nRevised Hypothesis: The three children near the lady are eating something. (Contradiction)\n(a)\n(b)\nFigure 1: Illustration of counterfactual examples in natural language inference. Augmenting examples like (a) hurts performance on examples like (b) where a different robust feature has been perturbed, since the first example encourages the model to exclusively focus on the highlighted words.\ntwo crowdsourced CAD datasets collected for NLI and Question Answering (QA). In the empirical analysis, we identify the robust features by categorizing the edits into different perturbation types (Wu et al., 2021) (e.g. negating a sentence or changing the quantifiers), and show that models do not generalize well to unseen perturbation types, sometimes even performing worse than models trained on unaugmented data.\nOur analysis of the relation between perturbation types and generalization can help explain other observations such as CAD being more beneficial in the low-data regime. With increasing data size, improvement from using CAD plateaus compared to unaugmented data, suggesting that the number of perturbation types in existing CAD datasets does not keep increasing.\nAnother consequence of the lack of diversity in edits is annotation artifacts, which may produce spurious correlations similar to what happens in standard crowdsourcing procedures. While CAD is intended to debias the dataset, surprisingly, we find that crowdsourced CAD for NLI exacerbates word overlap bias (McCoy et al., 2019) and negation bias (Gururangan et al., 2018a) observed in existing benchmarks.\nIn sum, we show that while CAD can help the model ignore spurious feature, its effectiveness in current CAD datasets is limited by the set of robust features that are perturbed. Furthermore, CAD may exacerbate spurious correlations in existing benchmarks. Our results highlight the importance of increasing the diversity of counterfactual perturbations during crowdsourcing: We need to elicit more diverse edits of examples that make models more robust to the complexity of language.",
    "conclusion": "In this work, we first analyzed CAD theoretically using a linear model and showed that models do not generalize to unperturbed robust features. We then empirically demonstrated this issue in two CAD datasets, where models do not generalize well to unseen perturbation types. We also showed that CAD amplifies existing spurious correlations, pointing out another concern. Given these results, a natural question is: How can we fix these problems and make CAD more useful for OOD generalization? We discuss a few directions which we think could be helpful:\n• We can use generative models (Raffel et al., 2020; Lewis et al., 2020) to generate diverse minimal perturbations and then crowdsource labels for them (Wu et al., 2021). We can improve the diversity of the generations by masking different spans in the text to be infilled, thus covering more robust features.\n• An alternative to improving the crowdsourcing procedure is to devise better learning algorithms which mitigate the issues pointed out in this work. For example, given that we know the models do not always generalize well to unperturbed features, we can regularize the model to limit the reliance on the perturbed features.\nWe hope that this analysis spurs future work on CAD, making them more useful for OOD generalization."
  },
  {
    "sample_id": 30,
    "title": "API-Assisted Code Generation for Question Answering on Varied Table Structures",
    "abstract": "A persistent challenge to table question answering (TableQA) by generating executable programs has been adapting to varied table structures, typically requiring domain-specific logical forms. In response, this paper introduces a unified TableQA framework that: (1) provides a unified representation for structured tables as multi-index Pandas data frames, (2) uses Python as a powerful querying language, and (3) uses few-shot prompting to translate NL questions into Python programs, which are executable on Pandas data frames. Furthermore, to answer complex relational questions with extended program functionality and external knowledge, our framework allows customized APIs that Python programs can call. We experiment with four TableQA datasets that involve tables of different structures — relational, multi-table, and hierarchical matrix shapes — and achieve prominent improvements over past state-of-the-art systems. In ablation studies, we (1) show benefits from our multi-index representation and APIs over baselines that use only an LLM, and (2) demonstrate that our approach is modular and can incorporate additional APIs.",
    "introduction": "Tables are an important and widely used format for storing and retrieving information. However, since they are often constructed to present information in a visually effective way, they consequently occur in diverse formats (Chen and Cafarella, 2013; Nishida et al., 2017; Wang et al., 2021b). Thus, to effectively answer questions about all tabular information, we must consider information stored in relational (Pasupat and Liang, 2015), matrix, and hierarchically indexed tables (Cheng et al., 2022), and also address scenarios where multiple tables are presented conjointly (Yu et al., 2018).\nIn the past, works have focused on achieving strong results on datasets with particular table struc-\nWho is more likely to have cancer, the elder\ndata.loc[(‘Illness’, ‘Cancer’, ‘percent’), (‘All patients’, ‘Elders’)], data.loc[(‘Illness’, ‘Cancer’, ‘percent’), (‘All patients’, ‘Young’)]\n],\n[‘Elders’, ‘Young’]\n)\nPython program\nFigure 1: Our approach answers questions about complex tables by representing the tables as a Pandas multiindex data frame, and using a code generation LM to translate the question into a Python program that uses assistant API functions to operate on the data frame.\ntures and required tailoring logical forms to each specific type of table structure (Wang et al., 2015; Guo et al., 2019). These methods struggle to work on tables with structures outside of their original domain. For example, the neural-symbolic machine (NSM) approach designed for relational tables (Liang et al., 2017) only achieves 29.2% accuracy on the hierarchical matrix table dataset HiTab (Cheng et al., 2022) due to the ineffectiveness of its logical forms on hierarchical tables.\nOur work focuses on developing a unified framework to solve TableQA tasks across diverse table structures using Python as an intermediate language. In contrast with previous approaches that use custom logical forms to query table data (Guo et al., 2019; Cheng et al., 2022), we propose to query entries and perform operations within the widely used Python Pandas library. This allows us to leverage the strong few-shot Python generation capabilities of large code generation models, whilst saving costs by using only a few training examples. An overview of the framework is presented in Figure 1, with more details in Figure 3. Our framework consists of three main parts:\nFirst, we transform tables with varied structures into a unified multi-index data frame representation adopted from the Python Pandas library (§3.1). The multi-index objects can effectively retain the structural information in a wide range of table formats. From the implicit first step illustration in Figure 1 and more detailed one in Figure 2c, we convert tables from a hierarchical format to the multi-index representation, which will enable generated code to successfully query elements in the table.\nSecond, we translate TableQA questions into Python programs as an executable intermediate language by prompting code generation models (§3.2). Specifically, we use a few-shot paradigm where we provide the multi-index headers of the table, a few rows of the table in textual form, the question, and exemplar program annotations. As shown in Figure 1, the table and question are input to the code generation model along with the few-shot prompt, and the generated Python code uses the Pandas data.loc function to query the appropriate cells.\nThird, we introduce assistant API functions to extend the capabilities of our framework beyond Python Pandas and achieve broader coverage over various TableQA tasks. These functions enable a model-generated program to query external knowledge and perform various additional operations on the multi-index representation. In this paper, we demonstrate the usage of two simple types of API functions, Operation APIs and QA API (§3.3), and show through ablations that they increase the performance of our framework across various datasets. For example in Figure 1, the model outputs code using one of our API functions, compare_larger, to solve the question.\nWe evaluate our method across relational, hierarchical, and matrix tables, as well as multiple table paradigms using the WikiTableQuestions (WikiTQ) (Pasupat and Liang, 2015), HiTab (Cheng et al., 2022), AIT-QA (Katsis et al., 2022), and Spider (Yu et al., 2018) datasets (§4). For code generation models, we use a more capable proprietary model, CODEX (Chen et al., 2021), and an open-source model, STARCODER (Li et al., 2023).\nWe find that our framework surpasses the state-ofthe-art few-shot baselines on the HiTab, AIT, and Spider datasets, achieving absolute improvements of 24.5%, 26.2%, and 2.3%, respectively, while retaining non-trivial performance on WikiTQ. Furthermore, we perform an ablation study on the API functions we introduced and find that they bring significant improvements across datasets and models. Our framework also allows the use of existing API operations within Pandas in a modular way. For example, we show that combining Pandas’ SQL interface API with our multi-index representation and APIs improves performance by 3.7–7.6% on relational datasets.",
    "conclusion": "We introduced a framework for solving general TableQA tasks. This framework is built upon three core components: a unifying multi-index representation, Python programs as the query language, and code generation by prompting a large pretrained code model. Using CODEX as the code model, we demonstrate improvements over the state-of-the-art performance for multiple datasets, each with its unique table structures and challenges. Through ablations, we show that our proposed multi-index and API functions are both critical to the success of the framework, with the largest improvements on datasets involving hierarchical tables. We also observe improvements in performance for an opensource model, STARCODER, demonstrating the effectiveness of our approach with different code models."
  },
  {
    "sample_id": 31,
    "title": "APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning",
    "abstract": "Logical reasoning over text is an important ability that requires understanding the semantics of the text and reasoning through them to arrive at correct inferences. Prior works on pretraining language models to improve the logical reasoning ability require complex processing of training data (e.g., aligning symbolic knowledge to text), yielding task-specific solutions that are not easy to adapt to any general text corpus. In this work, we propose APOLLO, a simple adaptive pretraining approach to improve the logical reasoning skills of language models. We select a subset of Wikipedia for adaptive pretraining using a set of logical inference keywords as filter words. Further, we propose two self-supervised loss functions for training. First, we modify the masked language modeling loss to mask specific parts-of-speech words that likely require higher-order reasoning to predict them. Second, we propose a sentence-level classification loss that teaches the model to distinguish between entailment and contradiction types of sentences. The proposed pretraining paradigm is both simple and independent of task formats. We demonstrate the effectiveness of APOLLO by comparing it with prior baselines on two logical reasoning datasets. APOLLO performs comparably on ReClor and outperforms baselines on LogiQA. The code base has been made publicly available.1",
    "introduction": "Logical reasoning is an important ability of humans that helps us in making rational decisions based on known information. It is an important ability for text understanding across various downstream tasks, e.g., in open-domain question answering (Yang et al., 2018; Zhu et al., 2021), machine\nmasking (Devlin et al., 2019), a word is masked at random. Predicting these words often require more of language understanding than higher-order reasoning (e.g., predicting “would” at the 2nd [MASK] place). In selective masking, a word is masked if its POS tag is from a specific set. These candidate words are marked in the blue box in the input sentence. Filling these words requires more reasoning (e.g., to predict “more” at the 2nd [MASK] place instead of “less”, which is also grammatically valid, the model needs a better understanding of the semantics of the sentence).\nreading comprehension (MRC) (Baradaran et al., 2022), etc. Recently, there has been an increasing focus on evaluating the logical reasoning abilities of language models by using MRC tasks that specifically require a significant amount of logical reasoning to obtain the correct answer (Yu et al., 2020; Liu et al., 2021). In these datasets, the model needs to understand a given context, reason logically about a question to infer new conclusions, and then select the correct answer from a set of options. With the advent of large pre-trained language models (PLMs) in NLP (Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2020), understanding and improving the logical reasoning abilities of these models has become even more important as these are increasingly being used across a wide variety of real-world tasks.\nThere have been some recent works on improving the logical reasoning abilities of PLMs (Wang et al., 2022; Ouyang et al., 2022; Jiao et al., 2022). These works typically generate a dataset containing symbolic structures such as logical graphs from\nWikipedia Keyword-based\nDataset Selection\nImplication\nDataset\nContinued\nPretraining L = s-MLM + e-CLS\nFigure 2: Overview of APOLLO. We filter Wikipedia using specific logical keywords to create the IMPLICATION dataset. This\nis then used for continued pretraining of a model using two loss objectives: selective masked language modeling (S-MLM) loss and entailment classification (E-CLS) loss. Please refer to Section 2 and Figure 3 for more details on the data selection process and loss function designs.\ntext, logical contrast sets, etc., and then train the LM using custom loss objectives to learn logical reasoning abilities. While the performance improvements achieved by these methods are encouraging, the proposed solutions generally require complex data processing to generate the additional structural information (graphs, contrast data, etc.) required for training the model. For example, Jiao et al. (2022) constructs synthetic context-answer pairs using the entity-level graph from Wikipedia for training the model. Further, the loss functions proposed in these works are very specifically designed in accordance with their respective data augmentation technique and widely differs from the typical masked language modeling loss used for LM pretraining (Devlin et al., 2019). Additionally, some of these works usually require task-specific design choices, which are not necessarily learning generalizable logical reasoning ability that is reusable across different task formats. For example, Wang et al. (2022) parses symbolic logical structures from the training data of a specific dataset, which might not generalize to a new dataset or task. Overall, it is unclear if these highly specific inductive biases are indeed essential for improving the logical reasoning abilities in language models, or if a simpler approach is possible.\nOn the other hand, prior works (Gururangan et al., 2020) have shown that continual domainadaptive pretraining of PLMs leads to performance gains on downstream tasks. Inspired by this, we propose APOLLO, a continual pretraining-based approach to inject logical reasoning abilities in language models that requires minimal data processing and loss function modifications.\nFirstly, we present a simple way of selecting sentences for training a model that is more likely to involve logical implications. We achieve this by defining a set of logical inference keywords and selecting a subset of sentences from a large text corpus, each containing at least one of these keywords. We hypothesize that PLMs can learn logical reasoning capabilities more easily using such sentences since the premise/conclusions are explicitly stated. We note that in contrast to previous works (Gururangan et al., 2020), our method can select sentences from any general text corpus, eliminating the need for any domain-specific corpus.\nSecondly, we modify the masked language modeling (MLM) loss (Devlin et al., 2019) to selectively mask specific words in the sentence, based on their parts-of-speech tags. Prior works (Lad et al., 2022) have shown the benefit of selective masking of words on task-guided fine-tuning. We hypothesize that masking words with parts-of-speech (POS) tags that are related to higher-order reasoning (such as adverbs, conjunctions, etc.) present more challenging masked positions for the PLM to predict. For instance, in Figure 1, we observe that the words marked in blue boxes are more related to reasoning compared to the non-highlighted words that mainly involve knowledge about specific nouns or English grammar.\nLastly, we design a sentence-level classification loss to predict if the reasoning in the sentence describes an entailment in the reasoning process or a contradiction. This enables the model to better understand the differences between positive and negative implications in a sentence, thus improving logical reasoning.\nTo test APOLLO, we evaluate it on two downstream logical reasoning tasks: ReClor (Yu et al., 2020) and LogiQA (Liu et al., 2021), and compare it with other baselines. We achieve state-ofthe-art performance on LogiQA and comparable performance on ReClor. We demonstrate that our method generalizes across different model types. Further, we show that using our proposed loss functions does not induce any catastrophic forgetting (Kirkpatrick et al., 2017) of the original language modeling skills. This demonstrates that our simple, continual pretraining approach is generalizable to different datasets and enables the PLM to acquire strong logical reasoning abilities.\nOverall, compared to prior works, our proposed pretraining paradigm for APOLLO 1) Uses sentences from text corpus for training instead of complex data structures such as entity graphs, etc. 2) Uses simple learning objectives that are closer to language modeling compared to the contrastive loss. 3) Is agnostic to both task format and downstream datasets. 4) Achieves state-of-the-art performance on LogiQA.",
    "conclusion": "In this paper, we proposed APOLLO, an adaptive pre-trained language model with logical reasoning abilities. We use a subset of Wikipedia sentences for continued pretraining of the model using two self-supervised loss functions. The choice of the training dataset and loss functions are guided by the goal to include more reasoning-related sentences and training signals, respectively. Through experiments on two logical reasoning datasets and ablation studies, we demonstrate the effectiveness of our proposed approach. Overall, we show that APOLLO is a generalized solution to improving logical reasoning in language models.\nA key advantage of APOLLO is that the pretraining steps are independent of the dataset used to train the model and the downstream task format. This opens the scope to use a larger text corpus for training such as C4 (Raffel et al., 2020). Additionally, expanding on the keywords beyond positive and negative implications (for example, conditionals such as “if-then”, “either-or”, etc.) can also benefit the training pipeline."
  },
  {
    "sample_id": 32,
    "title": "AraT5: Text-to-Text Transformers for Arabic Language Generation",
    "abstract": "Transfer learning with a unified Transformer framework (T5) that converts all language problems into a text-to-text format was recently proposed as a simple and effective transfer learning approach. Although a multilingual version of the T5 model (mT5) was also introduced, it is not clear how well it can fare on non-English tasks involving diverse data. To investigate this question, we apply mT5 on a language with a wide variety of dialects–Arabic. For evaluation, we introduce a novel benchmark for ARabic language GENeration (ARGEN), covering seven important tasks. For model comparison, we pre-train three powerful Arabic T5-style models and evaluate them on ARGEN. Although pre-trained with ∼ 49% less data, our new models perform significantly better than mT5 on all ARGEN tasks (in 52 out of 59 test sets) and set several new SOTAs. Our models also establish new SOTA on the recently-proposed, large Arabic language understanding evaluation benchmark ARLUE (Abdul-Mageed et al., 2021). Our models are publicly available. We also link to individual ARGEN datasets through our public repository.1",
    "introduction": "Due to their remarkable ability to transfer knowledge from unlabeled data to downstream tasks, pre-trained Transformer-based language models have emerged as important components of modern natural language processing (NLP) systems. In particular, the unified framework that converts all text-based language problems into a text-to-text format presented through the T5 model (Raffel et al., 2019) is attractive. In addition to its simplicity, this approach is effective since it allows knowledge transfer from high-resource to low-resource tasks\nFigure 1: Our AraT5 encoder-decoder model and prompt\nsamples from four investigated tasks, namely: title generation, machine translation, question generation, and paraphrasing.\nwithout the need for changing model architecture. Unlike models such as BERT (Devlin et al., 2019), which are based on encoders only, the T5 model is an encoder-decoder that can naturally be employed for natural language generation. Although the T5 model, originally pre-trained for English, was recently extended to the multilingual setting as mT5 (Xue et al., 2020), it is not clear how suited it is to individual languages (and varieties of these languages). In addition, systematic issues have been discovered in multilingual corpora on which language models have been trained (Kreutzer et al., 2021). In absence of comparisons with monolingual pre-trained language models that serve different non-English contexts, it remains unknown how multilingual models really fare against languagespecific models.\nIn this work, we offer the first comparison of the mT5 model to similar encoder-decoder models dedicated to Arabic. We choose Arabic as our context due to its large set of diverse varieties as well as its wide use on social media. Our work aims at uncovering the extent to which mT5 can serve Arabic’s different varieties. Our work also meets an existing need for pre-trained Transformer-based sequenceto-sequence models. In other words, while several BERT-based models have been pre-trained for Arabic (Antoun et al., 2020; Abdul-Mageed et al., 2021; Inoue et al., 2021), no such attempts have been made to create sequence-to-sequence models that we know of. Another motivation for our work is absence of an evaluation benchmark for Arabic language generation tasks. Apart from machine translation where researchers are starting to propose benchmarks such as AraBench (Sajjad et al., 2020), there are no benchmarks that can be used to methodically measure Arabic natural language generation performance.\nOur main contributions are as follows: (1) We introduce three powerful variants of the text-to-text transformer (T5) model dedicated to Modern Standard Arabic (MSA) and a diverse set of Arabic dialects. We include in our vocabulary 11 languages other than Arabic (e.g., English, French, German, Russian), which also allows us to evaluate our models under zero-shot pre-training conditions involving these languages. (2) We propose a novel unified benchmark for ARabic natural language GEeneration (ARGEN) composed of seven tasks: machine translation, code-switched text translation, summarization, news title generation, question generation, paraphrasing, and transliteration. ARGEN is collected from a total of 19 datasets, including 9 new datasets proposed in this work. (3) To show the utility of our new models, we evaluate them on ARGEN under both full and zero-shot pre-training conditions. Our models set new SOTA on the majority of datasets in all seven tasks. (4) Although the main focus of our work is language generation, we also show the effectiveness of our models on Arabic language understanding by fine-tuning our new models on a large, recently proposed Arabic language understanding benchmark. Again, our models establish new SOTA on the majority of language understanding tasks.\nThe rest of the paper is organized as follows: Section 2 describes our Arabic pre-tained models. In Section 3, we introduce ARGEN, our new natural language generation benchmark. We evaluate our models on ARGEN in Section 4. Section 5 is an analysis and discussion of our results. In Section 6, we provide an overview of related work. We conclude in Section 7. We now introduce our new pre-trained models.",
    "conclusion": "We introduced three powerful Arabic-specific textto-text Transformer models trained on large MSA and/or Arabic dialectal data. We also introduced ARGEN, a unified benchmark for Arabic Natural Language generation evaluation composed of seven tasks collected from a total of 19 datasets. Our models outperform mT5 on all ARGEN tasks (52 out of 59 test sets, i.e., 88.14%). This is true even for MT involving four foreign languages from which the models have seen marginal or no pretraining data (i.e., zero- and few-shot pre-training). Our models also set new SOTA on the large Arabic language understanding evaluation benchmark ARLUE. Our models involve vocabulary from 11 languages other than Arabic, and hence can easily be further pre-trained/fine-tuned in these languages. Our models are publicly available, and ARGEN datasets are accessible from our repository."
  },
  {
    "sample_id": 33,
    "title": "ARHNet - Leveraging Community Interaction for Detection of Religious Hate Speech in Arabic",
    "abstract": "The rapid widespread of social media has led to some undesirable consequences like the rapid increase of hateful content and offensive language. Religious Hate Speech, in particular, often leads to unrest and sometimes aggravates to violence against people on the basis of their religious affiliations. The richness of the Arabic morphology and the limited available resources makes this task especially challenging. The current state-of-theart approaches to detect hate speech in Arabic rely entirely on textual (lexical and semantic) cues. Our proposed methodology contends that leveraging Community-Interaction can better help us profile hate speech content on social media. Our proposed ARHNet (Arabic Religious Hate Speech Net) model incorporates both Arabic Word Embeddings and Social Network Graphs for the detection of religious hate speech.",
    "introduction": "Hate speech was a major tool employed to promote slavery in Colonial America, to aggravate tensions in Bosnia and in the rise of the Third Reich. The aim of such speech is to ridicule victims, to humiliate them and represent their grievances as less serious (Gelashvili, 2018). The relationship between religion and hate speech is complex and has been central to recent discussions of hate speech directed at religious people, especially members of religious minorities (Bonotti, 2017). This makes it important to develop automated tools to detect messages that use inflammatory sectarian language to promote hatred and violence against people.\nOur work extends on the work done by (Albadi et al., 2018) in terms of exploring the merits of introducing community interaction as a feature in the detection of religious hate speech in Arabic. Most previous work in the area of hate speech detection has targeted mainly English content (Davidson et al., 2017) (Djuric et al., 2015) (Badjatiya et al., 2017). Author profiling using community graphs has been explored by (Mishra et al., 2018) for abuse detection on Twitter. We propose a novel Cyber Hate Detection approach using multiple twitter graphs and traditional word embeddings.\nSocial network graphs are increasingly being used as a powerful tool for NLP applications (Mahata et al., 2018; Shah et al., 2016b), leading to substantial improvement in performance for tasks like text categorization, sentiment analysis, and author attribute identification ((Hovy, 2015); (Yang and Eisenstein, 2015); (Yang et al., 2016). The idea of using this type of information is best explained by the concept of homophily, i.e., the phenomenon that people, both in real life as well as on the Internet, tend to associate more with those who appear similar. Here, similarity can be defined based on various parameters like location, age, language, etc. The basic idea behind leveraging community interaction is that if we have information about members of a community defined by some similarity measure, then we can infer information about a person based on which community they belong to. For our study, knowing that members of a particular community are prone to proliferating religious hate speech content, and knowing that the user is connected to this community, we can use this information beyond linguistic cues and more accurately predict the use of hateful/nonhateful language. Our work seeks to address two main questions: • Is one community more prone to spreading hateful content than the other?\n• Can such information be effectively leveraged to improve the performance of the current state of the art in the detection of religious hate speech within Arabic speaking users?\nIn this paper, we do an in-depth analysis of how adding community features may enhance the performance of classification models that detect religious hate speech in Arabic.",
    "conclusion": "In this paper, we explored the effectiveness of community-interaction information about authors for the purpose of categorizing religious hate speech in the Arabic Twittersphere and build upon existing work in the linguistic aspects of social media (Shah et al., 2016c,a; Mahata et al., 2015). Working with a dataset of 3950 tweets annotated for Hate and Non-Hate, we first comprehensively replicated three established and currently bestperforming hate speech detection methods based on character n-grams and GRUs as our baselines. We then constructed a graph of all the authors of tweets in our dataset and extracted communitybased information in the form of dense lowdimensional embeddings for each of them using Node2Vec. We showed that the inclusion of community graph embeddings significantly improves system performance over the baselines and advances the state of the art in this task. Users prone to proliferate hate do tend to form social groups online, and this stresses the importance of utilizing community-based information for automatic religious hate speech detection."
  },
  {
    "sample_id": 34,
    "title": "Aspect Extraction Using Coreference Resolution and Unsupervised Filtering",
    "abstract": "Aspect extraction is a widely researched field of natural language processing in which aspects are identified from the text as a means for information. For example, in aspect-based sentiment analysis (ABSA), aspects need to be first identified. Previous studies have introduced various approaches to increasing accuracy, although leaving room for further improvement. In a practical situation where the examined dataset is lacking labels, to fine-tune the process a novel unsupervised approach is proposed, combining a lexical rule-based approach with coreference resolution. The model increases accuracy through the recognition and removal of coreferring aspects. Experimental evaluations are performed on two benchmark datasets, demonstrating the greater performance of our approach to extracting coherent aspects through outperforming the baseline approaches.",
    "introduction": "Aspect-based sentiment analysis (ABSA) is a task involving the identification of key terms (words and phrases) that refer to important parts, features, attributes, or properties of a targeted product, object or service, along with associated sentimental emotions, opinions or evaluations. What started out as a simple document-level classification task (Hu and Liu, 2004), i.e., using reviews to differentiate positive from negative, has evolved into a heavily researched field of natural language processing and information retrieval (Godbole et al., 2007). As social presence becomes more standard, the need for detecting opinions in comments or reviews becomes more present. Due to the multi-perspective opinion-oriented nature of the comments, this task will require sentence or phrase-level aspect extraction. The system must be able to locate the expressions of aspects on a sentence-level, for example in the following examples, the aspects and their associated sentiment are clear; seaweed and chewy, and coronavirus and hate, terrible respectively: “The seaweed was too chewy”, and “Hate it, the coronavirus is terrible”.\nThe existing approaches for extracting aspect are in two branches: supervised and unsupervised. Supervised approaches often formulate ATE as a token-level sequence labeling problem, achieving better accuracy than unsupervised methods in general (Li and Lam, 2017; Li et al., 2018; Zhou et al., 2019; Ma et al., 2019). However, these approaches generally require annotated data and can run into domain adaptation issues. Moreover, in reality human labelling is a time-consuming and laborious work, motivating the unsupervised approach. Topic model based approaches were proposed for this purpose (Mukherjee and Liu, 2012). These approaches model the text corpus as a mixture of opinion topics, treating the task as a problem in topic coreference resolution. This process labels aspects relating to the extracted opinion topic while dealing with coreferring aspects (Stoyanov and Cardie, 2008; Brody and Elhadad, 2010; Poria et al., 2016). Although the aspects interpreted by these models express a corpus well, they aren’t coherent; individual aspects are of low quality, consisting of irrelevant or distantly-related concepts. The work in (Hu and Liu, 2004) first proposed a manually-defined rulebased approach to extract product features through observing frequent nouns and noun chunks. This approach sparked the development of numerous approaches based on frequent term mining and dependency parsing (Zhuang et al., 2006). Later, the work in (Qiu et al., 2011) proposed a unique approach to learn syntactic relations using dependency trees. Although innovative, the rule-based models heavily relied on predefined rules which only worked well when the aspect terms are confined to a small group of nouns.\nIn our project, we target the issue of conducting aspect-based sentiment analysis when there is the lack of labelled data, which presents a practical challenge. To this end, as a starting point, we propose an unsupervised approach for aspect extraction on the data corpus, forming the foundation of our following works. We particularly seek an advanced rule-based approach due to its efficiency and independence from manual efforts. We first extract candidate aspects using dependency parsing and coreference resolution. A careful selection process is then applied using unsupervised techniques; inspecting the candidates for duplicate and incorrect aspects. Specifically, syntactic rules are applied on the part of speech (POS) and dependency information of a document to convert it into a candidate aspect list. This candidate list is then reduced to a final list by first applying coreference resolution, removing candidates that refer to an already existing aspect to avoid duplicity. Finally, an unsupervised filtering technique is applied on the candidates, calculating the cosine similarity of an aspect’s word embedding to its neighbours and removing those that don’t meet an optimal threshold. Overall, our proposed approach consists of several stages where in each the candidate list is reduced. This allows our model to overcome the small noun group restraint by first taking in a broad list of noun phrases. A clustering process is applied to complete the categorisation task to an extent.",
    "conclusion": "We have proposed and implemented an approach to aspect extraction utilising an unsupervised rulebased coreference resolution model. The basis of this approach is to apply a rule-based checking system on noun chunks extracted from the text. What started as a simple model has proven itself to be a valid approach, outperforming previous similarly unsupervised approaches. Additionally, the clusters produced on each aspect’s word vector are coherent to a satisfactory level, reflecting the eligibility of our baseline model.\nTo improve the purging process, word vectors can be learned for a much larger vocabulary. If this can be implemented, foreign dish words such as rasamalai won’t be incorrectly ruled out as aspects due to them not being in the vocabulary. Slang interpretations such as rule in “the food options rule!” can be investigated by using a similar technique to the stop word list. We will also involve machine learning techniques to improve the rule-based approach. Through training our model with the output of rules as an indicator feature for a discriminative learning model, we can expect that our rules are fine-tuned and adaptable to different corpora. Furthermore, to avoid mistakes in clustering where similar words included in different categories are graphed in similar locations, additional learning can be acquired by our model. Extra checks can be performed once a certain black-listed word is found in an aspect, and word embeddings can be trained further. In addition, we will perform sentiment analysis on the extracted aspects and investigate whether public sentiment can reflect the real-estate prices."
  },
  {
    "sample_id": 35,
    "title": "ASQA: Factoid Questions Meet Long-Form Answers",
    "abstract": "An abundance of datasets and availability of reliable evaluation metrics have resulted in strong progress in factoid question answering (QA). This progress, however, does not easily transfer to the task of long-form QA, where the goal is to answer questions that require in-depth explanations. The hurdles include (i) a lack of high-quality data, and (ii) the absence of a well-defined notion of the answer’s quality. In this work, we address these problems by (i) releasing a novel dataset and a task that we call ASQA (Answer Summaries for Questions which are Ambiguous); and (ii) proposing a reliable metric for measuring performance on ASQA. Our task focuses on factoid questions that are ambiguous, that is, have different correct answers depending on interpretation. Answers to ambiguous questions should synthesize factual information from multiple sources into a long-form summary that resolves the ambiguity. In contrast to existing long-form QA tasks (such as ELI5), ASQA admits a clear notion of correctness: a user faced with a good summary should be able to answer different interpretations of the original ambiguous question. We use this notion of correctness to define an automated metric of performance for ASQA. Our analysis demonstrates an agreement between this metric and human judgments, and reveals a considerable gap between human performance and strong baselines.",
    "introduction": "In the last few years, the factoid question answering (QA) task—extracting short answers to factoid questions—has witnessed significant progress (Lee et al., 2019; Guu et al., 2020; Karpukhin et al., 2020; Lewis et al., 2020; Izacard and Grave, 2021). The progress was achieved in large part thanks to (i) the availability of high-quality datasets (Voorhees and Tice, 2000; Joshi et al., 2017; Yang et al., 2018; Abujabal et al., 2019; Kwiatkowski et al., 2019), and (ii) a well-defined notion of correctness. A key challenge for ongoing research now lies in longform question answering where the goal is to generate detailed explanations in response to questions that require elaborate and in-depth answers.\nThere is much less data available for the task of long-form QA. One of the primary data sources is the ELI5 dataset (Fan et al., 2019) that pairs open-ended questions with paragraph-long answers written by users of the “Explain Like I’m Five” Reddit forum. However, questions in ELI5 are very general (e.g., “How can different animals perceive different colors?”) and can be answered in myriad different ways, making it hard to define objective criteria for a good answer. As a result, Krishna et al. (2021) identify several hurdles in using this data towards meaningful modeling progress, including a lack of reliable evaluation metrics.\nIn this work, we address the lack of data sources and unreliability of evaluations by constructing a long-form QA dataset for factoid questions. Our paper is motivated by the work of Min et al. (2020) who observe that more than half of the factoid questions that occur naturally are ambiguous. For example, a seemingly simple question: “Who was the ruler of France in 1830?\" is ambiguous because there were two rulers of France in 1830. Min et al. (2020) collected the AMBIGQA dataset that connects ambiguous factoid questions with disambiguations: pairs of disambiguated questions and unique short answers to these questions (see example on the right side of Figure 1).\nWe note, however, that ambiguous questions often arise when a user lacks background knowledge about why there might be multiple answers to their question, and how those answers relate to each other. Thus, the list of disambiguations may not be satisfactory for the user. For example, the fact that in 1830 the ruler of France changed due to the revolution is highly salient but is not captured in\nInput Question: Who was the ruler of France in 1830?\nFigure 1: The input questions in ASQA are sourced from AMBIGQA. Long-form answers must be sufficient to answer disambiguated questions from AMBIGQA (short answers are marked in blue and green), and should introduce additional knowledge from Wikipedia (highlighted in red) to resolve ambiguity and clarify the relationship between different short answers. The DR score we propose combines ROUGE and Disambiguation-accuracy (Disambig-Acc) metrics, overcoming the issues with long-form QA evaluation outlined by Krishna et al. (2021).\nthe AMBIGQA disambiguations.\nIn this paper, we argue the importance of generating long-form answers to ambiguous factoid questions. In that, we present ASQA (Answer Summaries for Questions which are Ambiguous)— a novel dataset that pairs each ambiguous question from AMBIGQA with a crowdsourced long-form answer.1 The answers we collect aim to (i) explain the source of ambiguity in the question, and (ii) connect all the valid short answers into a coherent passage. An example ASQA instance is shown in Figure 1.\nThe main feature of ASQA is a combination of (i) a well-defined notion of correctness pertinent to factoid QA and (ii) the complexity of long-form QA. First, observe that a good answer to an ambiguous question should be sufficient for the user to answer different interpretations of the question. This observation induces a notion of correctness that is conceptually similar to the conventional accuracy in factoid QA. Second, to answer an ambiguous question, a system needs to retrieve a diverse set of documents that talk about different interpretations of the question and synthesize this information into a coherent summary. Thus, the key challenges of long-form QA—precise retrieval and high-quality summarization—are present in ASQA.\nContributions Overall, our work makes several contributions:\n• First, we carefully develop a crowdsourcing pipeline and collect ASQA—a dataset of highquality long-form answers to 6,316 ambiguous factoid questions.\n• Second, we design principled evaluation procedures for ASQA: (i) we propose a novel automated evaluation metric (DR) that combines the correctness aspect of factoid QA and the fluency aspect of long-form QA; (ii) we develop and release a convenient interface for human evaluations; (iii) we conduct a small-scale human study that shows a high agreement between our automated metric DR and human judgments.\n• Third, we establish strong baselines for our task by combining joint passage retrieval (Min et al., 2021) and T5-large (Raffel et al., 2019). Our extensive evaluations demonstrate that there is a large gap between the baselines and human performance. Additionally, we highlight areas of improvement for future research on ASQA.",
    "conclusion": "In contrast to existing datasets for long-form QA, ASQA admits a clear notion of correctness that we use to define an overall metric of performance (DR). Our empirical evaluations demonstrate that DR correlates well with the human judgment; and there is a large gap between human performance and the strong baselines. Thus, we believe that ASQA is an appealing task for the QA community. Our analysis suggests that strong performance on ASQA is contingent upon both high-quality retrieval and summarization. These aspects constitute important directions for future work on ASQA."
  },
  {
    "sample_id": 36,
    "title": "Assessing Benefit from Feature Feedback in Active Learning for Text Classification",
    "abstract": "Feature feedback is an alternative to instance labeling when seeking supervision from human experts. Combination of instance and feature feedback has been shown to reduce the total annotation cost for supervised learning. However, learning problems may not benefit equally from feature feedback. It is well understood that the benefit from feature feedback reduces as the amount of training data increases. We show that other characteristics such as domain, instance granularity, feature space, instance selection strategy and proportion of relevant text, have a significant effect on benefit from feature feedback. We estimate the maximum benefit feature feedback may provide; our estimate does not depend on how the feedback is solicited and incorporated into the model. We extend the complexity measures proposed in the literature and propose some new ones to categorize learning problems, and find that they are strong indicators of the benefit from feature feedback.",
    "introduction": "Linear classifiers model the response as a weighted linear combination of the features in input instances. A supervised approach to learning a linear classifier involves learning the weights for the features from labeled data. A large number of labeled instances may be needed to determine the class association of the features and learn accurate weights for them. Alternatively, the user may directly label the features. For example, for a sentiment classification task, the user may label features, such as words or phrases, as expressing positive or negative sentiment. Prior work (Raghavan et al., 2006; Zaidan et al., 2007) has demonstrated that users are able to reliably provide useful feedback on features.\nDirect feedback on a list of features (Raghavan et al., 2006; Druck et al., 2008) is limited to simple features like unigrams. However, unigrams are limited in the linguistic phenomena they can capture. Structured features such as dependency relations, paths in syntactic parse trees, etc., are often needed for learning the target concept (Pradhan et al., 2004; Joshi and Ros´ e, 2009). It is not clear how direct feature feedback can be extended straightforwardly to structured features, as they are difficult to present visually for feedback and may require special expertise to comprehend. An alternative approach is to seek indirect feedback on structured features (Arora and Nyberg, 2009) by asking the user to highlight spans of text, called rationales, that support the instance label (Zaidan et al., 2007). For example, when classifying the sentiment of a movie review, rationales are spans of text in the review that support the sentiment label for the review.\nAssuming a fixed cost per unit of work, it might be cheaper to ask the user to label a few features, i.e. identify relevant features and their class association, than to label several instances. Prior work (Raghavan et al., 2006; Druck et al., 2008; Druck et al., 2009; Zaidan et al., 2007) has shown that a combination of instance and feature labeling can be used to reduce the total annotation cost required to learn the target concept. However, the benefit from feature feedback may vary across learning problems. If we can estimate the benefit from feature feedback for a given problem, we can minimize the total annotation cost for achieving the desired performance by selecting the optimal annotation strategy (feature feedback or not) at every stage in learning. In this paper, we present the ground work for this research problem by analyzing how benefit from feature feedback varies across different learning problems and what characteristics of a learning problem have a significant effect on benefit from feature feedback.\nWe define a learning problem (P = {D, G, F , L, I, S}) as a tuple of the domain (D), instance granularity (G), feature representation (F ), labeled data units (L), amount of irrelevant text (I) and instance selection strategy (S).\nWith enough labeled data, we may not benefit from feature feedback. Benefit from feature feedback also depends on the features used to represent the instances. If the feature space is large, we may need several labeled instances to identify the relevant features, while relatively fewer labeled features may help us quickly find these relevant features. Apart from the feature space size, it also matters what types of features are used. When hand crafted features from a domain expert are used (Pradhan et al., 2004) we expect to gain less from feature feedback as most of the features will be relevant. On the other hand, when features are extracted automatically as patterns in annotation graphs (Arora et al., 2010) feature feedback can help to identify relevant features from the large feature space.\nIn active learning, instances to be labeled are selectively sampled in each iteration. Benefit from feature feedback will depend on the instances that were used to train the model in each iteration. In the case of indirect feature feedback through rationales or direct feature feedback in context, instances selected will also determine what features receive feedback. Hence, instance selection strategy should affect the benefit from feature feedback.\nIn text classification, an instance may contain a large amount of text, and even a simple unigram representation will generate a lot of features. Often only a part of the text is relevant for the classification task. For example, in movie reviews, often the reviewers talk about the plot and characters in addition to providing their opinion about the movie. Often this extra information is not relevant to the classification task and bloats the feature space without adding many useful features. With feature feedback, we hope to filter out some of this noise and improve the model. Thus, the amount of irrelevant information in the instance should play an important role in determining the benefit from feature feedback. We expect to see less of such noise when the text instance is more concise. For example, a movie review snippet (about a sentence length) tends to have less irrelevant text than a full movie review (several sentences). In addition to analyzing document instances with varying amount of noise, we also compare the benefit from feature feedback for problems with different granularity. Granularity for a learning problem is defined based on the average amount of text in its instances.\nBenefit from feature feedback will also depend on how feedback is solicited from the user and how it is incorporated back into the model. Independently from these factors, we estimate the maximum possible benefit and analyze how it varies across problems. Next we describe measures proposed in the literature and propose some new ones for categorizing learning problems. We then discuss our experimental setup and analysis.",
    "conclusion": "In this work, we analyze how the benefit from feature feedback varies with different problem characteristics and how measures for categorizing learning problems correlate with benefit from feature feedback. We define a problem instance as a tuple of domain, instance granularity, feature representation, labeled data, amount of irrelevant text and selective sampling strategy.\nWe compare the two annotation strategies, with and without feature feedback, in terms of both improvement in performance at a given stage in learning and improvement in learning rate. Instead of evaluating the benefit from feature feedback using a specific feedback incorporation approach, we estimate and compare how the maximum benefit from feature feedback varies across different learning problems. This tells us what is the best feature feedback can do for a given learning problem.\nWe find a strong and significant correlation between feature complexity measures and the two measures of maximum benefit from feature feedback. However, these measures require an ‘oracle’, simulated using a large amount of labeled data which is not available in real world annotation tasks. We present measures based on the uncertainty of the model on its prediction that do not require an oracle. The proposed measures have a low but significant correlation with benefit from feature feedback. In our current work, we are exploring other measures of uncertainty of the model. It is intuitive that a metric that measures the uncertainty of the model on parameter estimates should correlate strongly with benefit from feature feedback. Variance in parameter estimates is one measure of uncertainty. The Bootstrap or Jacknife method (Efron and Tibshirani, 1994) of resampling from the training data is one way of estimating variance in parameter estimates that we are exploring.\nSo far only a linear relationship of various measures with benefit from feature feedback has been considered. However, some of these relationships may not be linear or a combination of several measures together may be stronger indicators of the benefit from feature feedback. We plan to do further analysis in this direction in the future.\nWe only considered one selective sampling strategy based on model’s uncertainty which we found to provide more benefit from feature feedback. In the future, we plan to explore other selective sampling strategies. For example, density-based sampling (Donmez and Carbonell, 2008) selects the instances that are representative of clusters of similar instances, and may facilitate more effective feedback on a diverse set of features.\nIn this work, feature feedback was simulated using an oracle. Feedback from the users, however, might be less accurate. Our next step will be to analyze how the benefit from feature feedback varies as the quality of feature feedback varies.\nOur eventual goal is to estimate the benefit from feature feedback for a given problem so that the right annotation strategy can be selected for a given learning problem at a given stage in learning and the total annotation cost for learning the target concept can be minimized. Note that in addition to the characteristics of the labeled data analyzed so far, expected benefit from feature feedback will also depend on the properties of the data to be labeled next for the two annotation strategies - with or without feature feedback."
  },
  {
    "sample_id": 37,
    "title": "Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning",
    "abstract": "Visual language grounding is widely studied in modern neural image captioning systems, which typically adopts an encoder-decoder framework consisting of two principal components: a convolutional neural network (CNN) for image feature extraction and a recurrent neural network (RNN) for language caption generation. To study the robustness of language grounding to adversarial perturbations in machine vision and perception, we propose Show-and-Fool, a novel algorithm for crafting adversarial examples in neural image captioning. The proposed algorithm provides two evaluation approaches, which check whether neural image captioning systems can be mislead to output some randomly chosen captions or keywords. Our extensive experiments show that our algorithm can successfully craft visually-similar adversarial examples with randomly targeted captions or keywords, and the adversarial examples can be made highly transferable to other image captioning systems. Consequently, our approach leads to new robustness implications of neural image captioning and novel insights in visual language grounding.",
    "introduction": "In recent years, language understanding grounded in machine vision and perception has made remarkable progress in natural language processing (NLP) and artificial intelligence (AI), such as image captioning and visual question answering. Image captioning is a multimodal learning task and has been used to study the interaction between language and vision models (Shekhar et al., 2017). It takes an image as an input and generates a language caption that best describes its visual contents, and has many important applications such as developing image search engines with complex natural language queries, building AI agents that can see and talk, and promoting equal web access for people who are blind or visually impaired. Modern image captioning systems typically adopt an encoder-decoder framework composed of two principal modules: a convolutional neural network (CNN) as an encoder for image feature extraction and a recurrent neural network (RNN) as a decoder for caption generation. This CNN+RNN architecture includes popular image captioning models such as Show-and-Tell (Vinyals et al., 2015), Show-Attend-and-Tell (Xu et al., 2015) and NeuralTalk (Karpathy and Li, 2015).\nRecent studies have highlighted the vulnerability of CNN-based image classifiers to adversarial examples: adversarial perturbations to benign images can be easily crafted to mislead a well-trained classifier, leading to visually indistinguishable adversarial examples to human (Szegedy et al., 2014; Goodfellow et al., 2015). In this study, we investigate a more challenging problem in visual language grounding domain that evaluates the robustness of multimodal RNN in the form of a CNN+RNN architecture, and use neural image captioning as a case study. Note that crafting adversarial examples in image captioning tasks is strictly harder than in well-studied image classification tasks, due to the following reasons: (i) class attack v.s. caption attack: unlike classification tasks where the class labels are well defined, the output of image captioning is a set of top-ranked captions. Simply treating different captions as distinct classes will result in an enormous number of classes that can even precede the number of training images. In addition, semantically similar Figure 1: Adversarial examples crafted by Showand-Fool using the targeted caption method. The target captioning model is Show-and-Tell (Vinyals et al., 2015), the original images are selected from the MSCOCO validation set, and the targeted captions are randomly selected from the top-1 inferred caption of other validation images.\ncaptions can be expressed in different ways and hence should not be viewed as different classes; and (ii) CNN v.s. CNN+RNN: attacking RNN models is significantly less well-studied than attacking CNN models. The CNN+RNN architecture is unique and beyond the scope of adversarial examples in CNN-based image classifiers.\nIn this paper, we tackle the aforementioned challenges by proposing a novel algorithm called Show-and-Fool. We formulate the process of crafting adversarial examples in neural image captioning systems as optimization problems with novel objective functions designed to adopt the CNN+RNN architecture. Specifically, our objective function is a linear combination of the distortion between benign and adversarial examples as well as some carefully designed loss functions. The proposed Show-and-Fool algorithm provides two approaches to craft adversarial examples in neural image captioning under different scenarios: 1. Targeted caption method: Given a targeted caption, craft adversarial perturbations to any image such that its generated caption matches the targeted caption.\n2. Targeted keyword method: Given a set of keywords, craft adversarial perturbations to any image such that its generated caption contains the specified keywords. The captioning model has the freedom to make sentences with target keywords in any order.\nAs an illustration, Figure 1 shows an adversarial example crafted by Show-and-Fool using the targeted caption method. The adversarial perturbations are visually imperceptible while can successfully mislead Show-and-Tell to generate the targeted captions. Interestingly and perhaps surprisingly, our results pinpoint the Achilles heel of the language and vision models used in the tested image captioning systems. Moreover, the adversarial examples in neural image captioning highlight the inconsistency in visual language grounding between humans and machines, suggesting a possible weakness of current machine vision and perception machinery. Below we highlight our major contributions:\n• We propose Show-and-Fool, a novel optimization based approach to crafting adversarial examples in image captioning. We provide two types of adversarial examples, targeted caption and targeted keyword, to analyze the robustness of neural image captioners. To the best of our knowledge, this is the very first work on crafting adversarial examples for image captioning.\n• We propose powerful and generic loss functions that can craft adversarial examples and evaluate the robustness of the encoder-decoder pipelines in the form of a CNN+RNN architecture. In particular, our loss designed for targeted keyword attack only requires the adversarial caption to contain a few specified keywords; and we allow the neural network to make meaningful sentences with these keywords on its own.\n• We conduct extensive experiments on the MSCOCO dataset. Experimental results show that our targeted caption method attains a 95.8% attack success rate when crafting adversarial examples with randomly assigned captions. In addition, our targeted keyword attack yields an even higher success rate. We also show that attacking CNN+RNN models is inherently different and more challenging than only attacking CNN models.\n• We also show that Show-and-Fool can produce highly transferable adversarial examples: an adversarial image generated for fooling Showand-Tell can also fool other image captioning models, leading to new robustness implications of neural image captioning systems.",
    "conclusion": "In this paper, we proposed a novel algorithm, Show-and-Fool, for crafting adversarial examples and providing robustness evaluation of neural image captioning. Our extensive experiments show that the proposed targeted caption and keyword methods yield high attack success rates while the adversarial perturbations are still imperceptible to human eyes. We further demonstrate that Showand-Fool can generate highly transferable adversarial examples. The high-quality and transferable adversarial examples in neural image captioning crafted by Show-and-Fool highlight the inconsistency in visual language grounding between humans and machines, suggesting a possible weakness of current machine vision and perception machinery. We also show that attacking neural image captioning systems are inherently different from attacking CNN-based image classifiers.\nOur method stands out from the well-studied adversarial learning on image classifiers and CNN models. To the best of our knowledge, this is the very first work on crafting adversarial examples for neural image captioning systems. Indeed, our Show-and-Fool algorithm1 can be easily extended to other applications with RNN or CNN+RNN architectures. We believe this paper provides potential means to evaluate and possibly improve the robustness (for example, by adversarial training or data augmentation) of a wide range of visual language grounding and other NLP models."
  },
  {
    "sample_id": 38,
    "title": "The Architectural Bottleneck Principle",
    "abstract": "In this paper, we seek to measure how much information a component in a neural network could extract from the representations fed into it. Our work stands in contrast to prior probing work, most of which investigates how much information a model’s representations contain. This shift in perspective leads us to propose a new principle for probing, the architectural bottleneck principle: In order to estimate how much information a given component could extract, a probe should look exactly like the component. Relying on this principle, we estimate how much syntactic information is available to transformers through our attentional probe, a probe that exactly resembles a transformer’s self-attention head. Experimentally, we find that, in three models (BERT, ALBERT, and RoBERTa), a sentence’s syntax tree is mostly extractable by our probe, suggesting these models have access to syntactic information while composing their contextual representations. Whether this information is actually used by these models, however, remains an open question.\nhttps://github.com/rycolab/\nattentional-probe",
    "introduction": "The surprising performance of pretrained language models on diverse natural language processing tasks has sparked interest in their analysis. Probing is one of the most prevalent methods employed to engage in such an analysis. In a typical probing study (Alain and Bengio, 2016; Belinkov et al., 2017; Adi et al., 2017, inter alia), the weights of the model under consideration are first frozen. A probe is then trained on top of the model’s contextual representations in an attempt to predict one of the input sentence’s properties, e.g., its syntactic parse. Unfortunately, best practices on how to design such probes remain contested.\nOn one side of the debate, some argue for simplicity, suggesting that simple probes are to be preferred so that we can distinguish probing from simply learning an NLP task (Hewitt and Liang, 2019). On the other side of the debate, some argue we need complex probes in order to extract all relevant information from the representations (Saphra and Lopez, 2019; Pimentel et al., 2020b). Bridging the gap, some have also called for a compromise, advocating that all probes on the complexity–accuracy Pareto curve should be considered (Pimentel et al., 2020a).\nIn this paper, we propose the architectural bottleneck principle (ABP) as a guideline for constructing useful probes. Under the ABP, a probe’s architecture should mirror a component of the model being probed. Previous work has mostly focused on how much information is contained in a set of representations. However, if we care about whether the information is in fact used by the model, we should instead ask how much information the model in question could use.1 Under this perspective, the probed model’s architecture acts as a natural bottleneck to how much information the model could use—and should thus also act as a constraint when probing.2\nAs a concrete example, we posit that a transformer’s attention head serves as a bottleneck to its use of syntactic information, as these are the only components in a transformer with access to multiple tokens at once. Following the ABP, we thus propose the attentional probe, which looks exactly like an attention head. This probe allows us to answer one specific question: How much syntactic information could a transformer use while computing its attention weights?\nOur results reveal that most—albeit not all— syntactic information is extractable with this simple attention head architecture: While we estimate English sentences to contain on average 31.2 bits of information about their syntactic tree structure, the attentional probe can extract up to 28.0 bits. Furthermore, while these results hold for three popular transformer-based language models (BERT, ALBERT and RoBERTa), they do not for a similar but untrained model. This suggests that training a model shapes its representations to encode syntactic information. We find this trend holds across four typologically diverse languages (Basque, English, Tamil, and Turkish). In contrast, when we keep BERT’s pretrained parameters frozen and analyze the weights of its pretrained attention heads, we observe that they do not seem to encode syntax under our operationalisation. Ergo, while we know these models could use syntactic information to compute attention weights, whether they actually do remains an open question.",
    "conclusion": "In this paper, we have approached probing from a new perspective. Rather than asking how much information is encoded by the model, we ask how much information its components could extract. We then quantify this amount using V-information.\nEvaluating the attention mechanism of popular transformer language models, we find that the majority of the information about the syntax tree of a sentence is in fact extractable by the model. This, however, is not true for randomly initialised transformer models. Our results, thus, lead us to conclude that a transformer’s training leads its attention heads to have the potential to decode syntax trees."
  },
  {
    "sample_id": 39,
    "title": "Attention Guided Graph Convolutional Networks for Relation Extraction",
    "abstract": "Dependency trees convey rich structural information that is proven useful for extracting relations among entities in text. However, how to effectively make use of relevant information while ignoring irrelevant information from the dependency trees remains a challenging research question. Existing approaches employing rule based hard-pruning strategies for selecting relevant partial dependency structures may not always yield optimal results. In this work, we propose Attention Guided Graph Convolutional Networks (AGGCNs), a novel model which directly takes full dependency trees as inputs. Our model can be understood as a soft-pruning approach that automatically learns how to selectively attend to the relevant sub-structures useful for the relation extraction task. Extensive results on various tasks including cross-sentence n-ary relation extraction and large-scale sentence-level relation extraction show that our model is able to better leverage the structural information of the full dependency trees, giving significantly better results than previous approaches.",
    "introduction": "Relation extraction aims to detect relations among entities in the text. It plays a significant role in a variety of natural language processing applications including biomedical knowledge discovery (Quirk and Poon, 2017), knowledge base population (Zhang et al., 2017) and question answering (Yu et al., 2017). Figure 1 shows an example about expressing a relation sensitivity among three entities L858E, EGFR and gefitinib in two sentences.\nMost existing relation extraction models can be categorized into two classes: sequence-based and dependency-based. Sequence-based models operate only on the word sequences (Zeng et al., 2014; Wang et al., 2016), whereas dependencybased models incorporate dependency trees into the models (Bunescu and Mooney, 2005; Peng et al., 2017). Compared to sequence-based models, dependency-based models are able to capture non-local syntactic relations that are obscure from the surface form alone (Zhang et al., 2018). Various pruning strategies are also proposed to distill the dependency information in order to further improve the performance. Xu et al. (2015b,c) apply neural networks only on the shortest dependency path between the entities in the full tree. Miwa and Bansal (2016) reduce the full tree to the subtree below the lowest common ancestor (LCA) of the entities. Zhang et al. (2018) apply graph convolutional networks (GCNs) (Kipf and Welling, 2017) model over a pruned tree. This tree includes tokens that are up to distance K away from the dependency path in the LCA subtree.\nHowever, rule-based pruning strategies might eliminate some important information in the full tree. Figure 1 shows an example in cross-sentence n-ary relation extraction that the key tokens partial response would be excluded if the model only takes the pruned tree into consideration. Ideally, the model should be able to learn how to maintain a balance between including and excluding information in the full tree. In this paper, we propose the novel Attention Guided Graph Convolutional Networks (AGGCNs), which operate directly on the full tree. Intuitively, we develop a “soft pruning” strategy that transforms the original dependency tree into a fully connected edgeweighted graph. These weights can be viewed as the strength of relatedness between nodes, which can be learned in an end-to-end fashion by using self-attention mechanism (Vaswani et al., 2017).\nIn order to encode a large fully connected graph, we next introduce dense connections (Huang et al., 2017) to the GCN model following (Guo et al.,\nThe deletion mutation on exon-19 of EGFR gene was present in 16 patients, while the L858E point mutation on exon-21 was noted.\nAll patients were treated response. with gefitinib and showed a partial\nNN PREP_ON\nCONJ_AND DOBJDETAMOD\nFigure 1: An example dependency tree for two sentences expressing a relation (sensitivity) among three entities. The shortest dependency path between these entities is highlighted in bold (edges and tokens). The root node of the LCA subtree of entities is present. The dotted edges indicate tokens K=1 away from the subtree. Note that tokens partial response off these paths (shortest dependency path, LCA subtree, pruned tree when K=1).\n2019). For GCNs, L layers will be needed in order to capture neighborhood information that is L hops away. A shallow GCN model may not be able to capture non-local interactions of large graphs. Interestingly, while deeper GCNs can capture richer neighborhood information of a graph, empirically it has been observed that the best performance is achieved with a 2-layer model (Xu et al., 2018). With the help of dense connections, we are able to train the AGGCN model with a large depth, allowing rich local and non-local dependency information to be captured.\nExperiments show that our model is able to achieve better performance for various tasks. For the cross-sentence relation extraction task, our model surpasses the current state-of-theart models on multi-class ternary and binary relation extraction by 8% and 6% in terms of accuracy respectively. For the largescale sentence-level extraction task (TACRED dataset), our model is also consistently better than others, showing the effectiveness of the model on a large training set. Our code is available at http://www.statnlp.org/\nresearch/information-extraction1\nOur contributions are summarized as follows:\n• We propose the novel AGGCNs that learn a “soft pruning” strategy in an end-to-end fashion, which learns how to select and discard information. Combining with dense connections, our AGGCN model is able to learn a better graph representation.\n• Our model achieves new state-of-the-art results without additional computational overhead when compared with previous GCNs.2 Unlike tree-structured models (e.g., TreeLSTM (Tai et al., 2015)), it can be efficiently applied over dependency trees in parallel.",
    "conclusion": "We introduce the novel Attention Guided Graph Convolutional Networks (AGGCNs). Experimental results show that AGGCNs achieve state-ofthe-art results on various relation extraction tasks. Unlike previous approaches, AGGCNs operate directly on the full tree and learn to distill the useful information from it in an end-to-end fashion. There are multiple venues for future work. One natural question we would like to ask is how to make use of the proposed framework to perform improved graph representation learning for graph related tasks (Bastings et al., 2017)."
  },
  {
    "sample_id": 40,
    "title": "Attention over Heads: A Multi-Hop Attention for Neural Machine Translation",
    "abstract": "In this paper, we propose a multi-hop attention for the Transformer. It refines the attention for an output symbol by integrating that of each head, and consists of two hops. The first hop attention is the scaled dot-product attention which is the same attention mechanism used in the original Transformer. The second hop attention is a combination of multi-layer perceptron (MLP) attention and head gate, which efficiently increases the complexity of the model by adding dependencies between heads. We demonstrate that the translation accuracy of the proposed multi-hop attention outperforms the baseline Transformer significantly, +0.85 BLEU point for the IWSLT-2017 German-toEnglish task and +2.58 BLEU point for the WMT-2017 German-to-English task. We also find that the number of parameters required for a multi-hop attention is smaller than that for stacking another self-attention layer and the proposed model converges significantly faster than the original Transformer.",
    "introduction": "Multi-hop attention was first proposed in end-toend memory networks (Sukhbaatar et al., 2015) for machine comprehension. In this paper, we define a hop as a computational step which could be performed for an output symbol many times. By “multi-hop attention”, we mean that some kind of attention is calculated many times for generating an output symbol. Previous multihop attention can be classified into “recurrent attention” (Sukhbaatar et al., 2015) and “hierarchical attention” (Libovick´ y and Helcl, 2017). The former repeats the calculation of attention many times to refine the attention itself while the latter integrates attentions for multiple input information sources. The proposed multi-hop attention for the Transformer is different from previous recurrent attentions because the mechanism for the first hop attention and that for the second hop attention is different. It is also different from previous hierarchical attention because it is designed to integrate attentions from different heads for the same information source.\nIn neural machine translation, hierarchical attention (Bawden et al., 2018; Libovick´ y and Helcl, 2017) can be thought of a multi-hop attention because it repeats attention calculation to integrate the information from multiple source encoders. On the other hand, in the Transformer (Vaswani et al., 2017), the stateof-the-art model for neural machine translation, feed-forward neural network (FFNN) integrates information from multiple heads. In this paper, we propose a multi-hop attention mechanism as a possible alternative to integrate information from multi-head attention in the Transformer.\nWe find that the proposed Transformer with multi-hop attention converges faster than the original Transformer. This is likely because all heads learn to influence each other, through a head gate mechanism, in the second hop attention (Figure 1). Recently, many Transformer-based pretrained language models such as BERT have been proposed and take about a month for training. The speed at which the proposed model converges may be even more important than the fact that its accuracy is slightly better.",
    "conclusion": "In this paper, we have proposed a multi-hop attention mechanism for a Transformer model in which all heads depend on each other repeatedly. We found that the proposed method significantly outperforms the original Transformer in accuracy and converges faster with little increase in the number of parameters. In future work, we would like to implement a multi-hop attention mechanism to the decoder side and investigate other language pairs."
  },
  {
    "sample_id": 41,
    "title": "Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In",
    "abstract": "Retrieval augmentation can aid language models (LMs) in knowledge-intensive tasks by supplying them with external information. Prior works on retrieval augmentation usually jointly fine-tune the retriever and the LM, making them closely coupled. In this paper, we explore the scheme of generic retrieval plug-in: the retriever is to assist target LMs that may not be known beforehand or are unable to be fine-tuned together. To retrieve useful documents for unseen target LMs, we propose augmentation-adapted retriever (AAR), which learns LM’s preferences obtained from a known source LM. Experiments on the MMLU and PopQA datasets demonstrate that our AAR trained with a small source LM is able to significantly improve the zero-shot generalization of larger target LMs ranging from 250M Flan-T5 to 175B InstructGPT. Further analysis indicates that the preferences of different LMs overlap, enabling AAR trained with a single source LM to serve as a generic plug-in for various target LMs. Our code is open-sourced at https://github.com/OpenMatch/AugmentationAdapted-Retriever.",
    "introduction": "Large language models (LMs) that possess billions of parameters are able to capture a significant amount of human knowledge, leading to consistent improvements on various downstream tasks (Brown et al., 2020; Kaplan et al., 2020; Roberts et al., 2020). However, the undeniable drawback of large LMs lies in their high computational cost, which negatively impacts their efficiency (Strubell et al., 2019; Bender et al., 2021). Furthermore, the knowledge memorized from pretraining and the implicit reasoning process of LMs can be inaccurate and intractable sometimes, hindering their applications on knowledge-intensive tasks (Guu et al., 2020; Lewis et al., 2020; Mallen et al., 2022; Wei et al., 2022).\nLM w/ Adaptive Retrieval\nLM w/ AAR (Ours)\nFigure 1: Performance of LM w/ AAR (Ours).\nInstead of leveraging the knowledge and reasoning abilities embedded within the parameters of the LMs, retrieval augmentation (Guu et al., 2020; Lewis et al., 2020; Borgeaud et al., 2022) enhances the LM with a retriever that can retrieve knowledge from an external corpus. On the other hand, prior retrieval augmentation methods (Izacard and Grave, 2021a; Izacard et al., 2022) necessitate fine-tuning the backbone LM to adjust to the retriever and tackle specific downstream tasks. This kind of fine-tuning can be expensive when more and more unique demands emerge (Maronikolakis and Schütze, 2021). More importantly, many toptier LMs can only be accessed through black-box APIs (Ouyang et al., 2022; OpenAI, 2023). These APIs allow users to submit queries and receive responses but typically do not support fine-tuning.\nIn this paper, we introduce AugmentationAdapted Retriever (AAR) to assist black-box LMs with downstream tasks as generic plug-in. To retrieve valuable documents for many unseen LMs, we propose to leverage a small source LM to provide LM-preferred signals for retriever’s training. The retriever after training (i.e., AAR) can be directly utilized to assist a large target LM by plugging in the retrieved documents.\nSpecifically, we choose a small encoder-decoder LM as the source LM and utilize its fusionin-decoder attention scores (Izacard and Grave, 2021a) to annotate LM-preferred documents. The LM-preferred documents are then combined with human-preferred documents to form the positive document set. Negative documents are mined by the retriever itself using the ANCE (Xiong et al., 2021) technique. After fine-tuning the retriever with LM’s preferences, it can directly assist unseen target LMs in the zero-shot task generalization.\nWe evaluate AAR on a multi-task language understanding dataset MMLU (Hendrycks et al., 2021) and an entity-centric question answering dataset PopQA (Mallen et al., 2022). For the target LMs, we choose Flan-T5 (Chung et al., 2022) series as our backbone for encoder-decoder LMs and InstructGPT (Ouyang et al., 2022) as our backbone for decoder-only LMs. Figure 1 shows that assisted with a generic AAR, LMs of different sizes and architectures can consistently outperform the standalone LMs; the performance of smaller LMs can sometimes surpass the standalone counterparts of significantly larger sizes (e.g., Flan-T5 Large w/ AAR outperforms standalone Flan-T5 XL by 0.6%). AAR also demonstrates advantages over other augmentation approaches such as few-shot prompting and adaptive retrieval (Mallen et al., 2022).\nFurther analysis reveals that the preferences obtained from different-sized source LMs are similar, and LMs with near capacities tend to yield closer preferred document sets. As a result, our AAR model trained from a small source LM can be considered as a generic plug-in to enhance the zeroshot generalization of a significantly larger target LM. We also discover that the documents preferred by LMs can provide assistance to the model from alternative perspectives, rather than relying solely on the full information favored by search users.",
    "conclusion": "This paper introduces generic retrieval plug-in that utilizes a generic retriever to enhance target LMs that may be unknown in advance or are unable to be fine-tuned jointly. Our proposed retriever, AAR, can directly support black-box LMs without requiring any fine-tuning of the LMs. This is accomplished by building the AAR’s training data with preferred documents from a small source LM together with the ground truth.\nEmpirical results on MMLU and PopQA demonstrate that AAR-assisted LMs greatly outperform the standalone ones in zero-shot scenarios, and AAR generalizes well to LMs of different sizes and structures. Analytical results reveal that LMpreferred and human-preferred documents complement each other; LM-preferred documents from different LMs overlap significantly, and LMs with similar sizes tend to yield closer document sets.\nWe leave a more detailed explanation of how different LMs interact with augmentation documents and a more reasonable selection of LM-preferred documents for future work. We hope our work shed light on a path to a generic way of treating large LMs as black boxes and adapting retrievers to augment them."
  },
  {
    "sample_id": 42,
    "title": "Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories",
    "abstract": "In this paper we improve the zero-shot generalization ability of language models via MixtureOf-Memory Augmentation (MoMA), a mechanism that retrieves augmentation documents from multiple information corpora (“external memories”), with the option to “plug in” unseen memory at inference time. We develop a joint learning mechanism that trains the augmentation component with latent labels derived from the end retrieval task, paired with hard negatives from the memory mixture. We instantiate the model in a zero-shot dense retrieval setting by augmenting strong T5-based retrievers with MoMA. With only T5-base, our model obtains strong zero-shot retrieval accuracy on the eighteen tasks included in the standard BEIR benchmark, outperforming some systems with larger model sizes. As a plug-inplay model, our model can efficiently generalize to any unseen corpus, meanwhile achieving comparable or even better performance than methods relying on target-specific pretraining. Our analysis further illustrates the necessity of augmenting with mixture-of-memory for robust generalization, the benefits of augmentation learning, and how MoMA utilizes the plugin memory at inference time without changing its parameters. Our code can be found at https://github.com/gesy17/MoMA.",
    "introduction": "Scaling up language models—with more parameters and pretraining data—improves model generalization ability on downstream applications (Raffel et al., 2019; Brown et al., 2020; Smith et al., 2022), but with diminishing return: linear improvements on downstream metrics often require exponentially more parameters and computing cost (Kaplan et al., 2020; Hoffmann et al., 2022). Hence, scaling pretrained language models in this way is economically unsustainable (Strubell et al., 2020; Bender et al., 2021; Zhang et al., 2022).\nRetrieval augmented language models provide a promising alternative. They allow language models to efficiently access vast resources from an external corpus (Guu et al., 2020; Borgeaud et al., 2022) that serves as a kind of “memory” they can refer to when making predictions, alleviating the need to memorize as much information in their own network parameters (Roberts et al., 2020). This open-book approach helps language models to better generalize on token prediction tasks and machine translation (Khandelwal et al., 2019; Borgeaud et al., 2022), and tasks which already involve a first-stage retrieval component, e.g., OpenQA (Borgeaud et al., 2022; Izacard et al., 2022). Existing retrieval augmentation methods usually stick to one single retrieval corpus throughout training and inference so that the retrieval component can be indirectly guided by the supervision from end tasks.\nIn this paper we improve the zero-shot generalization ability of language models using “mixtureof-memory” (MoMA), a new retrieval augmentation mechanism. Instead of a single corpus, MoMA retrieves documents from a “mixture” of multiple external corpora and enjoys the merits of a larger and more comprehensive source of knowledge. This mechanism also allows removing and/or “plugging-in” new corpora during inference time, when more information from the target task is revealed, or as an additional way for users to control the model. Specifically, we apply MoMA on the zero-shot dense retrieval task, which is the foundation of many important real-world applications (Thakur et al., 2021a; Kim, 2022) and also the retrieval component of recent retrieval augmented language models (Guu et al., 2020; Izacard et al., 2022). However, it is not trivial to guide a retrieval model to leverage multiple corpora. We need to jointly train the augmentation component and dense retriever using supervised relevance signals and self-mined hard negatives.\nWe instantiate MoMA with a T5-base model (Ni et al., 2022) and apply it to the dense retrieval task (Karpukhin et al., 2020). Our end task retriever uses a set of augmenting documents from the mixture-of-memories to enhance its representation of the query with important context; the retriever then uses the enhanced query representation to retrieve a final candidate set. At inference time, we plug in the target task’s corpus to the memory mixture to introduce in-domain context information, without updating any parameter.\nAs a plug-in-play method, MoMA provides an flexible but powerful solution to zero-shot dense retrieval: Unlike recent state-of-the-art methods (Yu et al., 2022; Neelakantan et al., 2022), it does not require pretraining on target corpus or large-scale web corpus, enabling it to generalize to arbitrary unseen corpus without additional effort. It can also be used as an efficient alternative for recent large language model (LLM) based generative retrieval models (Gao et al., 2022). Given the target query, MoMA only involves the T5-base model for query encoding, which is significantly cheaper than querying an LLM to generate pseudo answers and re-encoding it.\nWe experimented on eighteen zero-shot dense retrieval tasks included in BEIR (Thakur et al., 2021a), the standard ZeroDR benchmark. The results demonstrate the improved zero-shot ability of MoMA. MoMA achieves comparable or even stronger performance to recent state-of-the-art dense retrieval systems with larger model scales and heavier computation costs. Our further analysis reveals that large and diverse corpora in the memory leads to the best performance; while only using a single corpus during training does not improve performance on unseen target tasks. The learning of augmentation component is also important for MoMA to utilize the diverse information from the mixture. Our analysis and case studies illustrate how MoMA leverages the plug-in memory at testing time to enrich its query representations.",
    "conclusion": "In this paper we propose a new plug-in mixtureof-memory mechanism for the retrieval augmented language models to improve their zero-shot ability on the dense retrieval task. To learn the memory mixture we develop a new joint learning approach that trains the augmentation component using the positive signals from the end task, the language model’s attention scores, and hard negatives retrieved from the mixture of augmentation corpora. This leads to our final model MoMA (T5ANCE) and MoMA (coCondenser) that achieve strong zero-shot accuracy on 18 retrieval tasks included in BEIR. Our analysis shows the importance of augmenting with diverse memory sources and indomain information for robust generalization. We hope our findings can inspire more future research in better augmenting language models, to provide other alternatives to achieve generalization ability beyond solely relying on model scale."
  },
  {
    "sample_id": 43,
    "title": "Automated Cross-language Intelligibility Analysis of Parkinson's Disease Patients Using Speech Recognition Technologies",
    "abstract": "Speech deficits are common symptoms among Parkinson’s Disease (PD) patients. The automatic assessment of speech signals is promising for the evaluation of the neurological state and the speech quality of the patients. Recently, progress has been made in applying machine learning and computational methods to automatically evaluate the speech of PD patients. In the present study, we plan to analyze the speech signals of PD patients and healthy control (HC) subjects in three different languages: German, Spanish, and Czech, with the aim to identify biomarkers to discriminate between PD patients and HC subjects and to evaluate the neurological state of the patients. Therefore, the main contribution of this study is the automatic classification of PD patients and HC subjects in different languages with focusing on phonation, articulation, and prosody. We will focus on an intelligibility analysis based on automatic speech recognition systems trained on these three languages. This is one of the first studies done that considers the evaluation of the speech of PD patients in different languages. The purpose of this research proposal is to build a model that can discriminate PD and HC subjects even when the language used for train and test is different.",
    "introduction": "Parkinsons disease (PD) (i.e. Shaking palsy (Parkinson, 2002)) is the second most common neurodegenerative disorder after Alzheimers disease. PD displays a great prevalence in individuals of advanced age (Dexter and Jenner, 2013), especially, over the age of fifty (Fahn, 2003). The signs and symptoms of PD can significantly influence the quality of life of patients. They are grouped into two categories: motor and non-motor symptoms. Speech impairments are one of the earliest manifestations in PD patients.\nEarly diagnosis of PD is a vital challenge in this field. The first step in analyzing this disease is the development of markers of PD progression through collecting data from several cohorts. To reach this aim, different clinical rating scales have been developed, such as the Unified Parkinson’s Disease Rating Scale (UPDRS), Movement Disorders Society - UPDRS (MDS-UPDRS) 1 (Goetz et al., 2008) and Hoehn & Yahr (H & Y) staging, (Visser et al., 2006).\nThe UPDRS is the most widely used rating tool for the clinical evaluation of PD patients. The examination requires observation and interview by a professional clinician. The scale is distributed into 4 sections: (i) Mentation, behavior and mood, (ii) Activities of daily living (ADL), (iii) Motor sections, and (iv) Motor complications.\nOne of the most common motor problems is related to speech impairments in PD (Jankovic, 2008). Most of the patients with PD show disabilities in speech production. The most common speech disturbances are monotonic speech, hypophonia (a speech weakness in the vocal musculature and vocal sounds) and hypokinetic dysarthria. These symptoms reduce the intelligibility of the patients, and affect different aspects of the speech production such as articulation, phonation, nasality, and prosody (Little et al., 2009; Goetz et al., 2008; Ramig et al., 2001). Therefore, there is a great interest to develop tools or methods to evaluate and improve the speech production of PD paRecently, there has been a proliferation of new speech recognition-based tools for the acoustic analysis of PD. The use of speech recognition software in clinical examinations could make a powerful supplement to the state-of-the-art subjective reports of experts and clinicians that are costly and time-consuming (e.g., Little et al., 2009; Hernandez-Espinosa et al., 2002). In the clinical field, the detection of PD is a complex task due to the fact that the symptoms of this disease are more related to clinicians’ observations and perception of the way patients move and speak.\nRecently, machine learning tools are used to develop speech recognition systems that make the whole process of objective evaluation and recognition faster and more accurate than analytical clinicians’ methods (Yu and Deng, 2016; HernandezEspinosa et al., 2002). Using machine learning techniques to extract acoustic features for detecting the PD has become widely used in recent studies (e.g., Dahl et al., 2012; Little et al., 2009).\nAutomatic speech recognition (ASR) systems are used to decode and transcribe oral speech. In other words, the goal of ASR systems is to find and recognize the words that best represent the acoustic signal. For example, automatic speech recognition systems are used to evaluate how speech intelligibility is affected by the disease.\nThis study will seek to further investigate the speech patterns of HC and PD groups using recordings from patients speaking in German, Spanish, and Czech. Most of the previous studies only considered recordings in one language and focused on it for detecting PD, but in this study, we plan to evaluate the effect of the PD in three different languages.",
    "conclusion": "In this research proposal, we introduced and described the background for speech recognition of PD patients. The focus is on Parkinsons disease speech recognition based on the acoustic analysis of their voice. A brief overview of clinical and machine learning research in this field was provided. The goal is to improve the ASR system to be able to model and detect PD patients independently from their language by taking speech as an input and using machine learning and natural language processing technologies to advance healthcare and provide an overview of the patients mental health. All in all, the proposed method should be able to detect the patient with PD and discriminate them from HC subjects."
  },
  {
    "sample_id": 44,
    "title": "Automated essay scoring with string kernels and word embeddings",
    "abstract": "In this work, we present an approach based on combining string kernels and word embeddings for automatic essay scoring. String kernels capture the similarity among strings based on counting common character ngrams, which are a low-level yet powerful type of feature, demonstrating state-of-theart results in various text classification tasks such as Arabic dialect identification or native language identification. To our best knowledge, we are the first to apply string kernels to automatically score essays. We are also the first to combine them with a high-level semantic feature representation, namely the bag-of-super-word-embeddings. We report the best performance on the Automated Student Assessment Prize data set, in both indomain and cross-domain settings, surpassing recent state-of-the-art deep learning approaches.",
    "introduction": "Automatic essay scoring (AES) is the task of assigning grades to essays written in an educational setting, using a computer-based system with natural language processing capabilities. The aim of designing such systems is to reduce the involvement of human graders as far as possible. AES is a challenging task as it relies on grammar as well as semantics, pragmatics and discourse (Song et al., 2017). Although traditional AES methods typically rely on handcrafted features (Larkey, 1998; Foltz et al., 1999; Attali and Burstein, 2006; Dikli, 2006; Wang and Brown, 2008; Chen and He, 2013; Somasundaran et al., 2014; Yannakoudakis et al., 2014; Phandi et al., 2015), recent results indicate that state-of-the-art deep learning methods reach better performance (Alikaniotis et al., 2016; Dong and Zhang, 2016; Taghipour and Ng, 2016; Dong et al., 2017; Song et al., 2017; Tay et al., 2018), perhaps because these methods are able to capture subtle and complex information that is relevant to the task (Dong and Zhang, 2016).\nIn this paper, we propose to combine string kernels (low-level character n-gram features) and word embeddings (high-level semantic features) to obtain state-of-the-art AES results. Since recent methods based on string kernels have demonstrated remarkable performance in various text classification tasks ranging from authorship identification (Popescu and Grozea, 2012) and sentiment analysis (Gim´ enez-P´ erez et al., 2017; Popescu et al., 2017) to native language identification (Popescu and Ionescu, 2013; Ionescu et al., 2014; Ionescu, 2015; Ionescu et al., 2016; Ionescu and Popescu, 2017) and dialect identification (Ionescu and Popescu, 2016; Ionescu and Butnaru, 2017), we believe that string kernels can reach equally good results in AES. To the best of our knowledge, string kernels have never been used for this task. As string kernels are a simple approach that relies solely on character n-grams as features, it is fairly obvious that such an approach will not to cover several aspects (e.g.: semantics, discourse) required for the AES task. To solve this problem, we propose to combine string kernels with a recent approach based on word embeddings, namely the bag-of-super-word-embeddings (BOSWE) (Butnaru and Ionescu, 2017). To our knowledge, this is the first successful attempt to combine string kernels and word embeddings. We evaluate our approach on the Automated Student Assessment Prize data set, in both in-domain and cross-domain settings. The empirical results indicate that our approach yields a better performance than state-of-the-art approaches (Phandi et al., 2015; Dong and Zhang, 2016; Dong et al., 2017; Tay et al., 2018).",
    "conclusion": "In this paper, we described an approach based on combining string kernels and word embeddings for automatic essay scoring. We compared our approach on the Automated Student Assessment Prize data set, in both in-domain and cross-domain settings, with several state-of-the-art approaches (Phandi et al., 2015; Dong and Zhang, 2016; Dong et al., 2017; Tay et al., 2018). Overall, the indomain and the cross-domain comparative studies indicate that string kernels, both alone and in combination with word embeddings, attain the best performance on the automatic essay scoring task. Using a shallow approach, we report better results compared to recent deep learning approaches (Dong and Zhang, 2016; Dong et al., 2017; Tay et al., 2018)."
  },
  {
    "sample_id": 45,
    "title": "Automatic Derivation of Semantic Representations for Thai Serial Verb Constructions: A Grammar-Based Approach",
    "abstract": "Deep semantic representations are useful for many NLU tasks (Droganova and Zeman, 2019; Schuster and Manning, 2016). Manual annotation to build these representations is timeconsuming, and so automatic approaches are preferred (Droganova and Zeman, 2019; Bender et al., 2015). This paper demonstrates how rich semantic representations can be automatically derived for Thai Serial Verb Constructions (SVCs), where the semantic relationship between component verbs is not immediately clear from the surface forms. I present the first fully-implemented, unified analysis for Thai SVCs, deriving appropriate semantic representations (MRS; Copestake et al., 2005) from syntactic features, implemented within a DELPH-IN computational grammar (Slayden, 2009). This analysis increases verified coverage of SVCs by 73% and decreases ambiguity by 46%. The final grammar can be found at: https://github.com/VipashaB94/ThaiGrammar",
    "introduction": "This paper presents the first fully-implemented analysis of a broad range of Thai SVCs in a computational grammar. An example of a Thai SVC is seen in (1).1\nMy grammar implementation uses HPSG (Pollard and Sag, 1994; Müller et al., 2021), and produces semantic representations in the Minimal Recursion Semantics (MRS) framework (Copestake\nwork, but presented with slight modifications. In particular,\net al., 2005). These representations model the semantic relationships in the construction, which are derived from the syntactic features of component verbs. The analysis was implemented on the basis of a DELPH-IN computational grammar, originally developed by Slayden (2009). The final grammar was tested against 216 development sentences, 205 regression sentences, and 85 held-out sentences, of which 77 were from naturally-occurring data. I show that this implementation increases verified coverage of Thai SVCs by 73% and decreases ambiguity by 46% on held-out data.\nSemantic parsing is beneficial for performing various Natural Language Understanding (NLU) tasks such as biomedical text mining or open domain relation extraction (Schuster and Manning, 2016; Bender et al., 2015). Rich semantic representations can greatly improve the performance of systems on such tasks. For example, in dependency parsing, dependency trees containing deep semantic representations are more useful than surfacesyntactic dependency trees (Droganova and Zeman, 2019; Schuster and Manning, 2016), which often rely too strongly on the surface structure of sentences, and do not show the relationships between content words (Schuster and Manning, 2016).\nArriving at these deep semantic representations requires complex semantic annotation (Droganova and Zeman, 2019), which can be either manual or grammar-driven. For example, the Enhanced (and Enhanced ++) Universal Dependency representations aim to make certain implicit relationships between content words more explicit by adding relations and augmenting relation names (Schuster and Manning, 2016). Alternatively, the English Resource Grammar (ERG; Flickinger 2000, 2011) takes a grammar-driven approach to produce compositional meaning annotations, and can successfully derive syntactic and semantic analyses for 85-95% of utterances in English text corpora (Bender et al., 2015).\nManual annotation, particularly for previously unannotated languages, is time-consuming and resource intensive, and so automatic approaches are extremely beneficial (Droganova and Zeman, 2019). Bender et al. (2015) argue that task- and domain-independent, automatically derivable methods to generate semantic representations would benefit the development of NLU systems, making them more comprehensive, consistent, and scalable. This can be achieved by using a compositional, linguistically-informed approach (Bender et al., 2015).\nThis paper focuses on the automatic derivation of semantic representations of a specific linguistic phenomenon which requires enrichment — Serial Verb Constructions (SVC). SVCs have been attested in numerous languages across West Africa, Central America, South-East Asia, and Oceania (Müller and Lipenkova, 2009). They can have a wide range of semantic interpretations, but the specific relationships between component verbs are not explicitly indicated by the surface forms; they are instead constrained by grammatical properties. By encoding these grammatical properties, we can get from the surface string to the semantic representation. Thai makes extensive use of SVCs – in Pongsutthi et al. (2013)’s study of 76 news articles (over 10,000 words) taken from the THAI-NEST corpus, 74.63% of the verb tokens were part of an SVC. Given their frequency, to successfully complete any NLU task for Thai, we must be able to deal with these constructions.\nThe analysis developed in this paper makes the implicit relationships between component verbs explicit, without the need for manual annotation. This implementation is the first step towards building an SVC library within the LinGO Grammar Matrix customization system (Bender et al., 2002, 2010; Zamaraeva et al., 2022), which will allow for efficient implementation of the phenomenon across typologically distinct languages.",
    "conclusion": "This paper has demonstrated how deep semantic representations of Thai SVCs can be automatically derived from syntactic properties of component verbs and the structure of the phrase as a whole. This was implemented into a computational grammar using an HPSG analysis, and tested against development and held-out sentences. I showed that this analysis can successfully account for Thai SVCs, increasing accuracy and reducing overgeneration and spurious ambiguity in both development and held-out data. This allows for the creation of richer, more precise semantic representations of Thai SVCs, which explicitly model the relationship between component verbs.\nThe LinGO Grammar Matrix (Bender et al., 2002, 2010; Zamaraeva et al., 2022) both draws on and supports typological work (Bender, 2016). Its goal is to combine typological research and syntactic analysis, allowing for both cross-linguistic generalizations and language-specific constraints, in order to map from surface strings to semantic representations (Bender, 2016). This analysis follows this approach, allowing for flexibility in argumentsharing, constituent structure, and verbal features used for derivation, while situated within the typological constraints presented in Section 2.1."
  },
  {
    "sample_id": 46,
    "title": "Automatic Gloss Dictionary for Sign Language Learners",
    "abstract": "A multi-language dictionary is a fundamental tool for language learning, allowing the learner to look up unfamiliar words. Searching an unrecognized word in the dictionary does not usually require deep knowledge of the target language. However, this is not true for sign language, where gestural elements preclude this type of easy lookup. This paper introduces GlossFinder, an online tool supporting 2, 000 signs to assist language learners in determining the meaning of given signs. Unlike alternative systems of complex inputs, our system requires only that learners imitate the sign in front of a standard webcam. A user study conducted among sign language speakers of varying ability compared our system against existing alternatives and the interviews indicated a clear preference for our new system. This implies that GlossFinder can lower the barrier in sign language learning by addressing the common problem of sign finding and make it accessible to the wider community.",
    "introduction": "Unlike most language systems, which are composed of their written and spoken forms, sign languages (e.g., American Sign Language (ASL) and the Australian Auslan language) used by the Deaf or Hard-of-Hearing (DHH) community are represented by the rich inputs including facial and gesture movements. As of the year 2020, 430 million people worldwide have developed hearing loss— that is, one in every ten people—and it is estimated that this number may increase to 700 million by 2050 (WHO, 2021). Sign languages are also used by people suffering the loss of ability to speak (e.g., aphasia) or brain stroke. They are spoken by individuals with various relational connections to sign language speakers, e.g., family members or co-workers. Additionally, a substantial and growing number of people are learning a sign language as a second language, e.g., among U.S. university students (Goldberg et al., 2015). Despite the efforts made in building tools to support their learning (Lee et al., 2005; Schioppo et al., 2019; Hou et al., 2019; Scassellati et al., 2018; Li et al., 2021), many sign language learners have limited means of seeking assistance, and are restricted to class offerings or relying on other experienced sign language speakers. It is therefore increasingly important to support the sign language learner community to facilitate better education and communication.\nAs a fundamental tool in language study, a dictionary is more than a tool to assist sign language learners in searching unfamiliar words. The rich content present in current online dictionaries (e.g., example pronunciation recordings and visual materials) also provide positive feedback to foster the learner’s understanding and proficiency in the target language (Corbeil and Archambault, 2006; Laska, 1993). Most existing sign language dictionaries (e.g., AslSearch (ASLSearch, 2009), Handspeak (Lapiak, 1995), and Signing Savvy (Signing Savvy, 2021)) are text-based and centered on one spoken language, with signs presented in an alphabetical order of their corresponding gloss, i.e., the spoken language counterpart. This does not serve the important scenario when someone encounters an unfamiliar sign and does not know its spoken language translation. Another issue with the text-based dictionary is the fact a one-toone correlation between sign and spoken language words does not always exist, and no standard convention exists for handling these discrepancies. The absence of these types of dictionary for sign language learner is due to the difficulty of processing visual input and the lack of intuitive alphabetics assumed in most language dictionaries. An early effort made towards a sign-centric dictionary is Tennant et al. (1998) where researchers use pre-defined handshapes (finger poses) to formalise the signs so that they can be arranged similar to a conventional dictionary. Follow-up work (Lapiak, 1995; Neidle et al., 2012; Alonzo et al., 2019) parameterised the signs by key properties (e.g., handshape, position of hands, and whether the sign involves repetitive movement) to make a filtering-based search system. In addition, Elliott et al. (2011) used the Microsoft Kinect to collect human body movements from the sign language speaker and match the performed signs against the database.\nModern advancements in deep learning algorithms enable processing of unstructured video inputs, and these algorithms have been applied to sign language. Progress has been made in identifying isolated (Li et al. (2020a,c); Albanie et al. (2020); Sincan and Keles (2020); Momeni et al. (2020)) or continuous signs (Li et al. (2020b); Zhou et al. (2021); Bull et al. (2021); Duarte et al. (2021); Chen et al. (2022)) from a video. This presents opportunity to develop a dictionary system, which accepts direct video inputs from a user performing a sign, and attempts to return the meaning of that sign. One of the early attempts on such videobased system is Alonzo et al. (2019) where the author discussed some characteristics in the design and evaluation metrics regarding the user satisfaction. Notably, the work did not build an actual automatic recognition technique and the users are only presented with a predetermined set of results during the study.\nIn this paper, we present the platform of GlossFinder, our new video-based sign dictionary, where users directly provide videos of the target sign by performing it to their webcam or via uploaded clips, and the system will retrieve matched signs without any extra input. To the best of our knowledge, it is the first attempt of user study with a functioning system built. The study identifies some key considerations in designing for this specific sign language context. It also verifies sign language learners’ frustration when using previous sign dictionaries, either due to the steep learning curve or the poor quality of results.",
    "conclusion": "We construct, to the best of our knowledge, the first automatic sign dictionary digesting direct video capture as its inputs. Our user study validates the improved usability from the new system. The participants describe it as less demanding to learn in comparison to the existing parameter-based systems. Retrieved results are said to be more accurate and able to accommodate the varying video capture quality. Enriched results include example videos and explanations are agreed to largely help the user in correctly locating and refining the search. Overall, the reported success rate in reaching the searched sign is on average 66% from GlossFinder, significantly surpassing the benchmarks. We also conduct analysis to compare different views for presenting the results. It is favored by the participants for the system to include more examples of varieties even at the cost that less glosses can be shown in a single page. Our study strengthens the belief that the sign language dictionary design should be visual-based to imitate the practical form of actual sign language teaching and learning. We hope it can also motivate the related research to make sign language learning increasingly accessible to a broader community.\nAs one of the early attempts in building such system, we notice some limitations in the current study:\n• The benchmark systems are comparatively weak, for which it is to blame the fact that sign language learners community is receiving insufficient support and no such stronger peers are public available. Existing systems are in majority made with voluntary contribution and limited in resource. While the incorporated benchmark systems are still receiving some positive feedback, stronger benchmarks are subject to encourage the participants to discover more places to improve in the current designs.\n• The target audience of this study is set to general sign language learners, which is in concept a larger community covering DHH. We recruit people of both intermediate and junior level of sign knowledge to collect plausible data. Yet future research may be framed to be more customized for the DHH community. Space may still remain to improve based on their need."
  },
  {
    "sample_id": 47,
    "title": "Automatic Keyphrase Extraction by Bridging Vocabulary Gap",
    "abstract": "Keyphrase extraction aims to select a set of terms from a document as a short summary of the document. Most methods extract keyphrases according to their statistical properties in the given document. Appropriate keyphrases, however, are not always statistically significant or even do not appear in the given document. This makes a large vocabulary gap between a document and its keyphrases. In this paper, we consider that a document and its keyphrases both describe the same object but are written in two different languages. By regarding keyphrase extraction as a problem of translating from the language of documents to the language of keyphrases, we use word alignment models in statistical machine translation to learn translation probabilities between the words in documents and the words in keyphrases. According to the translation model, we suggest keyphrases given a new document. The suggested keyphrases are not necessarily statistically frequent in the document, which indicates that our method is more flexible and reliable. Experiments on news articles demonstrate that our method outperforms existing unsupervised methods on precision, recall and F-measure.",
    "introduction": "Information on the Web is emerging with the development of Internet. It is becoming more and more important to effectively search and manage information. Keyphrases, as a brief summary of a document, provide a solution to help organize and\n∗ Zhiyuan Liu and Xinxiong Chen have equal contribution\nretrieve documents, which have been widely used in digital libraries and information retrieval (Turney, 2000; Nguyen and Kan, 2007). Due to the explosion of information, it is ineffective for professional human indexers to manually annotate documents with keyphrases. How to automatically extract keyphrases from documents becomes an important research problem, which is usually referred to as keyphrase extraction.\nMost methods for keyphrase extraction try to extract keyphrases according to their statistical properties. These methods are susceptible to low performance because many appropriate keyphrases may not be statistically frequent or even not appear in the document, especially for short documents. We name the phenomenon as the vocabulary gap between documents and keyphrases. For example, a research paper talking about “machine transliteration” may less or even not mention the phrase “machine translation”. However, since “machine transliteration” is a sub-field of “machine translation”, the phrase “machine translation” is also reasonable to be suggested as a keyphrase to indicate the topics of this paper. Let us take another example: in a news article talking about “iPad” and “iPhone”, the word “Apple” may rarely ever come up. However, it is known that both “iPad” and “iPhone” are the products of “Apple”, and the word “Apple” may thus be a proper keyphrase of this article.\nWe can see that, the essential challenge of keyphrase extraction is the vocabulary gap between documents and keyphrases. Therefore, the task of keyphrase extraction is how to capture the semantic relations between the words in documents and in keyphrases so as to bridge the vocabulary gap. In this paper, we provide a new perspective to documents and their keyphrases: each document and its keyphrases are descriptions to the same object, but the document is written using one language, while keyphrases are written using another language. Therefore, keyphrase extraction can be regarded as a translation problem from the language of documents into the language of keyphrases.\nBased on the idea of translation, we use word alignment models (WAM) (Brown et al., 1993) in statistical machine translation (SMT) (Koehn, 2010) and propose a unified framework for keyphrase extraction: (1) From a collection of translation pairs of two languages, WAM learns translation probabilities between the words in the two languages. (2) According to the translation model, we are able to bridge the vocabulary gap and succeed in suggesting appropriate keyphrases, which may not necessarily frequent in their corresponding documents.\nAs a promising approach to solve the problem of vocabulary gap, SMT has been widely exploited in many applications such as information retrieval (Berger and Lafferty, 1999; Karimzadehgan and Zhai, 2010), image and video annotation (Duygulu et al., 2002), question answering (Berger et al., 2000; Echihabi and Marcu, 2003; Murdock and Croft, 2004; Soricut and Brill, 2006; Xue et al., 2008), query expansion and rewriting (Riezler et al., 2007; Riezler et al., 2008; Riezler and Liu, 2010), summarization (Banko et al., 2000), collocation extraction (Liu et al., 2009b; Liu et al., 2010b) and paraphrasing (Quirk et al., 2004; Zhao et al., 2010). Although SMT is a widely adopted solution to vocabulary gap, for various applications using SMT, the crucial and non-trivial problem is to find appropriate and enough translation pairs for SMT.\nThe most straightforward translation pairs for keyphrase extraction is document-keyphrase pairs. In practice, however, it is time-consuming to annotate a large collection of documents with keyphrases for sufficient WAM training. In order to solve the problem, we use titles and summaries to build translation pairs with documents. Titles and summaries are usually accompanying with the corresponding documents. In some special cases, titles or summaries may be unavailable. We are also able to extract one or more important sentences from the corresponding documents to construct sufficient translation pairs.",
    "conclusion": "In this paper, we provide a new perspective to keyphrase extraction: regarding a document and its keyphrases as descriptions to the same object written in two languages. We use IBM Model-1 to bridge the vocabulary gap between the two languages for keyphrase generation. We explore various methods to construct translation pairs. Experiments show that our method can capture the semantic relations between words in documents and keyphrases. Our method is also language-independent, which can be performed on documents in any languages.\nWe will explore the following two future work: (1) Explore our method on other types of articles and on other languages. (2) Explore more complicated methods to extract important sentences for constructing translation pairs."
  },
  {
    "sample_id": 48,
    "title": "Automatic Prompt Optimization with “Gradient Descent” and Beam Search",
    "abstract": "Large Language Models (LLMs) have shown impressive performance as general purpose agents, but their abilities remain highly dependent on prompts which are hand written with onerous trial-and-error effort. We propose a simple and nonparametric solution to this problem, Prompt Optimization with Textual Gradients (ProTeGi), which is inspired by numerical gradient descent to automatically improve prompts, assuming access to training data and an LLM API. The algorithm uses minibatches of data to form natural language “gradients” that criticize the current prompt, much like how numerical gradients point in the direction of error ascent. The natural language gradients are then “propagated” into the prompt by editing the prompt in the opposite semantic direction of the gradient. These gradient descent steps are guided by a beam search and bandit selection procedure which significantly improves algorithmic efficiency. Preliminary results across three benchmark NLP tasks and the novel problem of LLM jailbreak detection suggest that Automatic Prompt Optimization can outperform prior prompt editing techniques and improve an initial prompt’s performance by up to 31%, by using data to rewrite vague task descriptions into more precise annotation instructions.1",
    "introduction": "Large Language Models (LLMs) trained on webscale text have recently demonstrated unprecedented abilities across a variety of NLP tasks (OpenAI, 2023; Bubeck et al., 2023). These LLMs use prompt inputs to follow human instructions. Writing prompts in natural language remains a manual trial-and-error process requiring significant human effort (Jiang et al., 2022) and expertise (Reynolds and McDonell, 2021; Zamfirescu-Pereira et al.,\nFigure 1: Overview of the proposed Prompt Optimization with Textual Gradients (ProTeGi).\nAccordingly, there is need for automatic or semiautomatic procedures to help humans write the best prompts. This would help reduce manual effort, improve task performance, and produce interpretable descriptions of a cognitive decision process.\nA recent body of work has investigated this problem by training auxiliary models or differentiable representations of the prompt (Qin and Eisner, 2021; Deng et al., 2022). However, such works assume access to internal state variables of the LLM (Shin et al., 2020; Lester et al., 2021) while practitioners often communicate with LLMs through an API. Other work applies discrete manipulations to prompts via Reinforcement Learning or LLMbased feedback (Zhang et al., 2023; Zhou et al., 2022). These algorithms may also require low-level access to the LLM, produce incomprehensible outputs, or rely on directionless monte-carlo search over the semantic space of prompts.\nWe propose Prompt Optimization with Textual Gradients (ProTeGi), a general purpose and nonparametric algorithm for automatic prompt optimization that connects these two bodies of research by applying discrete improvements to prompts in a directed way.\nUnlike prior work, we overcome the discrete optimization barrier by mirroring the steps of gradient descent within a text-based Socratic dialogue (Zeng et al., 2022), substituting differentiation with LLM feedback and backpropagation with LLM editing. In detail, we use minibatches of training data to produce “gradients” in natural language, i.e., descriptions of the current prompts’ flaws with respect to the minibatch, then edit the current prompt in the opposite semantic direction of the gradient. These steps become the expansion part of a wider beam search over the space of prompts, increasing algorithmic efficiency by treating the problem of beam candidate selection as an instance of the best arm identification problem (Audibert et al., 2010).\nWe then offer a preliminary case study of ProTeGi. We evaluate the proposed framework in multiple configurations across 4 NLP tasks, including the novel problem of LLM jailbreak detection. The results suggest that the proposed algorithm can improve on the performance of the initial prompt input by up to 31%, exceeding state-of-the-art prompt learning baselines by an average of 4-8% while relying on fewer LLM API calls. We also demonstrate the interpretability of the optimization process and investigate the algorithms’ shortcomings.",
    "conclusion": "In this paper, we proposed Prompt Optimization with Textual Gradients (ProTeGi), a simple and general-purpose framework for the automatic optimization of LLM prompts. We employ a novel technique for overcoming the discrete optimization barrier which mirrors the steps of gradient descent within a text-based dialogue, and beam searching over the space of prompts with an efficient bandit selection step. Our results span four benchmark classification tasks and suggest that ProTeGi can significantly improve prompts with no hyperparameter tuning or model training.\nThere are many directions for future work, including generalizing the technique to more tasks with new metric functions, incorporating step sizes into the learning process, and expanding the conceptual framework of textual gradient descent."
  },
  {
    "sample_id": 49,
    "title": "AutoNLU: An On-demand Cloud-based Natural Language Understanding System for Enterprises",
    "abstract": "With the renaissance of deep learning, neural networks have achieved promising results on many natural language understanding (NLU) tasks. Even though the source codes of many neural network models are publicly available, there is still a large gap from open-sourced models to solving real-world problems in enterprises. Therefore, to fill this gap, we introduce AUTONLU, an on-demand cloud-based system with an easy-to-use interface that covers all common use-cases and steps in developing an NLU model. AUTONLU has supported many product teams within Adobe with different use-cases and datasets, quickly delivering them working models. To demonstrate the effectiveness of AUTONLU, we present two case studies. i) We build a practical NLU model for handling various image-editing requests in Photoshop. ii) We build powerful keyphrase extraction models that achieve stateof-the-art results on two public benchmarks. In both cases, end users only need to write a small amount of code to convert their datasets into a common format used by AUTONLU.",
    "introduction": "In recent years, many deep learning methods have achieved impressive results on a wide range of tasks, ranging from question answering (Seo et al., 2017; Lai et al., 2018b) to named entity recognition (NER) (Lin et al., 2019; Jiang et al., 2019) to intent detection and slot filling (Wang et al., 2018; Chen et al., 2019). Even though the source codes of many models are publicly available, going from an open-sourced implementation of a model for a public dataset to a production-ready model for an inhouse dataset is not a simple task. Furthermore, in an enterprise, only few engineers are familiar with deep learning research and frameworks. Therefore, to facilitate the development and adoption of deep learning models within Adobe, we introduce a new system named AUTONLU. It is an on-demand cloud-based system that enables multiple users to create and edit datasets and to train and test different state-of-the-art NLU models. AUTONLU’s main principles are:\n• Ease of use. AUTONLU aims to help users with limited technical knowledge to train and test models on their datasets. We provide GUI modules to accommodate the most common use-cases, from creating/cleaning a dataset to training/evaluating/debugging a model.\n• State-of-the-art models. Users should not sacrifice performance for ease-of-use. Our built-in models provide state-of-the-art performance on multiple public datasets. AUTONLU also supports hyperparameter tuning using grid search, allowing users to fine-tune the models even further.\n• Scalability. AUTONLU aims to be deployed in enterprises where computing costs could be a limiting factor. We provide an on-demand architecture so that the system could be utilized as much as possible.\nAt Adobe, AUTONLU has been used to train NLU models for different product teams, ranging from Photoshop to Document Cloud. To demonstrate the effectiveness of AUTONLU, we present two case studies. i) We build a practical NLU model for handling various image-editing requests in Photoshop. ii) We build powerful keyphrase extraction models that achieve state-of-the-art results on two public benchmarks. In both cases, end users only need to write a small amount of code to convert their datasets into a common format used by AUTONLU.",
    "conclusion": "In this work, we introduce AUTONLU, an ondemand cloud-based platform that is easy-to-use and has enabled many product teams within Adobe to create powerful NLU models. Our design principles make it an ideal candidate for enterprises who want to have an NLU system for themselves, with minimal deep learning expertise. AUTONLU ’s code is in the process to be open-sourced, and we invite contributors to contribute. In future work, we will implement more advanced features such as transfer learning, knowledge distillation and neural architecture search, which have been shown to be useful in building real-world NLP systems (Lai et al., 2018a; Jiang et al., 2019; Lai et al., 2019, 2020; Klyuchnikov et al., 2020). Furthermore, we will extend our system to have more advanced analytics features (Murugesan et al., 2019), and to better support other languages (Nguyen and Nguyen, 2020)."
  },
  {
    "sample_id": 50,
    "title": "AVEN-GR: Attribute Value Extraction and Normalization using product GRaphs",
    "abstract": "Getting a good understanding of the user intent is vital for e-commerce applications to surface the right product to a given customer query. Query Understanding (QU) systems are essential for this purpose, and many e-commerce providers are working on complex solutions that need to be data efficient and able to capture early emerging market trends. Query Attribute Understanding (QAU) is a sub-component of QU that involves extracting named attributes from user queries and linking them to existing e-commerce entities such as brand, material, color, etc. While extracting named entities from text has been extensively explored in the literature, QAU requires specific attention due to the nature of the queries, which are often short, noisy, ambiguous, and constantly evolving. This paper makes three contributions to QAU. First, we propose a novel end-to-end approach that jointly solves Named Entity Recognition (NER) and Entity Linking (NEL) and enables open-world reasoning for QAU. Second, we introduce a novel method for utilizing product graphs to enhance the representation of query entities. Finally, we present a new dataset constructed from public sources that can be used to evaluate the performance of future QAU systems.",
    "introduction": "Search queries are the main point of interaction between the customer and the search system. As such, extracting information from the queries is pivotal in surfacing the relevant products, making the task directly responsible for the quality of the overall customer experience. Query Understanding (QU) not only inherits all the challenges of standard natural language understanding but poses additional difficulties: queries are short and lack context, which makes them challenging to understand. They often contain implicit knowledge that is difficult to capture without external reference. For example, the query \"M2 laptop\" refers to Apple laptops since M2 processors are only sold by Apple. Furthermore, customers do not have technical writing skills, which can result in queries that are noisy or use inappropriate search terms.\nIn this work, we focus on the task of Query Attribute Understanding (QAU), which aims to extract the attribute values from the queries and make them usable for other downstream applications in the Search Engine (see fig. 1). QAU is related to another important task, Document Attribute Understanding (DAU), which aims to extract attributes from product descriptions. DAU has received significant attention from the community in the past years ((Zheng et al., 2018; Xu et al., 2019; Dong et al., 2020; Karamanolakis et al., 2020)) and does not suffer from the difficulties mentioned above and that are specific to queries. Both QAU and DAU are specific instances of Named Entity Recognition and Linking (NER/NEL), which aims to extract typed mentions from text. However, in contrast to classic NER, which usually handles fewer attribute types (such as Person, Location, and Organization), QAU and DAU deal with a larger number of attribute types (which can reach thousands in e-commerce as noted in (Xu et al., 2019)).\nWe claim that three critical elements need to be addressed to get a practical solution to QAU. Firstly, named entity recognition should be performed jointly with entity linking, in order to map the detected entities to our knowledge base. Solving these tasks separately is not practical in an industrial context, as it leads to error propagation (linking module cannot make up for a wrong attribute prediction by the NER module) and more generally hidden technical debt (see (Sculley et al., 2015)). Furthermore, separating the tasks precludes the possibility of inductive transfer, which has been shown to be crucial in related tasks (Zhang and Yang, 2021; Caruana, 1997; Ruder, 2017).\nFigure 1: Overview of the task. We ultimately want to detect that this query contains three mentions: (brown chocolate), (boot) and (suede). The first annotation row shows the ground truth for the attribute value extraction task, while the second one shows that of the normalization step, which may be understood as entity linking over the detected mentions.\nSecondly, product graphs (PG) are becoming a new standard to represent e-commerce concepts and the relations between searchable products. Therefore, QAU systems should be able to leverage this new source of knowledge to improve their performance. Finally, QAU systems should always be designed with an \"open-world\" setup in mind to deal dynamically with new concepts. For instance, if we consider the query ‘Sony A95K TV’, we should be able to detect that ‘A95K’ is a mention representing a product line even if this product does not exist in our knowledge base.\nNote that extreme classification (Jain et al., 2016) is a possible alternative to classic NER/NER stacking, but it does not consider the coarse-grained nature of attributes (entities belong to different attribute types) and does not easily take into account the open-world nature of the task. Users can search for attribute values that are not yet in the knowledge base or not associated with any known product, making it difficult to predict normalized attribute values directly.",
    "conclusion": "In this paper, we introduced a novel approach to tackle QAU in a multi-task fashion. We demonstrated its effectiveness on two datasets, compared to some simple baselines. However, further ablation studies on more datasets / baselines (e.g. Ayoola et al. 2022) are necessary to assess its generalization power. Additionally, future work will focus on improving the multitasking efficiency of AVEN, for instance by implementing (Chen et al., 2018)."
  },
  {
    "sample_id": 51,
    "title": "A Comprehensive Analysis of Preprocessing for Word Representation Learning in Affective Tasks",
    "abstract": "Affective tasks such as sentiment analysis, emotion classification and sarcasm detection have been popular in recent years due to abundance of user-generated data, accurate computational linguistic models, and broad range of relevant applications in various domains. At the same time, many studies have highlighted the importance of text preprocessing, as an integral step to any natural language processing prediction model and downstream task. While preprocessing in affective systems is well-studied, preprocessing in word vector based models applied to affective systems, is not. To address this limitation, we conduct a comprehensive analysis of the role of preprocessing techniques in affective analysis based on word vector models. Our analysis is the first of its kind and provides useful insights of the importance of each preprocessing technique when applied at the training phase, commonly ignored in pretrained word vector models, and/or at the downstream task phase.",
    "introduction": "Affective tasks such as sentiment analysis, emotion classification and sarcasm detection have enjoyed great popularity in recent years. This success can be largely attributed to the fundamental and straightforward nature of the methods employed, the availability of vast amounts of user-generated natural language data, and the wide range of useful applications, spanning from hate speech detection to monitoring the sentiment of financial markets and news recommendation (Djuric et al., 2015; Babanejad et al., 2019). Most early models of affect analysis employed pretrained word embeddings that have been obtained under the assumption of the distributional hypothesis (Mikolov et al., 2013; Devlin et al., 2018). The distributional hypothesis suggests that two words occurring frequently in similar linguistic contexts tend to be more semantically similar, and therefore should be represented closer to one another in the embedding space. However, while such embeddings are useful for several natural language processing (NLP) downstream tasks, they are known to be less suitable for affective tasks in particular (Tang et al., 2014; Agrawal et al., 2018). Although some authors claim that there is a need for post-processing word embeddings for affective tasks, others find that off-theshelf vectors are very powerful for affective lexicon learning (Lison and Kutuzov, 2017). For example, word2vec (Mikolov et al., 2013) estimates the pair of words ‘happy’ and ‘sad’ to be more similar than the pair of words ‘happy’ and ‘joy’, which is counterintuitive, and might affect the accuracy performance of the models that depend on it.\nTo address the limitations of traditional word embeddings, several techniques have been proposed, including task-specific fine-tuning (Devlin et al., 2018), retrofitting (Faruqui et al., 2014), representing emotion with vectors using a multi-task training framework (Xu et al., 2018) and generating affective word embeddings (Felbo et al., 2017), to name a few. Other attempts to overcome the limitation of word vectors include optimization of hyperparameters (Levy et al., 2015), as well as fine-tuned preprocessing strategies tailored to different NLP tasks. While these strategies have demonstrated evidence of improving the accuracy performance in tasks such as word similarity, word analogy, and others (Lison and Kutuzov, 2017), their effect in affective tasks has not received considerable attention and remains less explored. Our work is motivated by the observation that preprocessing factors such as stemming, stopwords removal and many others make up an integral part of nearly every improved text classification model, and affective systems in particular (Danisman and Alpkocak, 2008; Patil and Patil, 2013). However, little work has been\nFigure 1: Framework of applying preprocessing in different stages in affective systems; (a) Pre, (b) Post.\ndone towards understanding the role of preprocessing techniques applied to word embeddings in different stages of affective systems. To address this limitation, the overarching goal of this research, is to perform an extensive and systematic assessment of the effect of a range of linguistic preprocessing factors pertaining to three affective tasks, including sentiment analysis, emotion classification and sarcasm detection. Towards that end, we systematically analyze the effectiveness of applying preprocessing to large training corpora before learning word embeddings, an approach that has largely been overlooked by the community. We investigate the following research questions: (i) what is the effect of integrating preprocessing techniques earlier into word embedding models, instead of later on in a downstream classification models? (ii) which preprocessing techniques yield the most benefit in affective tasks? (iii) does preprocessing of word embeddings provide any improvement over stateof-the-art pretrained word embeddings? and if yes, how much?\nFigure 1 illustrates the difference between a) preprocessing word embeddings pipeline (Pre) vs. b) preprocessing classification dataset pipeline (Post), where preprocessing techniques in (a) are applied to the training corpus of the model and in (b) only to the classification dataset. In brief, the main contributions of our work are as follows:\n• We conduct a comprehensive analysis of the role of preprocessing techniques in affective tasks (including sentiment analysis, emotion classification and sarcasm detection), employing different models, over nine datasets;\n• We perform a comparative analysis of the accuracy performance of word vector models when preprocessing is applied at the training phase (training data) and/or at the downstream task phase (classification dataset). Interestingly, we obtain best results when preprocessing is applied only to the training corpus or when it is applied to both the training corpus and the classification dataset of interest.\n• We evaluate the performance of our best preprocessed word vector model against state-ofthe-art pretrained word embedding models;\n• We make source code and data publicly available to encourage reproducibility of results1.\nThe rest of the paper is organized as follows: Section 2 presents an overview of the related work. Section 3 elaborates on the preprocessing techniques employed in the evaluation of models. Section 4 describes the experimental evaluation framework. In Section 5 a comprehensive analysis of the results is provided. Section 6 concludes the paper with key insights of the research.",
    "conclusion": "We systematically examined the role of preprocessing training corpora used to induce word representations for affect analysis. While all preprocessing techniques improved performance to a certain ex-\nFigure 2: Absolute F-scores vs. relative improvement\ntent, our analysis suggests that the most noticeable increase is obtained through negation processing (neg). The overall best performance is achieved by applying all the preprocessing techniques, except stopwords removal (All-stop). Interestingly, incorporating preprocessing into word representations appears to be far more beneficial than applying it in a downstream task to classification datasets. Moreover, while all the three affective tasks (sentiment analysis, sarcasm detection and emotion classification) benefit from our proposed preprocessing framework, our analysis reveals that the multiclass emotion classification task benefits the most. Exploring the space of subsets of our preprocessing factors might yield more interesting combinations; we leave this for future work."
  },
  {
    "sample_id": 52,
    "title": "A Diffusion Weighted Graph Framework for New Intent Discovery",
    "abstract": "New Intent Discovery (NID) aims to recognize both new and known intents from unlabeled data with the aid of limited labeled data containing only known intents. Without considering structure relationships between samples, previous methods generate noisy supervisory signals which cannot strike a balance between quantity and quality, hindering the formation of new intent clusters and effective transfer of the pre-training knowledge. To mitigate this limitation, we propose a novel Diffusion Weighted Graph Framework (DWGF) to capture both semantic similarities and structure relationships inherent in data, enabling more sufficient and reliable supervisory signals. Specifically, for each sample, we diffuse neighborhood relationships along semantic paths guided by the nearest neighbors for multiple hops to characterize its local structure discriminately. Then, we sample its positive keys and weigh them based on semantic similarities and local structures for contrastive learning. During inference, we further propose Graph Smoothing Filter (GSF) to explicitly utilize the structure relationships to filter high-frequency noise embodied in semantically ambiguous samples on the cluster boundary. Extensive experiments show that our method outperforms state-of-the-art models on all evaluation metrics across multiple benchmark datasets. Code and data are available at https://github.com/yibai-shi/DWGF.",
    "introduction": "Even though current machine learning methods have achieved superior performance on many NLP tasks, they often fail to meet application requirements in an open-world environment. For instance, general intent classification models trained on predefined intents cannot recognize new intents from unlabeled dialogues, which is a clear obstacle for real-world applications. Therefore, research on\nFigure 1: Illustration of the transformation of supervisory signal generation method. Bottom Left: generating supervisory signals indiscriminately along all directions of the hypersphere, which is sensitive to threshold changing. Top: an example of selecting samples with semantic paths. Bottom Right: generating supervisory signals directionally with structure relationships composed of multiple semantic paths in a relaxed feature hypersphere.\nNew Intent Discovery (NID), which aims to discover new intents from unlabeled data automatically, has attracted much attention recently.\nMost existing NID methods (Lin et al., 2020; Zhang et al., 2021; Wei et al., 2022; Zhang et al., 2022; An et al., 2023) adopt a two-stage training strategy: pre-training on labeled data, then learning clustering-friendly representation with pseudo supervisory signals. However, previous methods only rely on semantic similarities to generate supervisory signals based on the assumption that samples within the feature hypersphere belong to the same category as the hypersphere anchor, e.g. cluster centroids (Zhang et al., 2021), class prototypes (An et al., 2022b), or query samples (Zhang et al., 2022).\nEven though these methods can learn some discriminative features, they still face limitations in generating both adequate and reliable supervisory signals, which we call the Quantity and Quality Dilemma. Specifically, as shown in Fig.1 Bottom Left, these methods rely on a fixed threshold to determine the search radius of the hypersphere. Shrinking the threshold (blue solid line) helps retrieve more accurate positive keys, but it loses information from positive keys out of the hypersphere, resulting in a low recall. However, simply relaxing the threshold (red dashed line) will introduce much noise and lead to low accuracy.\nQuantity and Quality Dilemma is caused by the fact that the previous methods searched positive keys indiscriminately along all directions of the hypersphere with a fixed search radius. In order to selectively sample both adequate and reliable positive keys to ensure the formation of new intent clusters, we propose to model and utilize structure relationships inherent in data, which reflect the semantic correlations between samples from the perspective of connectivity. As shown in Fig.1 Top, for each sample, we first initialize its k-nearest neighbors with a tightened threshold. Then we connect any two samples if they have at least one shared neighbor since the semantics of the shared neighbor are highly correlated with the samples on both sides. According to this rule, we identify two samples (with brown borders in Fig.1 Top) that can be used as bridges and diffuse the anchor along them to search positive keys near the boundary of the hypersphere, forming the final semantic path. In the case of the same semantic similarity, we additionally require the positive keys to appear on the semantic paths diffused from the anchor.\nIn this paper, we propose a novel Diffusion Weighted Graph Framework to model and utilize structure relationships. Specifically, from any anchor, we diffuse neighborhood relationships along the nearest neighbor-guided semantic paths for multiple hops to construct the final DWG. As shown in Fig.1 Bottom Right, then we sample positive keys along the semantic paths (arrow lines) in DWG within the relaxed feature hypersphere. Moreover, sampled keys are assigned to different contrastive weights according to their frequency of being sampled on different semantic paths, where keys that are diffused repeatedly from different outsets will accumulate larger values and vice versa. We conduct contrastive learning with sampled positive keys and corresponding weights in the embedding space. Apart from considering the sample-sample structure relationships from the local view, we adopt the idea of Xie et al. (2016) to help learn clustering-friendly representations from the global view through self-training.\nDuring the inference stage, in order to filter highfrequency noise embodied in the semantically ambiguous samples on the cluster boundary, we propose a novel inference improvement Graph Smoothing Filter (GSF), which utilizes normalized graph Laplacian to aggregate neighborhood information revealed by structure relationships of testing samples. Smoothed testing features help to obtain better clustering results.\nOur main contributions can be summarized as follows:\n• We propose a Diffusion Weighted Graph Framework (DWGF) for NID, which can capture both semantic similarities and structure relationships inherent in data to generate adequate and reliable supervisory signals.\n• We improve inference through Graph Smoothing Filter (GSF), which exploits structure relationships to correct semantically ambiguous samples explicitly.\n• We conduct extensive experiments on multiple benchmark datasets to verify the effectiveness.",
    "conclusion": "In this paper, we propose a novel Diffusion Weighted Graph Framework (DWGF) for new intent discovery, which models structure relationships inherent in data through nearest neighborguided diffusion. Combined with structure relationships, we improve both the sampling and weighting strategy in contrastive learning and adopt supervision from local and global views. We further propose Graph Smoothing Filter (GSF) to explore the potential of structure relationships in inference, which effectively filters noise embodied in semantically ambiguous samples on the cluster boundary. Extensive experiments on all three clustering metrics across multiple benchmark datasets fully validate the effectiveness and robustness of our method."
  },
  {
    "sample_id": 53,
    "title": "A Girl Has A Name: Detecting Authorship Obfuscation",
    "abstract": "Authorship attribution aims to identify the author of a text based on the stylometric analysis. Authorship obfuscation, on the other hand, aims to protect against authorship attribution by modifying a text’s style. In this paper, we evaluate the stealthiness of state-of-the-art authorship obfuscation methods under an adversarial threat model. An obfuscator is stealthy to the extent an adversary finds it challenging to detect whether or not a text modified by the obfuscator is obfuscated – a decision that is key to the adversary interested in authorship attribution. We show that the existing authorship obfuscation methods are not stealthy as their obfuscated texts can be identified with an average F1 score of 0.87. The reason for the lack of stealthiness is that these obfuscators degrade text smoothness, as ascertained by neural language models, in a detectable manner. Our results highlight the need to develop stealthy authorship obfuscation methods that can better protect the identity of an author seeking anonymity.",
    "introduction": "Authorship attribution aims to identify the author of a text using stylometric techniques designed to capitalize on differences in the writing style of different authors. Owing to recent advances in machine learning, authorship attribution methods can now identify authors with impressive accuracy (Abbasi and Chen, 2008) even in challenging settings such as cross-domain (Overdorf and Greenstadt, 2016) and at a large-scale (Narayanan et al., 2012; Ruder et al., 2016). Such powerful authorship attribution methods pose a threat to privacyconscious users such as journalists and activists who may wish to publish anonymously (Times, 2018; Anonymous, 2018).\nAuthorship obfuscation, a protective countermeasure, aims to evade authorship attribution by obfuscating the writing style in a text. Since it is challenging to accomplish this manually, researchers have developed automated authorship obfuscation methods that can evade attribution while preserving semantics (PAN, 2018). However, a key limitation of prior work is that authorship obfuscation methods do not consider the adversarial threat model where the adversary is “obfuscation aware” (Karadzhov et al., 2017; Potthast et al., 2018; Mahmood et al., 2019). Thus, in addition to evading attribution and preserving semantics, it is important that authorship obfuscation methods are “stealthy” – i.e., they need to hide the fact that text was obfuscated from the adversary.\nIn this paper, we investigate the stealthiness of state-of-the-art authorship obfuscation methods. Our intuition is that the application of authorship obfuscation results in subtle differences in text smoothness (as compared to human writing) that can be exploited for obfuscation detection. To capitalize on this intuition, we use off-theshelf pre-trained neural language models such as BERT and GPT-2 to extract text smoothness features in terms of word likelihood. We then use these as features to train supervised machine learning classifiers. The results show that we can accurately detect whether or not a text is obfuscated.\nOur findings highlight that existing authorship obfuscation methods themselves leave behind stylistic signatures that can be detected using neural language models. Our results motivate future research on developing stealthy authorship obfuscation methods for the adversarial threat model where the adversary is obfuscation aware.\nOur key contributions are as follows:\n• We study the problem of obfuscation detection for state-of-the-art authorship obfuscation methods. This and the underlying property of stealthiness has been given scant attention in the literature. We also note that this problem is potentially more challenging than the related one of synthetic text detection since most of the original text can be retained during obfuscation.\n• We explore 160 distinct BERT and GPT-2 based neural language model architectures designed to leverage text smoothness for obfuscation detection.\n• We conduct a comprehensive evaluation of these architectures on 2 different datasets. Our best architecture achieves F1 of 0.87, on average, demonstrating the serious lack of stealthiness of existing authorship obfuscation methods.\nPaper Organization: The rest of this paper proceeds as follows. Section 2 summarizes related work on authorship obfuscation and obfuscation detection. Section 3 presents our proposed approach for obfuscation detection using neural language models. Section 4 presents details of our experimental setup including the description of various authorship obfuscation and obfuscation detection methods. We present the experimental results in Section 5 before concluding. The relevant source code and data are available at https://github.com/asad1996172/ Obfuscation-Detection.",
    "conclusion": "In this paper, we showed that the state-of-the-art authorship obfuscation methods are not stealthy. We showed that the degradation in text smoothness caused by authorship obfuscators allow a detector to distinguish between obfuscated documents and original documents. Our proposed\nFigure 4: Comparison between different obfuscators and original documents on the basis of average sorted probabilities extracted by BERT BASE for EBG obfuscated dataset.\nobfuscation detectors were effective at classifying obfuscated and evaded documents (F1 score as high as 0.92 and 0.95, respectively). Our findings point to future research opportunities to build stealthy authorship obfuscation methods. We suggest that obfuscation methods should strive to preserve text smoothness in addition to semantics."
  },
  {
    "sample_id": 54,
    "title": "A Gradually Soft Multi-Task and Data-Augmented Approach to Medical Question Understanding",
    "abstract": "Users of medical question answering systems often submit long and detailed questions, making it hard to achieve high recall in answer retrieval. To alleviate this problem, we propose a novel Multi-Task Learning (MTL) method with data augmentation for medical question understanding. We first establish an equivalence between the tasks of question summarization and Recognizing Question Entailment (RQE) using their definitions in the medical domain. Based on this equivalence, we propose a data augmentation algorithm to use just one dataset to optimize for both tasks, with a weighted MTL loss. We introduce gradually soft parameter-sharing: a constraint for decoder parameters to be close, that is gradually loosened as we move to the highest layer. We show through ablation studies that our proposed novelties improve performance. Our method outperforms existing MTL methods across 4 datasets of medical question pairs, in ROUGE scores, RQE accuracy and human evaluation. Finally, we show that our method fares better than single-task learning under 4 low-resource settings.",
    "introduction": "In order to retrieve relevant answers, one of the basic steps in Question Answering (QA) systems is understanding the intent of questions (Chen et al., 2012; Cai et al., 2017). This is particularly important for medical QA systems (Wu et al., 2020), as consumer health questions – questions asked by patients – may use a vocabulary distinct from doctors to describe similar health concepts (Ben Abacha and Demner-Fushman, 2019a). Consumer health questions may also contain peripheral information like patient history (Roberts and Demner-Fushman, 2016), that are not necessary to answer questions. There is a growing number of approaches to medical question understanding, including query relax-\nCHQ. Our method learns from the task of Recognizing Question Entailment to generate more informative summaries compared to the baseline.\nation (Ben Abacha and Zweigenbaum, 2015; Lei et al., 2020), question entailment (Ben Abacha and Demner-Fushman, 2016, 2019b; Agrawal et al., 2019), question summarization (Ben Abacha and Demner-Fushman, 2019a), and question similarity (Ben Abacha and Demner-Fushman, 2017; Yan and Li, 2018; McCreery et al., 2019).\nMedical question summarization is the task of summarizing consumer health questions into short, single-sentence questions that capture essential information needed to give a correct answer. The task of Recognizing Question Entailment (RQE) is defined by Ben Abacha and Demner-Fushman (2016) in the medical domain as a binary classification task. For the purpose of this task, a first question is considered to entail a second one if and only if every answer to the second question is a correct, and either full or partial answer to the first question.\nWe find in initial experiments (Mrini et al., 2021b) that RQE can teach question summarizers to distinguish salient information from peripheral details, and likewise that question summarization can benefit RQE classifiers. In our setting, we cast the medical question understanding task as a Multi-\nTask Learning (MTL) problem involving the two tasks of question summarization and Recognizing Question Entailment. We use a simple sum of learning objectives in Mrini et al. (2021b). In this paper, we introduce a novel, gradually soft multi-task and data-augmented approach to medical question understanding.1\nPrevious work on combining summarization and entailment uses at least 2 datasets – 1 from each task (Pasunuru et al., 2017; Guo et al., 2018). We first establish an equivalence between both tasks. This equivalence is the inspiration behind the data augmentation schemes introduced in our previous work (Mrini et al., 2021b). The goal of the data augmentation is to use a single dataset for MultiTask Learning. We propose to use a weighted loss function to simultaneously optimize for both tasks. Then, we propose a gradually soft parametersharing MTL approach. We conduct ablation studies to show that our two novelties – data augmentation and gradually soft parameter-sharing – improve performance in both tasks.\nOur proposed gradually soft multi-task and dataaugmented approach outperforms existing singletask and multi-task learning methods on architectures achieving state-of-the-art results in abstractive summarization. Compared to single-task learning, our approach achieves a 12% increase in accuracy on a medical RQE dataset, and an average increase of 3.5% in ROUGE-1 F1 scores across 3 medical question summarization datasets. Additionally, we perform human evaluation and find our approach generates more informative summarized questions. Finally, we find that our approach is more efficient at leveraging smaller amounts of data, and yields better performance under 4 low-resource settings.",
    "conclusion": "We propose a novel multi-task learning approach for medical question understanding. Our approach trains on the tasks of RQE and question summarization in a simultaneous, weighted MTL loss function, where we add a loss term to constrain the decoder layers to be close, and we loosen the constraint gradually as we move higher up the layers. We show using the definitions of both tasks in the medical domain that we can augment datasets, such that we only need one dataset for MTL. Our two ablation studies show that our gradually soft parameter-sharing and our data augmentation algorithm each increase performance individually. We compare our method to single-task learning and existing MTL work, and show improvements across 3 medical question summarization datasets and 1 medical RQE dataset. Finally, we test our approach under low-resource settings: we find that it is able to efficiently leverage small quantities of data, and that these performance increases do not only depend on additional data from augmentation."
  },
  {
    "sample_id": 55,
    "title": "A Human Subject Study of Named Entity Recognition (NER) in Conversational Music Recommendation Queries",
    "abstract": "We conducted a human subject study of named entity recognition on a noisy corpus of conversational music recommendation queries, with many irregular and novel named entities. We evaluated the human NER linguistic behaviour in these challenging conditions and compared it with the most common NER systems nowadays, fine-tuned transformers. Our goal was to learn about the task to guide the design of better evaluation methods and NER algorithms. The results showed that NER in our context was quite hard for both human and algorithms under a strict evaluation schema; humans had higher precision, while the model higher recall because of entity exposure especially during pre-training; and entity types had different error patterns (e.g. frequent typing errors for artists). The released corpus goes beyond predefined frames of interaction and can support future work in conversational music recommendation.",
    "introduction": "Music recommendation systems (RSs), fundamental to streaming services nowadays, learn from user listening history or music content which artists or tracks to suggest next (Schedl et al., 2018). Most of these algorithms provide personalized music content to the users when logging in the streaming apps or websites, or when triggered with pre-defined utterances via voice assistants (Ammari et al., 2019; Bontempelli et al., 2022). More recent conversational RSs aim to help users to express their recommendation needs by supporting interactions via queries in natural language (Jannach et al., 2021). However, despite existing in the scientific literature, such conversational RSs are not widely deployed because of multiple issues, one being NER.\nThe processing of recommendation queries entails the extraction of named entity mentions (Moon et al., 2019; Rongali et al., 2020). This sub-task faces multiple challenges, even when queries are framed as pre-defined utterances. The transcriptions of the voice queries results in lower-case noisy text, often with misspellings (Muralidharan et al., 2021). The lack of capitalisation in entities and misspelled words are often present in text-based queries too (Cheng et al., 2021). Music entities, or those coming from the creative content domains, are highly irregular: they do not follow inherent patterns as it is the case with people’s names, and there is little to no separation between the vocabularies of entity and context words, especially for creative works (Derczynski et al., 2017) (e.g. common words like \"I\" or \"love\" in track titles). Also, new music entities appear all the time. Major music streaming services ingest one new track almost every second (Ingham, 2021).\nPrevious works have already shown that NER systems struggle with the aforementioned challenges (Augenstein et al., 2017; Lin et al., 2020b; Epure and Hennequin, 2022). Thus, multiple approaches have been proposed to address them, either focused 1) on collecting more and relevant data for training / fine-tuning standard NER sequential models (Lison et al., 2020); or 2) on model’s design choices that favour generalisation (Guerini et al., 2018; Lin et al., 2020a). Most solutions focused on the latter objective have been motivated by the human NER linguistic behaviour, e.g. make the model rely more on context cues than on named entity mentions or learn from a few examples only, as humans do. However, apart from some scarce, partially related works (Derczynski et al., 2016; Ding et al., 2021), there is no systematic investigation of how humans actually perform NER on noisy text with many new and irregular named entities. Moreover, in the case of music recommendation, we are not aware of any existing dataset of queries in natural language, annotated with named entities.\nThus, our goal is to investigate the human NER linguistic behavior when confronted with these challenging conditions. For that, we create MusicRecoNER, a new corpus of noisy natural language queries for music recommendation in English that simulates human-music assistant interactions. We then conduct a human subject research study to establish a human baseline and learn from it. Finally, we perform a detailed comparison of humans and the most popular NER systems nowadays, finetuned transformers, that covers multiple evaluation schemes (strict named entity segmentation and typing, exact segmentation only, or partial segmentation with strict named entity typing) and scenarios including entities previously seen or unseen by the model or humans.\nThe results showed that the task was challenging for humans. Given an aggregated metric such as F1 score, human and algorithmic performances were on par. However, the detailed evaluation revealed that humans struggled more with recall while the best model with precision. The high recall obtained by the model was partially a result of entity exposure during pre-training or fine-tuning. Also, music entities had different error patterns and, in some queries, had ambiguous context that made their segmentation and typing quite hard.\nTo sum up, our research contribution1 are:\n1. MusicRecoNER, a corpus of noisy complex natural language queries for music recommendation collected from human-human conversations in English, but which simulates humanmusic assistant interactions, annotated with Artist and WoA (work of art) entities. This dataset is not limited to pre-defined utterances as it would be the case if collected from interactions with conversational or voice assistants. Thus, it contains entities in diverse context, being also a useful resource for future work on conversational music recommendation.\n2. A human subject study design for NER in noisy text with many new and irregular named entities. The proposed method is transferable to other creative content domains that face similar challenges to music such as books, movies, videos, but also to any other domain with scarce data, which wants to learn more about the NER task before building a system.\n3. An extensive music NER benchmark on noisy text which compares the performance of human versus automatic baselines under multiple evaluation schemes, scenarios and by controlling for the novelty of named entities.",
    "conclusion": "In this work, we investigated the human linguistic behavior when performing NER in the music domain. We created MusicRecoNER, a new corpus of complex noisy queries for recommendations annotated with Artist and WoA entities. We then designed and conducted a human subject research study to establish a human baseline and learn from its comparison with the most popular systems nowadays, fine-tuned transformers. We performed a thorough evaluation covering multiple metrics, schemes and scenarios, including a careful analysis of the impact of entity exposure on results.\nThe results obtained by the algorithmic baselines were comparable to the human ones. Yet, the detailed evaluation showed that humans yielded a better precision while the model had a better recall, linked also to entity exposure during pre-training and fine-tuning. Thus, when evaluating fine-tuned pre-trained models, checking their performance on new entities shows their real generalisation ability.\nTable 7: Recall scores under the strict evaluation schema on Seen and Unseen.\nRegarding the NER evaluation protocol, human performances were much better under a more relaxed schema focused on segmentation or typing only. Such a schema could prove a more realistic setup to aim to when training models too. Also, we noticed that the relevant schema depended on the entity type as Artist was better segmented, while WoA better typed.\nContrary to previous claims, we show that, in our domain, NER in challenging conditions such as noisy text, and irregular or novel entities is rather hard for humans even when provided with complex instructions and multiple examples. Thus, although we could learn from the human linguistic behaviour, we should not, by default, assume their results to be a target for any NLP problem. For some tasks, it is common when establishing a human baseline to consider it as an upper bound for the model. This is not necessarily a desirable outcome in our case as it would imply mislabelling 1/3 WoA entities. More generally, as we also showed by studying the impact of entity exposure, algorithms can store a lot more knowledge than humans and one may want to leverage this as much as possible.\nAs for proposing a better system to perform music NER, one next step would be to continue the model’s pre-training on more related data, in our case music, to get even more exposure, or to integrate gazetteers. Still, given the rate of new entities in our domain, forcing the model to rely more on context, when context is not confusing, is another desirable future direction. In case of context ambiguity, asking questions to clarify the request and supporting user interaction in natural language could be ultimately the answer towards a more suitable, but still very challenging solution. We plan to explore these ideas as future work."
  },
  {
    "sample_id": 56,
    "title": "A Japanese Word Segmentation Proposal",
    "abstract": "Current Japanese word segmentation methods, that use a morpheme-based approach, may produce different segmentations for the same strings. This occurs when these strings appear in different sentences. The cause is the influence of different contexts around these strings affecting the probabilistic models used in segmentation algorithms. This paper presents an alternative to the current morpheme-based scheme for Japanese word segmentation. The proposed scheme focuses on segmenting inflections as single words instead of separating the auxiliary verbs and other morphemes from the stems. Some morphological segmentation rules are presented for each type of word and these rules are implemented in a program which is properly described. The program is used to generate a segmentation of a sentence corpus, whose consistency is calculated and compared with the current morpheme-based segmentation of the same corpus. The experiments show that this method produces a much more consistent segmentation than the morpheme-based one.",
    "introduction": "In computational linguistics, the first step in textprocessing tasks is segmenting an input text into words. Most languages make use of white spaces as word boundaries, facilitating this segmentation step. However, Japanese is one of the few languages that does not use a word delimiter. This particular problem has been the focus of many researchers because its solution is key to subsequent processing tasks, such as Part-of-Speech (PoS) tagging, machine translation or file indexing.\nSegmenting a text requires the definition of a segmentation unit (Indurkhya and Damerau, 2010). This unit must be strictly defined to describe all the elements in a language. But languages are not perfect and have changed abruptly throughout the years, making it difficult or nearly impossible to define such a unit. The consensus has been that the unit to be used was the word, because it defines the majority of the elements in a language, elements that have a meaning and can stand by themselves (Katamba, 1994).\nEven though there still are some constructions that do not fit in the word definition (Bauer, 1983), this segmentation unit is useful in languages that use spaces because they separate the majority of words in a text. For Japanese, however, this is not the case. It is a language that does not use spaces in its written form.\n私たちの性格はまったく異なる。\n(Our personalities are completely different.)\nFurthermore, Japanese is an agglutinative language, which means that some constructions (specially inflected words) are formed by consecutively attaching morphemes to a stem (Kamermans, 2010). These long words are very important because they can work as full sentences without the need to add context that was previously stated, as illustrated in the following example:\n待たされていました。 (I have been kept waiting.)\n待つ (wait)\n待たされる (be kept waiting)\n待たされている (being kept waiting)\n待たされています (being kept waiting) (*P)1 待たされていました (been kept waiting) (*P)\nGiven the nature of the language and the lack of a need for native speakers to explicitly separate words, there is no standard on how to segment a written text. Because of this, the segmentation unit and rules for text processing tasks are set by each researcher, although most of them have chosen a morpheme-based approach (Matsumoto et al., 1991; Kudo, 2005; Matsumoto et al., 2007).\nThe downside about this morpheme approach is that, in many cases, there is no consistency when segmenting the same string. The cause seems to be the influence of different contexts on the probabilistic models used in segmentation algorithms. In other words, by producing short morphemes as candidates, there are many segmentation possibilities from which the final one may change due to the context. This inconsistency problem is visible in n-gram data produced by Kudo and Kazawa (2009) and Yata (2010). Within these files, there are various entries of the same string as result of different segmentations, as shown in Table 1. This problem directly affects any later processing task that relies on the resulting segmentation. In machine translation, for example, different segmentations for the same word would produce different incorrect translations.\n4gm-0056 行き ま し た 384\nTable 1: The word 行きました (went) (*P) as found in n-gram data files produced by Yata (2010).\nInflected words follow a limited set of rules. These rules properly define all possible inflections (Kamermans, 2010). As such, they can only lead to one possible correct segmentation. Taking this premise, the Proposed approach aims for a more consistent segmentation by focusing on the treatment of inflected words to limit their segmentation possibilities to a single one in all cases. Thus, reducing word segmentation inconsistency errors.\nThe present work is structured as follows: Section 2 describes the rules that lead the Proposed segmentation method. Section 3 describes the implementation of the algorithm that applies these rules; Section 4 introduces the evaluation parameters and the results obtained with the Proposed method, a comparison of these metrics with a morpheme-based method and discussion of the results; and Section 5 presents the conclusions of the work.",
    "conclusion": "We have demonstrated that by considering inflectional words (with all their auxiliary verbs) as single words, the number of possible segmentations for those words in different contexts gets reduced. Therefore, the resulting segmentation is more consistent and more accurate. Tasks that use word segmentation would also see an improvement, such as language models and machine translation systems.\nThis approach relies on the fact that it is possible to define all the inflectional rules of the Japanese language. The same method could be applied to other words that can be defined by rules, or to other unsegmented languages whose rules can be defined the same way."
  },
  {
    "sample_id": 57,
    "title": "A Joint Model for Dropped Pronoun Recovery and Conversational Discourse Parsing in Chinese Conversational Speech",
    "abstract": "In this paper, we present a neural model for joint dropped pronoun recovery (DPR) and conversational discourse parsing (CDP) in Chinese conversational speech. We show that DPR and CDP are closely related, and a joint model benefits both tasks. We refer to our model as DiscProReco, and it first encodes the tokens in each utterance in a conversation with a directed Graph Convolutional Network (GCN). The token states for an utterance are then aggregated to produce a single state for each utterance. The utterance states are then fed into a biaffine classifier to construct a conversational discourse graph. A second (multi-relational) GCN is then applied to the utterance states to produce a discourse relation-augmented representation for the utterances, which are then fused together with token states in each utterance as input to a dropped pronoun recovery layer. The joint model is trained and evaluated on a new Structure Parsing-enhanced Dropped Pronoun Recovery (SPDPR) dataset that we annotated with both two types of information. Experimental results on the SPDPR dataset and other benchmarks show that DiscProReco significantly outperforms the state-of-the-art baselines of both tasks.",
    "introduction": "Pronouns are often dropped in Chinese conversations as the identity of the pronoun can be inferred from the context (Kim, 2000; Yang et al., 2015) without causing the sentence to be incomprehensible. The task of dropped pronoun recovery (DPR) aims to locate the position of the dropped pronoun and identify its type. Conversational discourse parsing (CDP) is another important task that aims to analyze the discourse relations among utterances\nquestion\nexpansion\nFigure 1: Top: A conversation snippet in which the dropped pronoun is shown in bracket. Bottom: Pronoun recovery results by two baselines and the proposed DiscProReco. Baselines which ignore the relation “(B3 expands B2) replies A2” mistakenly recover the dropped pronoun 你(you) as 我(I) since the utterance B 3 is considered semantically similar to A 2.\nin a conversation, and plays a vital role in understanding multi-turn conversations.\nExisting work regards DPR and CDP as two independent tasks and tackles them separately. As an early attempt of DPR, Yang et al. (2015) employ a Maximum Entropy classifier to predict the position and type of dropped pronouns. Zhang et al. (2019) and Yang et al. (2019) attempt to recover the dropped pronouns by modeling the referents with deep neural networks. More recently, Yang et al. (2020) attempt to jointly predict all dropped pronouns in a conversation snippet by modeling dependencies between pronouns with general conditional random fields. A major shortcoming of these DPR methods is that they overlook the discourse relation (e.g., reply, question) between conversational utterances when exploiting the context of the dropped pronoun. At the same time, previous CDP methods (Li et al., 2014; Afantenos et al., 2015; Shi and Huang, 2019) first predict the relation for each utterance pair and then construct the discourse structure for the conversation with a decoding algorithm. The effectiveness of these methods are compromised since the utterances might be incomplete when they have dropped pronouns.\nTo overcome these shortcomings, we propose a novel neural model called DiscProReco to perform DPR and CDP jointly. Figure 1 is a Chinese conversation snippet between two speakers A and B that illustrates the advantages of such a joint approach. In this example, a pronoun “你 (you)” is dropped in utterance B 3. It is critical for the DPR model to know that both utterances B 2 and B 3 are in reply to the utterance A 2, when recovering this dropped pronoun. Methods which ignore the structure (“(B3 expands B2) replies A2”) will more likely consider the utterance B 3 to be semantically similar to A 2, and wrongly recover the pronoun as “我 (I)”.\nGiven a pro-drop utterance and its context, DiscProReco parses the discourse structure of the conversation and recovers the dropped pronouns in the utterance in four steps: (i) Each utterance is parsed into its dependency structure and fed into a directed GCN to output the syntactic token states. The utterance state is then obtained by aggregating the token states in the utterance. (ii) The utterance states of a conversation are fed into a biaffine classifier to predict the discourse relation between each utterance pair and the discourse structure of the conversation is constructed. (iii) Taking the discourse structure as input, another (multirelational) GCN updates the utterance states and fuses them into the token states for each utterance to produce discourse-aware token representations. (iv) Based on the discourse structure-aware context representation, a pronoun recovery module is designed to recover the dropped pronouns in the utterances. When training this model, all components are jointly optimized by parameter sharing so that CDP and DPR can benefit each other. As there is no public dataset annotated with both dropped pronouns and conversational discourse structures, we also construct Structure Parsing-enhanced Dropped Pronoun Recovery (SPDPR) corpus, which is the first corpus annotated with both types of information. Experimental results show that DiscProReco outperforms all baselines of CDP and DPR.\nContributions: This work makes the following contributions: (i) We propose a unified framework DiscProReco to jointly perform CDP and DPR, and show that these two tasks can benefit each other. (ii) We construct a new large-scale dataset SPDPR (Section 4) which supports fair comparison across different methods and facilitates future research on both DPR and CDP. (iii) We present experimental results which show that DiscProReco with its joint learning mechanism realizes knowledge sharing between its CDP and DPR components and results in improvements for both tasks (Section 5). The code and SPDPR dataset is available at https://\ngithub.com/ningningyang/DiscProReco.",
    "conclusion": "This paper presents that dropped pronoun recovery and conversational discourse parsing are two strongly related tasks. To make them benefit from each other, we devise a novel framework called DiscProReco to tackle these two tasks simultaneously. The framework is trained in a joint learning paradigm, and the parameters for the two tasks are jointly optimized. To facilitate the study of the problem, we created a large-scale dataset called SPDPR which contains the annotations of both dropped pronouns and discourse relations. Experimental results demonstrated that DiscProReco outperformed all baselines on both tasks."
  },
  {
    "sample_id": 58,
    "title": "A Little Linguistics Goes a Long Way: Unsupervised Segmentation with Limited Language Specific Guidance",
    "abstract": "We present de-lexical segmentation, a linguistically motivated alternative to greedy or other unsupervised methods, requiring language specific knowledge, but no direct supervision. Our technique involves creating a small grammar of closed-class affixes which can be written in a few hours. The grammar over generates analyses for word forms attested in a raw corpus which are disambiguated based on features of the linguistic base proposed for each form. Extending the grammar to cover orthographic, morphosyntactic or lexical variation is simple, making it an ideal solution for challenging corpora with noisy, dialect-inconsistent, or otherwise non-standard content. We demonstrate the utility of de-lexical segmentation on several dialects of Arabic. We consistently outperform competitive unsupervised baselines and approach the performance of state-of-the-art supervised models trained on large amounts of data, providing evidence for the value of linguistic input during preprocessing.",
    "introduction": "Non-standard domains, dialectal variation, and unstandardized spelling make segmentation challenging, though morphologically rich languages require good segmentation to enable downstream applications from syntactic parsing to machine translation (MT). For domains lacking sufficient annotated data to train segmenters, one must resort to language specific greedy techniques or language agnostic unsupervised techniques. Greedy techniques use maximum matching to identify base words, leveraging large dictionaries (Guo, 1997). Yet such dictionaries are often unavailable or too expensive for low resource languages. Language agnostic unsupervised options like MORFESSOR (Creutz and Lagus, 2005) and byte pair encoding (BPE) (Sennrich et al., 2016) assume no resources beyond raw text but can yield lower performance on downstream tasks (Vania and Lopez, 2017; Kann et al., 2018). They also suffer from typological biases and favor intended applications at the expense of others.\nTo this end, we present De-lexical Segmentation (DESEG), a slightly more expensive but powerful alternative to language agnostic morphological segmentation, realizing most of the benefits of supervised segmentation at far less a cost. DESEG requires language specific input in the form of a small grammar describing the combinatorics of closed-class affixes. We demonstrate that such a grammar can be constructed easily and rapidly for a new language or dialect. Hence, DESEG addresses the scenario in which there is no supervised segmenter available for a given language or dialect (or no segmenter trained on a domain with sufficient lexical overlap with the target domain in its training data), but the user does have linguistic knowledge of the target language/dialect.\nThe user-provided grammar is employed in conjunction with a large, raw corpus. The grammar over generates analyses for all words therein, allowing for maximal recall not only of the possible affix combinations, but also variant spellings and dialectal idiosyncrasies. The preferred analysis is disambiguated based on the fertility with which its proposed base attaches to different affixes in analyses of other words throughout the corpus. This follows from the logic that valid bases are more likely to productively combine with more exponents1 (Bertram et al., 2000). By leveraging language specific resources but learning to disambiguate empirically without supervision, we mitigate much of the sparsity inherent in processing non-standard domains.\nUsing a corpus of several Arabic dialects exhibiting rich and complex morphology, unstandardized spelling, and variation bordering on mutual unintelligibility, we evaluate DESEG intrinsically on language modeling (LM) and extrinsically on MT. DESEG consistently outperforms MORFESSOR and BPE while only costing a few hours of grammar-building labor; and in some environments it outperforms state-of-the-art supervised Arabic tokenizers MADAMIRA (Pasha et al., 2014) and FARASA (Abdelali et al., 2016). The success of such a simple model is strong evidence for the value of linguistic input during preprocessing. DESEG is publicly available at github. com/CAMeL-Lab/deSeg.",
    "conclusion": "We present an effective unsupervised means of introducing linguistic information for segmentation that greatly improves performance over other unsupervised systems as evaluated both intrinsically and extrinsically. We target robust handling of rich morphological phenomena and noisy corpora, achieving performance on a multi-dialect Arabic corpus comparable to state-of-the-art supervised systems. The success of our simple system is strong evidence for the value of linguistic input during preprocessing.\nIn the future, we plan to evaluate our models on natural (uncommissioned) dialectal corpora. We also plan to enhance our delexicalize models with non-concatenative components. And we also in tend to develop models that consider context."
  },
  {
    "sample_id": 59,
    "title": "A Model-Agnostic Data Manipulation Method for Persona-based Dialogue Generation",
    "abstract": "Towards building intelligent dialogue agents, there has been a growing interest in introducing explicit personas in generation models. However, with limited persona-based dialogue data at hand, it may be difficult to train a dialogue generation model well. We point out that the data challenges of this generation task lie in two aspects: first, it is expensive to scale up current persona-based dialogue datasets; second, each data sample in this task is more complex to learn with than conventional dialogue data. To alleviate the above data issues, we propose a data manipulation method, which is model-agnostic to be packed with any personabased dialogue generation model to improve its performance. The original training samples will first be distilled and thus expected to be fitted more easily. Next, we show various effective ways that can diversify such easier distilled data. A given base model will then be trained via the constructed data curricula, i.e. first on augmented distilled samples and then on original ones. Experiments illustrate the superiority of our method with two strong base dialogue models (Transformer encoderdecoder and GPT2).",
    "introduction": "The ability to generate responses with consistent personas is important towards building intelligent dialogue agents. In past years, there has been a growing interest in introducing explicit personas in dialogue generation models (Song et al., 2019; Wolf et al., 2019). A piece of persona text generally consists of profiles and background personal facts. A clipped persona-based dialogue from the PersonaChat (Zhang et al., 2018a) dataset is shown in Figure 1, which covers rich persona features. For\nPersonas of great . i was just reading a book 1.i work as a veterinarian .\n2.i am married and have work long hours as five children .\n3.i am a vegetarian .\n4.my favorite music is hip hop.\nhave have\n: persona consistency : dialogue coherence\nFigure 1: Each response in a persona-based dialogue is mostly related to one persona sentence and its latest dialogue history utterance. Persona sentences in grey are redundant for all responses.\na persona-based dialogue generation model, generated responses need to be relevant to the dialogue context as well as consistent with personas.\nMost existing generation models for this task rely heavily on training with sufficient personabased dialogues. However, available data are limited due to their expensive collection costs. Take the PersonaChat as an example, two crowd-sourced annotators are hired to play the part of a provided persona and converse naturally with each other. In total, about 162 thousand dialogue utterances are collected with less than 5 thousand unique persona profiles. Compared with conventional dialogue datasets such as OpenSubtitles (Lison and Tiedemann, 2016) and Weibo (Shang et al., 2015) with millions of utterances, persona-based dialogue datasets are relatively small.\nBesides the limited data scale, another data issue we want to point out is that a persona-based dialogue is more complex to learn with, in comparison with conventional dialogues. Recall that a persona-based dialogue involves not only multiple dialogue utterances, but also auxiliary persona sentences. Welleck et al. (2019) showed that not all responses in the PersonaChat dataset are consistent with the provided personas. This makes it difficult for a model to capture a reliable mapping from training data. Supposing we apply a similar dialogue model as in conventional dialogue generation tasks with a comparable parameter size, we should expect more data would be necessary to train a robust model on the more difficult data setting. Moreover, it may be difficult to use existing data augmentation methods (Li et al., 2019; Niu and Bansal, 2019) to automatically construct such complex persona-based dialogue data. For example, if we apply back translation (Sennrich et al., 2016) to every sentence in persona-based samples, the augmented ones may not maintain the coherence between the dialogue history and the response as well as the consistency between the persona and the response simultaneously.\nA few studies have been conducted to alleviate the above data issues by finetuning existing pretrained models such as GPT (Wolf et al., 2019; Golovanov et al.) or BERT (?Song et al., 2021). They often stick to a certain pretrained model. Sophisticated finetuning strategies, including proper network modifications and loss functions, are required to get satisfactory performance, making them not useful across different pretrained models. Moreover, they do not address the data difficulty issue explicitly. Most of them simply concatenate all persona and dialogue history sentences into a single input sequence for finetuning, and rely on the ability of the pretrained model to fast adapt to the target data domain. Hence, we want to design a model-agnostic method to address both the data scale and data difficulty issue, which can be packed with any base model, either trained from scratch or finetuned from a pretrained model.\nIn this work, we propose a data manipulation method for persona-based dialogue data, which is model-agnostic to be packed with any base model to improve their robustness and consistency. Our method includes three operations on data, namely D3, in sequence: (i) Data distillation: original training samples are simplified into contain only useful and less redundant persona sentences and dialogue utterances, which are expected to be fitted more easily; (ii) Data diversification: with the easier distilled samples, we can also perform data augmentation more reliably. We design various methods to edit new personas, and then align them with new and consistent responses to improve data diversity; (iii) Data curriculum: with both augmented distilled and original data at hand, we arrange them into a data curriculum for model learning (Bengio et al., 2009), where the base model is trained on the easier augmented distilled data and then the harder original data. To validate the effectiveness of our method, we perform experiments on two strong base dialogue models, Transformer-based encoder-decoder and GPT2.",
    "conclusion": "Our work targets the challenging personal-based dialogue generation task. Unlike previous work that designs a new dialogue model to improve the generation performance, we analyze the data issues affecting current models. On one hand, the data scale and diversity are expensive to increase by data collection. On the other hand, current data are difficult to learn with. Based on such an understanding, we propose a model-agnostic data manipulation method for this task. It first distills the original data and then augments both the amount and diversity of the distilled data. A curriculum training is then applied to utilize both augmented and original data. Experimental results showed that our method effectively improves the performance of two strong dialogue models, i.e. Transformer encoder-decoder and GPT2."
  },
  {
    "sample_id": 60,
    "title": "A New Surprise Measure for Extracting Interesting Relationships between Persons",
    "abstract": "One way to enhance user engagement in search engines is to suggest interesting facts to the user. Although relationships between persons are important as a target for text mining, there are few effective approaches for extracting the interesting relationships between persons. We therefore propose a method for extracting interesting relationships between persons from natural language texts by focusing on their surprisingness. Our method first extracts all personal relationships from dependency trees for the texts and then calculates surprise scores for distributed representations of the extracted relationships in an unsupervised manner. The unique point of our method is that it does not require any labeled dataset with annotation for the surprising personal relationships. The results of the human evaluation show that the proposed method could extract more interesting relationships between persons from Japanese Wikipedia articles than a popularity-based baseline method. We demonstrate our proposed method as a chrome plugin on google search.",
    "introduction": "Interesting facts are useful information for a variety of important tasks. For example, in data mining, the interesting facts can enhance user engagement in search engines (Fatma et al., 2017). In natural language processing, the interesting facts can improve user experience with automatic conversation systems (Niina and Shimada, 2018). However, if we rely on experts to gather the interesting facts, the cost becomes quite high.\nAs a solution, several approaches have been developed to extract interesting facts automatically. Lin and Chalupsky (2003) proposed a set of unsupervised link discovery methods that can compute interestingness on graph data represented as a set of entities connected by a set of binary relations.\nin Pepper Land, Ringo Starr, along with Beatles companions John Lennon, George Harrison, and Paul McCartney, went to\nrelationships between persons.\nPrakash et al. (2015) extracted interesting sentences about movie entities from Wikipedia articles and ordered them based on their interestingness by utilizing Rank-SVM, trained in a supervised manner. Tsurel et al. (2017) proposed an algorithm that automatically mines trivia facts from Wikipedia by utilizing its category structure. Their approach can rank categories for an entity based on their trivia quality induced from the categories. Fatma et al. (2017) proposed a method for automatically mining trivia facts for an entity of a given domain in knowledge graphs by utilizing deep convolutional neural networks, trained in a supervised manner. Korn et al. (2019) mined trivia facts from superlative tables in Wikipedia articles. Kwon et al. (2020) proposed a method to obtain sentences including trivia facts with utilizing paragraph structures in Wikipedia articles.\nHowever, some of these approaches work only on structured datasets such as knowledge graphs or Wikipedia categories. In addition, while supervised approaches can work on unstructured natural language texts, the applicable domain is restricted due to the lack of annotated datasets. Hence, the current approaches for extracting interesting facts\nthe Japanese search query Hayao Miyazaki, top five interesting relationships are presented at the top of the search results. The red texts are translations of them.\nare considered limited. In particular, although relationships between persons are important as a target for text mining, there are few effective approaches for extracting interesting relationships between persons.\nFigure 1 shows examples of interesting relationships between persons.1 The first example is a famous film director who initially had a fairly low regard for an actor who is now extremely famous and successful. The second example is about a famous baseball player who asked another famous baseball player for an autograph. The third example relates to famous musicians engaged in something completely unrelated to music. These examples illustrate that surprisingness is an important factor in interesting personal relationships.\nIn this paper, to extract such interesting relationships, we focus on surprising relationships between persons. We propose a method that extracts relationships between persons from natural language texts and then scores their surprise scores based on the Mahalanobis distance (De Maesschalck et al., 2000), which has been used in the outlier detection task. Our proposed method first extracts all personal relationships from dependency trees for each sentence and then calculates the surprise scores of the extracted relationships on a continuous vector space in an unsupervised manner. As such, our method does not require any labeled dataset for extracting the surprising personal relationships.\nThe results of our human evaluation show that the proposed method could extract more interesting relationships between persons from Japanese Wikipedia articles than a popularity-based baseline method. Furthermore, as shown in Figure 2, we incorporated our method into a google chrome plugin. You can watch our demo video for this plugin at a shared directory in our google drive.",
    "conclusion": "In this paper, we proposed a method for extracting interesting relationships between persons from natural language texts in an unsupervised manner.\nHuman evaluation of the personal relationships extracted from Japanese Wikipedia articles showed that the proposed method improved the interestingness compared to a popularity-based baseline. Through the result, we can conclude that considering the surprisingness of relationships between persons is effective in improving the interestingness of the extracted results.\nFurthermore, to demonstrate our proposed method, we incorporated the method into a google chrome plugin, which can work on google search.\nAs future work, we will investigate ways to extract personal relationships based on more detailed information about a dependency tree."
  },
  {
    "sample_id": 61,
    "title": "A Novel Table-to-Graph Generation Approach for Document-Level Joint Entity and Relation Extraction",
    "abstract": "Document-level relation extraction (DocRE) aims to extract relations among entities within a document, which is crucial for applications like knowledge graph construction. Existing methods usually assume that entities and their mentions are identified beforehand, which falls short of real-world applications. To overcome this limitation, we propose TAG, a novel tableto-graph generation model for joint extraction of entities and relations at document-level. To enhance the learning of task dependencies, TAG induces a latent graph among mentions, with different types of edges indicating different task information, which is further broadcast with a relational graph convolutional network. To alleviate the error propagation problem, we adapt the hierarchical agglomerative clustering algorithm to back-propagate task information at decoding stage. Experiments on the benchmark dataset, DocRED, demonstrate that TAG surpasses previous methods by a large margin and achieves state-of-the-art results1.",
    "introduction": "Relation extraction (RE) is the task to extract relational facts from natural language text, which plays a crucial role in various downstream tasks, e.g. knowledge graph construction and question answering (Yih et al., 2015; Trisedya et al., 2019; Li and Zou, 2022). Early studies mostly focus on sentence-level RE, i.e. predicting relations among entities in one single sentence. However, in realworld scenarios such as Wikipedia articles or scientific papers, large amounts of relational facts are expressed across multiple sentences, which necessitate inter-sentence reasoning skills. Hence, recent efforts have been moving towards the more realistic document-level RE (DocRE) (Yao et al., 2019; Nan et al., 2020; Zhou et al., 2021).\nSubject: Balboa Boneke Object: Equatorial Guinean\nRelation: country of citizenship\nSubject: Balboa Boneke Object: Valencia\nRelation: place of death\nSubject: Valencia Object: Spain\nRelation: country\nJuan Balboa Boneke (9 June 1938 – 10 March 2014) was\nan Equatorial Guinean politician and writer. … After his\nexile, he settled down in Valencia with his second wife\nand her family. Balboa Boneke died from renal problems,\ncoupled with a three-year depression caused by the\ndeath of his wife, on 10 March 2014 in Valencia , Spain .\nFigure 1: An example adapted from the DocRED dataset. Mentions refer to the same entity are in same color. We omit some relations and denote some entities with underline for clarity.\nDespite the rapid progress, most previous DocRE methods solely focus on the task of relation extraction, which assumes that entities and their corresponding mentions are given beforehand. As shown by Figure 1, to extract both of entities and relations at document-level, a natural idea is to use a pipeline approach. Traditionally, it first divide the whole task into subtasks of mention extraction (ME), coreference resolution (COREF) and relation extraction (RE), then use separate models to conduct each task step by step (Zaporojets et al., 2021). However, the pipeline framework ignores the underlying dependencies among subtasks, which may lead to suboptimal performance. Some progress on jointly considering the subtasks has been made (Eberts and Ulges, 2021; Xu and Choi, 2022), yet, previous attempts still model the tasks of COREF and RE separately, inducing possible bias at both encoding and decoding stages. On the one hand, these methods still suffer from the problem with lack of information sharing. They either completely rely on the shared language model (e.g. BERT) at representation level (Eberts and Ulges, 2021) , or only consider one-way information flow from RE to COREF and neglect other cross-task dependencies (Xu and Choi, 2022). On the other hand, prior approaches mostly employ the pipelinestyle decoding, which first recognize mention spans and form entity clusters, then perform relation classification for each entity pair. Such routine is not only time consuming, but faces with the error propagation problem (Li and Ji, 2014). The results of entity extraction may affect the performance of relation extraction and lead to cascading errors. Xu and Choi (2022) attempt to use a regularization term in COREF scorer to mitigate this issue, but the problem is still not fully resolved.\nIn this work, we propose TAG, a novel tableto-graph generation model, to address these aforementioned challenges. We first unify both tasks of COREF and RE with the classic table filling framework (Miwa and Sasaki, 2014; Gupta et al., 2016). We then devise a following table filler to encode original texts and make predictions for both tasks at a coarse level. Regarding mentions as nodes, we dynamically build two corresponding coreference and relation graphs, where the edges are weighted by the confidence scores of table filler. Besides, to alleviate the long-term dependency problem as well as explicitly model the syntactic information, we construct a syntactic graph over mentions. Given these three subgraphs, TAG regards them as three different types of edges and uses a relational graph convolutional network (RGCN, Schlichtkrull et al., 2018) to model implicit task dependencies at a fine level. Unlike previous multi-task systems that solely share span representations directly from the language model, our coarse-to-fine framework leverages rich node representations by propagating information through semantic and syntactic links.\nIntuitively, mentions within the same entity cluster should establish similar relation links with other entities (Xu and Choi, 2022). To avoid the error propagation problem, we exploit this postulation and adapt the hierarchical agglomerative clustering (HAC) algorithm to cluster mentions. The core of HAC is the computation of coreference distance between each cluster pair. To back-propagate relational information, we compute the relation vectors of nodes and use the average Hamming distances among different clusters as additional penalty.\nWe evaluate TAG on DocRED (Yao et al., 2019), a widely-adopted DocRE benchmark. Experiments show that: (1) The coarse-grained table filler baseline establishes competitive results, as compared with previous methods. (2) The finegrained information propagation module and enhanced HAC decoding algorithm can effectively promote cross-task interactions and better alleviate the error propagation problem. (3) Our proposed TAG achieves new state-of-the-art and outperforms prior approaches by a large margin. We also report the first result of joint entity and relation extraction on Re-DocRED (Tan et al., 2022), a revised version of DocRED, for future research.\nOur contributions can be summarized as follow:\n• We unify the tasks of COREF and RE in document-level joint entity and relation extraction with a table filling framework, and propose a novel table-to-graph generation method TAG to facilitate information sharing. During the decoding stage, we adapt the HAC algorithm to enhance COREF with RE predictions, thereby mitigating the issue of error propagation.\n• We demonstrate that TAG surpasses previous methods and achieves new state-of-the-art results on the standard DocRE benchmark.",
    "conclusion": "In this paper, we propose TAG, a novel table-tograph generation model, to jointly extract entities and relations within a document. Different from prior approaches, we unify the tasks of coreference resolution and relation extraction with a table filling framework, and leverage a coarse-to-fine strategy to facilitate information sharing among these subtasks. To avoid the error propagation problem, we adapt the HAC algorithm to enhance COREF with RE predictions at decoding stage. Experimental results on the widely-adopted benchmark, DocRED, demonstrate that TAG significantly outperforms previous methods. Further analysis also confirms the effectiveness of the modules in our model."
  },
  {
    "sample_id": 62,
    "title": "A Personalized Sentiment Model with Textual and Contextual Information",
    "abstract": "In this paper, we look beyond the traditional population-level sentiment modeling and consider the individuality in a person’s expressions by discovering both textual and contextual information. In particular, we construct a hierarchical neural network that leverages valuable information from a person’s past expressions, and offer a better understanding of the sentiment from the expresser’s perspective. Additionally, we investigate how a person’s sentiment changes over time so that recent incidents or opinions may have more effect on the person’s current sentiment than the old ones. Psychological studies have also shown that individual variation exists in how easily people change their sentiments. In order to model such traits, we develop a modified attention mechanism with Hawkes process applied on top of a recurrent network for a userspecific design. Implemented with automatically labeled Twitter data, the proposed model has shown positive results employing different input formulations for representing the concerned information.",
    "introduction": "Sentiment is one of the key factors affecting human behavior. Studying the way in which sentiment is perceived, evolved and expressed is an essential part in artificial intelligence. To analyze sentiment in text, researchers have made different assumptions on linguistic behaviors that are leveraged with approaches developed based on the nature of the text, the representation of the related information and the objectives. However, majority of the studies are conducted at the population-level which assumes that people follow a common understanding with regard to the use of language. Such approaches can be inaccurate in the cases where people use the same lexical choices to convey different messages or vice versa.\nHarris (2006) stated that ‘no two alike’ showing the inherent difference in human that motivates the research of personalized sentiment analysis. Grounded in the psychological works, we argue that it is significant to study the effect of individuality in the expressions and to investigate the possibility of providing a deeper understanding of the expressions from the writers’ own perspectives. In this work, we concern the use of preferred lexical choices when expressing sentiment (Reiter and Sripada, 2002) and the level of consistency in retaining a sentiment (Janis and Field, 1956).\nBesides the targeted text message itself, we exploit two types of contextual information for the purpose of realizing the psychological aspects: a person’s expressions in the past and the time when the expressions were made. With the goal of discovering the effect of the contextual information, distinct formulation methods are proposed to integrate the information in the personalized sentiment model. The backboned model is a hierarchical neural network which follows a conventional embedding – recurrent – attention structure with each part rectified for the task. The embedding block is used to generate representations for the used information; the recurrent network fulfills the task of relating to the information from the past; the attention model is shaped with Hawkes process (Laub et al., 2015) in order to model the information decay for each expresser. Generally, recurrent networks consider the order of the elements in a sequence but omit the different gaps between them. Hawkes process is utilized to compensate this issue. Furthermore, a novel approach with a user – factor transformation is employed to merge the Hawkes process within the attention model and to construct user-specific processes. For evaluation, we take the data from a number of frequent users on social platforms where Twitter is used as an example. The data is domain-independent, and it is possible to obtain self-labeled texts that aligns with our goal of understanding the expressers’ perspectives. Significant improvements are seen with certain input formulations, and different Hawkes processes applied for the users are visualized. In the end, we conclude that it is effective to introduce the contextual information to the model.",
    "conclusion": "This paper presents a personalized sentiment model that captures the individualities in expressing sentiment and analyzes the evolvement of sentiment over time. Particularly, we categorize the information used for the modeling into textual and contextual information, and evaluate the effectiveness of using the contextual information to boost the performance of the model. A novel attention mechanism with user-specific Hawkes process is employed for this purpose. Technically, it also provides an alternative for studying various time gaps in temporal sequences with neural networks. Different input formulations are applied in which the combined granular representation performs the best. Based on our findings, we can conclude that the individual variation indeed affects the analysis, and the contextual information, as an essential part in human interactions, positively contributes to the performance.\nBecause the informal text we have used deviates from the language standard, the representation of input text plays a significant role in improving the performance. In the future work, we will exploit phonetic representation which can provide another source of information for such text. The posts can be transcribed into phonetic sequences, for instance, by using the International Phonetic Alphabet, in order to handle certain misspellings and to study the trend of using letters with similar pronunciations as substitutions. Moreover, other types of contextual information should be explored as well to enhance the understanding of individual behaviors on social platforms. As an example, social relations can be used to identify abnormalities in the change of sentiment, especially in the case that a user is exceptionally stimulated by other users or special events which causes untypical behaviors. The personalized model can also be helpful in other scenarios, such as to offer deep understanding for user-tailored conversations or companionship."
  },
  {
    "sample_id": 63,
    "title": "A Pre-training Strategy for Zero-Resource Response Selection in Knowledge-Grounded Conversations",
    "abstract": "Recently, many studies are emerging towards building a retrieval-based dialogue system that is able to effectively leverage background knowledge (e.g., documents) when conversing with humans. However, it is non-trivial to collect large-scale dialogues that are naturally grounded on the background documents, which hinders the effective and adequate training of knowledge selection and response matching. To overcome the challenge, we consider decomposing the training of the knowledge-grounded response selection into three tasks including: 1) query-passage matching task; 2) query-dialogue history matching task; 3) multi-turn response matching task, and joint learning all these tasks in a unified pre-trained language model. The former two tasks could help the model in knowledge selection and comprehension, while the last task is designed for matching the proper response with the given query and background knowledge (dialogue history). By this means, the model can be learned to select relevant knowledge and distinguish proper response, with the help of ad-hoc retrieval corpora and a large number of ungrounded multi-turn dialogues. Experimental results on two benchmarks of knowledge-grounded response selection indicate that our model can achieve comparable performance with several existing methods that rely on crowd-sourced data for training.",
    "introduction": "Along with the very recent prosperity of artificial intelligence empowered conversation systems in the spotlight, many studies have been focused on building human-computer dialogue systems (Wen et al., 2017; Zhang et al., 2020) with either retrievalbased methods (Wang et al., 2013; Wu et al., 2017; Whang et al., 2020) or generation-based methods (Li et al., 2016; Serban et al., 2016; Zhang et al., 2020), which both predict the response with only the given context. In fact, unlike a person who may associate the conversation with the background knowledge in his or her mind, the machine can only capture limited information from the query message itself. As a result, it is difficult for a machine to properly comprehend the query, and to predict a proper response to make it more engaging. To bridge the gap of the knowledge between the human and the machine, researchers have begun to simulating this motivation by grounding dialogue agents with background knowledge (Zhang et al., 2018; Dinan et al., 2019; Li et al., 2020), and lots of impressive results have been obtained.\nIn this paper, we consider the response selection problem in knowledge-grounded conversion and specify the background knowledge as unstructured documents that are common sources in practice. The task is that given a conversation context and a set of knowledge entries, one is required 1): to select proper knowledge and grasp a good comprehension of the selected document materials (knowledge selection); 2): to distinguish the true response from a candidate pool that is relevant and consistent with both the conversation context and the background documents (knowledge matching).\nWhile there exists a number of knowledge documents on the Web, it is non-trivial to collect large-scale dialogues that are naturally grounded on the documents for training a neural response selection model, which hinders the effective and adequate training of knowledge selection and response matching. Although some benchmarks built upon crowd-sourcing have been released by recent works (Zhang et al., 2018; Dinan et al., 2019), the relatively small training size makes it hard for the dialogue models to generalize on other domains or topics (Zhao et al., 2020). Thus, in this work, we focus on a more challenging and practical scenario, learning a knowledge-grounded conversation agent without any knowledge-grounded dialogue data, which is known as zero-resource settings.\nSince knowledge-grounded dialogues are unavailable in training, it raises greater challenges for learning the grounded response selection model. Fortunately, there exists a large number of unstructured knowledge (e.g., web pages or wiki articles), passage search datasets (e.g., query-passage pairs coming from ad-hoc retrieval tasks) (Khattab and Zaharia, 2020) and multi-turn dialogues (e.g., context-response pairs collected from Reddit) (Henderson et al., 2019), which might be beneficial to the learning of knowledge comprehension, knowledge selection and response prediction respectively. Besides, in multi-turn dialogues, the background knowledge and conversation history (excluding the latest query) are symmetric in terms of the information they convey, and we assume that the dialogue history can be regarded as another format of background knowledge for response prediction.\nBased on the above intuition, in this paper, we consider decomposing the training of the grounded response selection task into several sub-tasks, and joint learning all those tasks in a unified model. To take advantage of the recent breakthrough on pretraining for natural language tasks, we build the grounded response matching models on the basis of a pre-trained language model (PLMs) (Devlin et al., 2019; Yang et al., 2019), which are trained with large-scale unstructured documents from the web. On this basis, we further train the PLMs with query-passage matching task, query-dialogue history matching task, and multi-turn response matching task jointly. The former two tasks could help the model not only in knowledge selection but also in knowledge (and dialogue history) comprehension, while the last task is designed for matching the proper response with the given query and background knowledge (dialogue history). By this means, the model can be learned to select relevant knowledge and distinguish proper responses, with the help of a large number of ungrounded dialogues and ad-hoc retrieval corpora. During the testing stage, we first utilize the trained model to select proper knowledge, and then feed the query, dialogue history, selected knowledge, and the response candidate into our model to calculate the final matching degree. Particularly, we design two strategies to compute the final matching score.\nIn the first strategy, we directly concatenate the selected knowledge and dialogue history as a long sequence of background knowledge and feed into the model. In the second strategy, we first compute the matching degree between each queryknowledge and the response candidates, and then integrate all matching scores.\nWe conduct experiments with benchmarks of knowledge-grounded dialogue that are constructed by crowd-sourcing, such as the Wizard-ofWikipedia Corpus (Dinan et al., 2019) and the CMU DoG Corpus (Zhou et al., 2018a). Evaluation results indicate that our model achieves comparable performance on knowledge selection and response selection with several existing models trained on crowd-sourced benchmarks.\nOur contributions are summarized as follows:\n• To the best of our knowledge, this is the first exploration of knowledge-grounded response selection under the zero-resource setting.\n• We propose decomposing the training of the grounded response selection models into several sub-tasks, so as to empower the model through these tasks in knowledge selection and response matching.\n• We achieve a comparable performance of response selection with several existing models learned from crowd-sourced training sets.",
    "conclusion": "In this paper, we study response matching in knowledge-grounded conversations under a zeroresource setting. In particular, we propose decomposing the training of the knowledge-grounded response selection into three tasks and joint train all tasks in a unified pre-trained language model. Our model can be learned to select relevant knowledge and distinguish proper response, with the help of ad-hoc retrieval corpora and amount of multiturn dialogues. Experimental results on two benchmarks indicate that our model achieves a comparable performance with several existing methods trained on crowd-sourced data. In the future, we would like to explore the ability of our proposed method in retrieval-augmented dialogues."
  },
  {
    "sample_id": 64,
    "title": "A Relational Memory-based Embedding Model for Triple Classification and Search Personalization",
    "abstract": "Knowledge graph embedding methods often suffer from a limitation of memorizing valid triples to predict new ones for triple classification and search personalization problems. To this end, we introduce a novel embedding model, named R-MeN, that explores a relational memory network to encode potential dependencies in relationship triples. RMeN considers each triple as a sequence of 3 input vectors that recurrently interact with a memory using a transformer self-attention mechanism. Thus R-MeN encodes new information from interactions between the memory and each input vector to return a corresponding vector. Consequently, R-MeN feeds these 3 returned vectors to a convolutional neural network-based decoder to produce a scalar score for the triple. Experimental results show that our proposed R-MeN obtains state-of-theart results on SEARCH17 for the search personalization task, and on WN11 and FB13 for the triple classification task.",
    "introduction": "Knowledge graphs (KGs) – representing the genuine relationships among entities in the form of triples (subject, relation, object) denoted as (s, r, o) – are often insufficient for knowledge presentation due to the lack of many valid triples (West et al., 2014). Therefore, research work has been focusing on inferring whether a new triple missed in KGs is likely valid or not (Bordes et al., 2011, 2013; Socher et al., 2013). As summarized in (Nickel et al., 2016; Nguyen, 2017), KG embedding models aim to compute a score for each triple, such that valid triples have higher scores than invalid ones.\nEarly embedding models such as TransE (Bordes et al., 2013), TransH (Wang et al., 2014), TransR (Lin et al., 2015), TransD (Ji et al., 2015), DISTMULT (Yang et al., 2015) and ComplEx (Trouillon et al., 2016) often employ simple linear operators such as addition, subtraction and multiplication. Recent embedding models such as ConvE (Dettmers et al., 2018) and CapsE (Nguyen et al., 2019b) successfully apply deep neural networks to score the triples.\nExisting embedding models are showing promising performances mainly for knowledge graph completion, where the goal is to infer a missing entity given a relation and another entity. But in real applications, less mentioned, such as triple classification (Socher et al., 2013) that aims to predict whether a given triple is valid, and search personalization (Vu et al., 2017) that aims to re-rank the relevant documents returned by a user-oriented search system given a query, these models do not effectively capture potential dependencies among entities and relations from existing triples to predict new triples.\nTo this end, we leverage the relational memory network (Santoro et al., 2018) to propose RMeN to infer a valid fact of new triples. In particular, R-MeN transforms each triple along with adding positional embeddings into a sequence of 3 input vectors. R-MeN then uses a transformer self-attention mechanism (Vaswani et al., 2017) to guide the memory to interact with each input vector to produce an encoded vector. As a result, R-MeN feeds these 3 encoded vectors to a convolutional neural network (CNN)-based decoder to return a score for the triple. In summary, our main contributions are as follows:\n• We present R-MeN – a novel KG embedding model to memorize and encode the potential dependencies among relations and entities for two real applications of triple classification and search personalization.\n• Experimental results show that R-MeN obtains better performance than up-to-date embedding models, in which R-MeN produces new state-of-the-art results on SEARCH17 for the search personalization task, and a new highest accuracy on WN11 and the secondhighest accuracy on FB13 for the triple classification task.",
    "conclusion": "We propose a new KG embedding model, named RMeN, where we integrate transformer self-attention mechanism-based memory interactions with a CNN decoder to capture the potential dependencies in the KG triples effectively. Experimental results show that our proposed R-MeN obtains the new state-of-the-art performances for both the triple classification and search personalization tasks. In future work, we plan to extend R-MeN for multihop knowledge graph reasoning. Our code is available at: https://github.com/daiquocnguyen/ R-MeN."
  },
  {
    "sample_id": 65,
    "title": "A Retrieve-and-Rewrite Initialization Method for Unsupervised Machine Translation",
    "abstract": "The commonly used framework for unsupervised machine translation builds initial translation models of both translation directions, and then performs iterative back-translation to jointly boost their translation performance. The initialization stage is very important since bad initialization may wrongly squeeze the search space, and too much noise introduced in this stage may hurt the final performance. In this paper, we propose a novel retrieval and rewriting based method to better initialize unsupervised translation models. We first retrieve semantically comparable sentences from monolingual corpora of two languages and then rewrite the target side to minimize the semantic gap between the source and retrieved targets with a designed rewriting model. The rewritten sentence pairs are used to initialize SMT models which are used to generate pseudo data for two NMT models, followed by the iterative back-translation. Experiments show that our method can build better initial unsupervised translation models and improve the final translation performance by over 4 BLEU scores.",
    "introduction": "Recent work has shown successful practices of unsupervised machine translation (UMT) (Artetxe et al., 2017; Lample et al., 2017, 2018; Artetxe et al., 2018b; Marie and Fujita, 2018; Ren et al., 2019; Lample and Conneau, 2019). The common framework is to build two initial translation models (i.e., source to target and target to source) and then do iterative back-translation (Sennrich et al., 2016a; Zhang et al., 2018) with pseudo data generated by each other. The initialization stage is important because bad initialization may wrongly squeeze the search space, and too much noise introduced in this stage may hurt the final performance.\nPrevious methods for UMT (Lample et al., 2018; Artetxe et al., 2018b; Marie and Fujita, 2018; Ren et al., 2019) usually use the following n-gram embeddings based initialization. They first build phrase translation tables with the help of unsupervised cross-lingual n-gram embeddings (Conneau et al., 2017; Artetxe et al., 2018a), and then use them to build two initial Phrase-based Statistical Machine Translation (PBSMT) (Koehn et al., 2003) models with two language models. However, there are two problems with their initialization methods. (1) Some complex sentence structures of original training sentences are hard to be recovered with the n-gram translation tables. (2) The initial translation tables inevitably contain much noise, which will be amplified in the subsequent process.\nIn this paper, we propose a novel retrieve-andrewrite initialization method for UMT. Specifically, we first retrieve semantically similar sentence pairs from monolingual corpora of two languages with the help of unsupervised cross-lingual sentence embeddings. Next, with those retrieved similar sentence pairs, we run GIZA++ (Och and Ney, 2003) to get word alignments which are used to delete unaligned words in the target side of the retrieved sentences. The modified target sentences are then rewritten with a designed sequence-to-sequence rewriting model to minimize the semantic gap between the source and target sides. Taking the pairs of the source sentences and corresponding rewritten targets as pseudo parallel data, we then build two initial PBSMT models (source-to-target and targetto-source), which are used to generate pseudo parallel data to warm up NMT models, followed by an iterative back-translation training process. Our code is released at https://github.com/ImagistShuo/RRforUNMT.git.\nOur contributions are threefold. (1) We propose a novel method to initialize unsupervised MT models with a retrieve-and-rewrite schema, which can\nFigure 1: Method overview. (In the figure, “embs” means “embeddings” and “x-lingual” means “cross-lingual”.)\npreserve the rich sentence structure and provide high-quality phrases. (2) We design an effective seq-to-seq architecture based on the Transformer to rewrite sentences with semantic constraints. (3) Our method significantly outperforms the previous non-pre-training based UMT results on en-fr and en-de translation tasks, and give the first unsupervised en-zh translation results on WMT17.",
    "conclusion": "In this paper, we propose a novel method for unsupervised machine translation with a retrieve-andrewrite schema. We first retrieve similar sentences from monolingual corpora and then rewrite the targets with a rewriting model. With the pseudo parallel data, we better initialize PBSMT models and significantly improve the final iteration performance as the experiments show."
  },
  {
    "sample_id": 66,
    "title": "A Review of Cross-Domain Text-to-SQL Models",
    "abstract": "WikiSQL and Spider, the large-scale crossdomain text-to-SQL datasets, have attracted much attention from the research community. The leaderboards of WikiSQL and Spider show that many researchers propose their models trying to solve the text-to-SQL problem. This paper first divides the top models in these two leaderboards into two paradigms. We then present details not mentioned in their original paper by evaluating the key components, including schema linking, pretrained word embeddings, and reasoning assistance modules. Based on the analysis of these models, we want to promote understanding of the text-toSQL field and find out some interesting future works, for example, it is worth studying the text-to-SQL problem in an environment where it is more challenging to build schema linking and also worth studying combing the advantage of each model toward text-to-SQL.",
    "introduction": "Text-to-SQL is a task to translate the natural language query (input) written by users into the SQL query (output) automatically. For example, in Table 3, we want to input the question in the table into the model to get the SQL output. Early work on text-to-SQL focused on small-scale domainspecific databases such as Restaurants, GeoQuery, ATIS, IMDB, and Yelp (Yaghmazadeh et al., 2017; Li and Jagadish, 2014; Iyer et al., 2017; Zelle and Mooney, 1996; Tang and Mooney, 2000; Popescu et al., 2003; Giordani and Moschitti, 2012). More recently, Zhong et al. (2017) proposed the first large-scale cross-domain text-to-SQL dataset, WikiSQL, which attracted much attention from the research community (Xu et al., 2017; Yu et al.; Dong and Lapata, 2018). Now, some models (He et al., 2019; Lyu et al., 2020; Anonymous, 2020) for WikiSQL have achieved over 90% execution accuracy, leading to the impression that the text-to-SQL problem has been solved. However, WikiSQL’s complexity is limited: its SQL queries only cover a single SELECT column and aggregation, together with relatively simple selection predicates in the WHERE clauses, thus lacking in terms of complex SQL queries. To facilitate the study of complex SQL generation, Yu et al. (2018b) introduced Spider, a large-scale cross-domain text-to-SQL benchmark with complex SQL queries. Experiments on Spider show previous models designed for WikiSQL suffer a significant performance drop.\nIn this paper, we discuss the top models for the WikiSQL and Spider benchmarks. Since relatively high generation accuracy has already been achieved for the WikiSQL benchmark, and the SQL structures in Spider cover all SQL structures in WikiSQL, we focus more on models designed for Spider. This paper starts from the comparison of the overall paradigms of the models and then discusses the key modules used by most models. Overall, our contributions are as follows:\n• We divide existing text-to-SQL models into two paradigms:\n1) Generate SQL structure ⇒ Fill schema 2) Label the question ⇒ Generate SQL.\n• We study that pretrained embeddings improve performance by improving schema linking and SQL structure generation.\n• We evaluate the applicability and advantages of the reasoning assistance modules of previous work.\n• We suggest three directions for the future.\n1) How to generate SQL if it is more challenging to build the schema linking.\n2) How to combine the different paradigms (in section 3) toward text-to-SQL.\n3) How to use graph neural networks to improve SQL structure generation.\nSELECT DestAirport FROM Flights )\nTable 1: A complex nested SQL with set operator",
    "conclusion": "We discuss the existing cross-domain SOTA textto-SQL models from the whole to the detailed modules to give a clear picture of the current textto-SQL research progress. We illustrate that pretrained embeddings improve the models by constructing a better schema linking and a more accurate SQL structure through experiments. This paper also provide many details that are not mentioned in the original papers, such as . However, due to space limitations, this paper cannot cover all the details of these SOTA models. We hope this paper will help you understand the key connections and differences between the previous models and have a comprehensive understanding of the text-to-SQL field."
  },
  {
    "sample_id": 67,
    "title": "A Risk-Averse Mechanism for Suicidality Assessment on Social Media",
    "abstract": "Recent studies have shown that social media has increasingly become a platform for users to express suicidal thoughts outside traditional clinical settings. With advances in Natural Language Processing strategies, it is now possible to design automated systems to assess suicide risk. However, such systems may generate uncertain predictions, leading to severe consequences. We hence reformulate suicide risk assessment as a selective prioritized prediction problem over the Columbia Suicide Severity Risk Scale (C-SSRS). We propose SASI, a risk-averse and self-aware transformer-based hierarchical attention classifier, augmented to refrain from making uncertain predictions. We show that SASI is able to refrain from 83% of incorrect predictions on real-world Reddit data. Furthermore, we discuss the qualitative, practical, and ethical aspects of SASI for suicide risk assessment as a human-in-the-loop framework.",
    "introduction": "Suicide is a global phenomenon responsible for 1.3% of deaths worldwide (WHO, 2019). While it is the leading cause of death among 14-35 year olds in the US (Hedegaard et al., 2021), suicide rates have increased by 13% in Japan between July to September 2020 (Tanaka and Okamoto, 2021). It hence becomes critical to extend clinical and psychiatric care, which relies heavily on identifying those at risk. While 80% of patients do not undergo clinical treatment, 60% of those who succumbed to suicide denied having suicidal thoughts to mental health experts (McHugh et al., 2019). However, studies show eight out of ten people shared suicidal thoughts on social media (Golden et al., 2009).\nThe advent of Natural Language Processing (NLP) shows promise for suicide risk assessment based on online user behavior (Ji et al., 2021b;\nFigure 1: End-to-end pipeline for suicide risk assessment. When SASI assesses the posts, it returns the predicted risk level along with a certainty score. With a human-in-the-loop framework, these predictions can be sorted into various risk levels. SASI assigns high priority to uncertain predictions, for an immediate review by mental health experts.\nChoudhury et al., 2016), with automatic risk assessment algorithms outperforming traditional clinical methods (Coppersmith et al., 2018; Linthicum et al., 2019). Numerous deep learning methods already exist, which include leveraging suiciderelated word-embeddings (Cao et al., 2019), social graphs (Mishra et al., 2019; Sinha et al., 2019; Cao et al., 2022; Sawhney et al., 2021b) and historical context (Matero et al., 2019; Gaur et al., 2019).\nHowever, mental health is a safety-critical realm, where technological failure could lead to severe harm to users on social media (Sittig and Singh, 2015). One such case was covered by Register (2020), wherein a medical bot suggested a mock patient kill themselves, demonstrating that unintended harmful behavior can emerge from AI systems (Amodei et al., 2016; Chandler et al., 2020).\nDespite the significant power of traditional NLP methods, such models are inherently designed to make a prediction even when not confident. This poses a challenge when working with critical tasks like suicide risk assessment, for which it may be hard to make a prediction due to various reasons such as task hardness or contained ambiguity. Such a system may associate a lower risk level to a user who needs urgent help. A resulting delayed response from mental health experts may lead to adverse consequences. We hence need systems that assign high priority to uncertain predictions, for immediate review and response.\nContributions: We reformulate suicide risk assessment as a prioritized prediction task which factors in uncertainty, and propose SASI: A Risk-Averse Mechanism for Suicidality Assessment on Social MedIa. SASI is risk-averse in the sense that it is self-aware, as it incorporates a selection function to measure uncertainty. Based on a set threshold value, SASI refrains from making a prediction when it is uncertain. We show that SASI can act as a tool to efficiently prioritize users who need immediate attention. Through a human-in-the-loop framework that involves a domain expert, SASI assigns high priority to uncertain predictions to avoid critical failure (Figure 1). We demonstrate the effectiveness of SASI using a real-world gold standard Reddit dataset. Through a series of experiments, we show SASI refrains from making 83% of incorrect predictions. We further demonstrate its effectiveness through a qualitative study and discuss the ethical implications.",
    "conclusion": "With a motivation to provide a robust solution to fine-grained suicide risk assessment on social media, we present SASI, a framework that integrates the concept of selective prioritization to existing deep learning based risk-assessment techniques. SASI is self-aware, wherein it refrains from making a prediction when uncertain, and instead assigns high priority to such data samples for immediate review by mental health experts. We demonstrated the effectiveness of SASI through quantitative evaluations on real-world data, wherein SASI avoided high-risk situations by refraining from making 83% of incorrect predictions. Through a qualitative analysis, we described how SASI can be used as a part of a human-in-the-loop framework, facilitating efficient responses from mental health experts."
  },
  {
    "sample_id": 68,
    "title": "A Scalable Framework for Table of Contents Extraction from Complex ESG Annual Reports",
    "abstract": "Table of contents (ToC) extraction centres on structuring documents in a hierarchical manner. In this paper, we propose a new dataset, ESGDoc, comprising 1,093 ESG annual reports from 563 companies spanning from 2001 to 2022. These reports pose significant challenges due to their diverse structures and extensive length. To address these challenges, we propose a new framework for Toc extraction, consisting of three steps: (1) Constructing an initial tree of text blocks based on reading order and font sizes; (2) Modelling each tree node (or text block) independently by considering its contextual information captured in node-centric subtree; (3) Modifying the original tree by taking appropriate action on each tree node (Keep, Delete, or Move). This construction-modellingmodification (CMM) process offers several benefits. It eliminates the need for pairwise modelling of section headings as in previous approaches, making document segmentation practically feasible. By incorporating structured information, each section heading can leverage both local and long-distance context relevant to itself. Experimental results show that our approach outperforms the previous state-of-theart baseline with a fraction of running time. Our framework proves its scalability by effectively handling documents of any length.1",
    "introduction": "A considerable amount of research has been proposed to comprehend documents (Xu et al., 2019; Zhang et al., 2021; Xu et al., 2021a,b; Peng et al., 2022; Li et al., 2022; Gu et al., 2022; Shen et al., 2022; Lee et al., 2022, 2023) , which typically involves the classification of different parts of a document such as title, caption, table, footer, and so on. However, such prevailing classification often centres on a document’s local layout structure, sidelining a holistic comprehension of its content and organisation. While traditional summarisation offers a concise representation of a document’s content, a Table of Contents (ToC) presents a structured and hierarchical summary. This structural organisation in a ToC provides a comprehensive pathway for pinpointing specific information. For example, when seeking information about a company’s carbon dioxide emissions, a ToC enables a systematic navigation through the information hierarchy. In contrast, conventional summarisation might only provide a vague indication of such information, requiring sifting through the entire document for precise detail.\nSeveral datasets have been proposed to facilitate the research in document understanding (Zhong et al., 2019b; Li et al., 2020; Pfitzmann et al., 2022). Most of these studies lack a structured construction of documents and primarily focus on wellstructured scientific papers. A dataset called HierDoc (Hierarchical academic Document) (Hu et al., 2022) was introduced to facilitate the development of methods for extracting the table of contents (ToC) from documents. This dataset was compiled from scientific papers downloaded from arXiv2, which are typically short and well-structured. The hierarchical structure can often be inferred directly from the headings themselves. For example, the heading “1. Introduction” can be easily identified as a first-level heading based on the section numbering. Moreover, due to the relatively short length of scientific papers, it is feasible to process the entire document as a whole. Hu et al. (2022) proposed the multimodal tree decoder (MTD) for ToC extraction from HierDoc. MTD first utilises text, visual, and layout information to encode text blocks identified by a PDF parser; then classifies all text blocks into two categories, headings and non-headings; and finally predicts the relationship of each pair of headings, facilitating the parsing of these headings into a tree structure representing ToC.\nFigure 1: Five examples of ESG reports, with the left three presented in portrait orientation and the right-most two in landscape orientation. They show a wide range of diverse structures. It is common to observe the absence of section numbering.\nHowever, understanding long documents such as ESG (Environmental, Social, and Governance) annual reports poses significant challenges compared to commonly used scientific papers. First, ESG reports tend to be extensive, often exceeding 100 pages, which is uncommon for scientific papers. Second, while scientific papers generally adhere to a standard structure that includes abstract, introduction, methods, results, discussion, and conclusion sections, ESG reports exhibit more diverse structures with a wide range of font types and sizes. Third, ESG reports often include visual elements such as charts, graphs, tables, and infographics to present data and key findings in a visually appealing manner, which adds complexity to the document parsing process. Some example ESG reports are illustrated in Figure 1.\nIn this paper, we develop a new dataset, ESGDoc, collected from public ESG annual reports3 from 563 companies spanning from 2001 to 2022 for the task of ToC extraction. The existing approach, MTD (Hu et al., 2022), faces difficulties when dealing with challenges presented in ESGDoc. MTD models relationships of every possible heading pairs and thus requires the processing of the entire document simultaneously, making it impractical for lengthy documents. As will be discussed in our experiments section, MTD run into out-of-memory issue when processing some lengthy documents in ESGDoc. Moreover, MTD only uses Gated Recurrent Unit (GRU) (Cho et al., 2014) to capture the context of a section heading, lacking long-distance interaction, particularly for high-level headings that may be tens of pages apart.\nIn order to overcome the challenges presented in ESGDoc, we propose a new scalable framework, consisting of three main steps: (1) Constructing an initial tree of text blocks based on reading order and font sizes; (2) Modelling each tree node (or text block) independently by considering its contextual information captured in node-centric subtree; (3) Modifying the original tree by taking appropriate action on each tree node (Keep, Delete, or Move). Our method is named as CMM (ConstructionModelling-Modification).\nThis approach allows higher-level headings to focus on capturing high-level and long-distance information, while lower-level headings focus more on local information. Additionally, CMM also models each heading independently, removing the need for modelling pairwise relationships among headings and enabling more effective document segmentation. Here, we can divide documents based on the tree structure instead of relying on page divisions. This ensures that each segment maintains both local and long-distance relationships, preserving the long-distance connections that would be lost if division were based on page boundaries. As CMM does not require the processing of a document as a whole, it can be easily scaled to deal with lengthy documents. Experimental results show that our approach outperforms the previous state-of-theart baseline with only a fraction of running time, verifying the scalability of our model as it is applicable to documents of any length. Our main contributions are summarised as follows:\n• We introduce a new dataset, ESGDoc, comprising 1,093 ESG annual reports specifically designed for table of contents extraction.\n• We propose a novel framework that processes documents in a construction-modellingmodification manner, allowing for the decoupling of each heading, preserving both local and long-distance relationships, and incorporating structured information.\n• We present a novel graph-based method for document segmentation and modelling, enabling the retention of both local and longdistance information within each segment.",
    "conclusion": "In this paper, we have constructed a new dataset, ESGDoc, and proposed a novel framework, CMM, for table of contents extraction. Our pipeline, consisting of tree construction, node-centric subtree modelling, and tree modification stages, effectively addresses the challenges posed by the diverse structures and lengthy nature of documents in ESGDoc. The methodology of representing a document as an initial full tree, and subsequently predicting node operations for tree modification, and further leveraging the tree structure for document segmentation, can provide valuable insights for other document analysis tasks."
  },
  {
    "sample_id": 69,
    "title": "A Self-training Framework for Automated Medical Report Generation",
    "abstract": "Medical report generation, focusing on automatically generating accurate clinical findings from medical images, is an important medical artificial intelligence task. It reduces the workload of physicians in writing reports. Many of the current methods depend heavily on labeled datasets that include image-report pairs, but such datasets labeled by physicians are hard to acquire in clinical practice. In this paper, we introduce a self-training framework named REMOTE (i.e., Revisiting sElf-training for Medical repOrT gEneration) to exploit the unlabeled medical images and a MedCLIPScore to augment a small-scale dataset for training the medical report generation model. Experiments conducted on the MIMIC-CXR benchmark dataset and a COVID-19 dataset demonstrate that, our REMOTE framework, using only 1% labeled training data, achieves competitive performance with previous methods that are trained on entire training data.",
    "introduction": "Generating medical reports automatically involves producing clinical descriptions based on the input visual medical images (Jing et al., 2018, 2019; Li et al., 2018; Liu et al., 2021b). This is similar to the task of image captioning (Xu et al., 2015; Chen et al., 2015), which aims to generate visual descriptions to describe the input images. Therefore, based on the benchmark dataset MIMIC-CXR (Johnson et al., 2019), inspired by the success of image captioning, various state-of-the-art data-driven models, especially those based on the encoder-decoder structure (Chen et al., 2020; Liu et al., 2021b; Wang et al., 2022a), have achieved significant advancements. However, medical data labeling requires specialized expertise from physicians and also involves privacy concerns. Therefore, acquiring medical report generation datasets is time-consuming and costly (Liu et al., 2021c). As a result, when compared to datasets used for general image captioning datasets such as Conceptual Captions (Soricut et al., 2018), the size of the medical dataset MIMIC-CXR is relatively small. This size limitation becomes a challenge when dealing with novel diseases like COVID-19, where collecting and labeling adequate training data promptly is difficult. It hinders the application of existing medical report generation models in addressing novel diseases to alleviate the workload of physicians efficiently.\nConsidering that there are a lot of public imageonly datasets, e.g., CheXpert (Irvin et al., 2019), RSNA Pneumonia (Shih et al., 2019), COVID images (Rahman et al., 2021), in the literature. To this end, we propose a self-training framework REMOTE, which enhances the performance of the medical report generation model by simultaneously utilizing high-quality paired image-report datasets and image-only datasets. In implementation, we adopt the Noisy Student self-training framework (Xie et al., 2020; He et al., 2020) as the basis to build our REMOTE for medical report generation, which consists of a “teacher” model and a “student” model. It begins by training a “teacher” model on the high-quality annotated image-report pairs, e.g., MIMIC-CXR dataset (Johnson et al., 2019). Subsequently, the teacher model is used to generate pseudo-reports for medical images in the imageonly dataset without annotated reports. We then employ MedCLIPScore to score each generated pseudo image-report pair and filter out low-scoring pseudo image-report pairs. Finally, we train a “student” model on both the annotated high-quality image-report pairs and the generated pseudo imagereport pairs. In the next training step, we consider the “student” model as the new “teacher” model, and by repeating the above steps, we can generate new pseudo image-report pairs and train new “student” models. Through iterating these steps, we ultimately obtain an accurate and robust medical report generation model.\nIt is worth noting that, while the self-training framework has been explored in uni-modal tasks such as image classification (Xie et al., 2020) and machine translation (He et al., 2020), self-training in medical report generation has not been well explored. This is because medical report generation is a multi-modal medical task, incorporating disparities between the visual and the textual modalities. Thus, inspired by the great success of CLIP (Radford et al., 2021), which is trained to align image and text modalities, we follow the CLIPScore (Hessel et al., 2021) to construct the MedCLIPScore to obtain a high-quality pseudo image-report pairs. In detail, we train the MedCLIP (Wang et al., 2022b) on the MIMIC-CXR dataset, and use it as MedCLIPScore to boost the performance and robustness of the medical report generation model.\nOverall, the main contributions of this paper are as follows:\n• Based on the noisy student self-training framework, we propose a self-training framework REMOTE for automated medical report generation with limited labeled training data.\n• Our proposed method includes three components: “teacher” model, MedCLIPScore, and “student” model. The “teacher” model and MedCLIPScore focus on obtaining highquality pseudo image-report pairs from imageonly datasets, which are used to obtain a robust “student” model. By taking the “student” model as the new “teacher” model and iterating the above steps, REMOTE can achieve strong performances with limited labeled data.\n• Experiments on two datasets show that our method can achieve competitive results with existing fully-supervised methods with only 1% labeled training data.",
    "conclusion": "In this work, we presented a novel self-training framework, REMOTE, aimed at boosting the performance of medical report generation from unlabeled visual medical images, especially in scenarios with limited labeled training data. By harnessing the power of both high-quality paired imagereport datasets and unlabeled image-only datasets, our approach effectively reduces the reliance on extensive labeled training data, which are both timeconsuming and costly to obtain. Our REMOTE introduces a “teacher” model and a “student” model to generate pseudo image-report pairs and refine the generation, respectively. Specifically, we introduced MedCLIPScore to ensure the quality of the generated pseudo image-report pairs. The experiments on the widely-used benchmark dataset MIMIC-CXR and a COVID-19 dataset validated the robustness and effectiveness of REMOTE. In particular, using only 1% of labeled training data, our approach could achieve competitive performances comparable to fully-supervised state-ofthe-art methods."
  },
  {
    "sample_id": 70,
    "title": "A Sequential Flow Control Framework for Multi-hop Knowledge Base Question Answering",
    "abstract": "One of the key challenges of knowledge base question answering (KBQA) is the multi-hop reasoning. Since in different hops, one attends to different parts of question, it is important to dynamically represent the question semantics for each hop. Existing methods, however, (i) infer the dynamic question representation only through coarse-grained attention mechanisms, which may bring information loss, (ii) and have not effectively modeled the sequential logic, which is crucial for the multi-hop reasoning process in KBQA. To address these issues, we propose a sequential reasoning selfattention mechanism to capture the crucial reasoning information of each single hop in a more fine-grained way. Based on Gated Recurrent Unit (GRU) which is good at modeling sequential process, we propose a simple but effective GRU-inspired Flow Control (GFC) framework to model sequential logic in the whole multi-hop process. Extensive experiments on three popular benchmark datasets have demonstrated the superior effectiveness of our model. In particular, GFC achieves new state-of-the-art Hits@1 of 76.8% on WebQSP and is also effective when KB is incomplete. Our code and data are available at https: //github.com/Xie-Minghui/GFC.",
    "introduction": "Knowledge base question answering (KBQA) aims to answer questions from structured knowledge bases. In real application scenarios of KBQA, reasoning with multiple hops over knowledge graph (KG) is necessary for answering complex questions. Therefore, how to perform multi-hop reasoning effectively becomes a key challenge for multi-hop KBQA task (Sun et al., 2018; Zhang et al., 2018; Ho et al., 2020; Shi et al., 2020; Han et al., 2020).\nExisting methods for multi-hop KBQA have three main strands. The first is semantic parsing\nhop 1\nhop 2\nr 1 r 1 r 2 r 2\nFigure 1: The above picture shows relations attention weights on the reasoning paths of GFC and the strong path-based method TransferNet. The final entity scores are the weighted sum of two hops which are positive correlation with relation attention weights. TransferNet tends to give r 1 high score in the 2nd hop, thus obtaining wrong answer (right). GFC can effectively weaken the attention of r 1 in the 2nd hop by introducing GRUlike sequential logic into the multi-hop process (left). People tend to pay more attention to current relations while pay less attention to past relations. Thus GFC is more consistent with human reasoning habit.\nbased methods, which generate query graphs or statements by parsing questions (Yih et al., 2015; Luo et al., 2018; Lan and Jiang, 2020). The second is embedding-based methods which score the embeddings of question objectives and candidate answers (Dong et al., 2015; Miller et al., 2016; Hao et al., 2017; Saxena et al., 2020). The third is pathbased methods, which start from topic entities of question and walk on KG to find answers. The third direction has its own advantages in terms of interpretability and extensibility (Sen et al., 2021). In recent years, more and more works have focused on path-based multi-hop reasoning methods (He et al., 2021; Sen et al., 2021; Shi et al., 2021).\nHowever, existing methods still face some critical problems. First, path-based methods and some embedding-based methods usually leverage coarsegrained attention mechanisms to capture reasoning information of each hop. For example, KVMemNN (Xu et al., 2019) adopts cross-attention between key-value memory and sentence-level question representation. IRN (Zhou et al., 2018) uses the sentence-level question representation to eliminate relation embeddings of previous hop. Some methods (He et al., 2021; Shi et al., 2021) adopt crossattention between the sentence-level question representation and question tokens. However, compressing all the necessary information into the sentencelevel representation may lose some crucial information. Although these methods have achieved good performance, there is still room for improvement.\nSecond, they lack modeling sequential logic effectively in the whole multi-hop process. Humans often reason sequentially and consider past and present information comprehensively, which is a kind of sequential logic. However, the dynamic question representation of each hop is relatively independent (Cohen et al., 2020; Shi et al., 2021). And they do not control information flow effectively in different hops. For example, in Figure 1, models need to inhibit past relations for getting the right answer. However, existing methods cannot do this well.\nIn response, we propose a novel model for multihop KBQA, dubbed GFC. First, we design a sequential reasoning self-attention mechanism to obtain more fine-grained reasoning information of each hop. Our update mechanism combines the self-attention mechanism with sequential logic in the reasoning scenario. It can capture more nuanced reasoning information to distinguish similar relations on KG. Second, we design a simple but effective GRU-inspired flow control framework to model the sequential logic in the whole multi-hop process more effectively. This framework controls reasoning information flow among different hops, which enables GFC to consider reasoning information of past and present comprehensively. Besides, it tactfully integrates the proposed update mechanism into itself through our heuristic thinking about GRU. Inspired by the gating mechanism of GRU, we also introduce a self-gate unit to filter out redundant past reasoning information. As integral parts of framework, these mechanisms further enhance the capability of the overall flow control framework. Our key contributions are as follows:\n• We design a sequential reasoning selfattention mechanism to extract the crucial reasoning information of single hop in a more fine-grained way.\n• We propose a GRU-inspired flow control framework to model the sequential logic in the whole multi-hop process more effectively.\n• Through controlling reasoning flow among hops and our novel update mechanism, GFC is superior to most existing methods. Specially, GFC achieves new state-of-the-art Hits@1 result of 76.8% on WebQSP and is also highly effective when KB is incomplete.",
    "conclusion": "In this paper, we design (i) a sequential reasoning self-attention mechanism to extract the crucial reasoning information of each single hop in a more fine-grained way and (ii) a GRU-inspired flow control framework to model sequential logic in the whole multi-hop process more effectively. Experimental results show the superior performance of GFC. Specially, GFC achieves new state-of-the-art Hits@1 performance on WebQSP. GFC also shows its high effectiveness when KB is incomplete. As a path-based method, GFC not only has better interpretability and extensibility, but also has better performance. In future work, we plan to investigate further on how to model the multi-hop reasoning process using the structures of language models."
  },
  {
    "sample_id": 71,
    "title": "A Synthetic Data Generation Framework for Grounded Dialogues",
    "abstract": "Training grounded response generation models often requires a large collection of grounded dialogues. However, it is costly to build such dialogues. In this paper, we present a synthetic data generation framework (SynDG) for grounded dialogues. The generation process utilizes large pre-trained language models and freely available knowledge data (e.g., Wikipedia pages, persona profiles, etc.). The key idea of designing SynDG is to consider dialogue flow and coherence in the generation process. Specifically, given knowledge data, we first heuristically determine a dialogue flow, which is a series of knowledge pieces. Then, we employ T5 to incrementally turn the dialogue flow into a dialogue. To ensure coherence of both the dialogue flow and the synthetic dialogue, we design a two-level filtering strategy, at the flow-level and the utterance-level respectively. Experiments on two public benchmarks show that the synthetic grounded dialogue data produced by our framework is able to significantly boost model performance in both full training data and low-resource scenarios.",
    "introduction": "Grounded dialogue systems are designed to engage in conversation with humans by incorporating external knowledge to provide relevant and informative responses (Ghazvininejad et al., 2018; Dinan et al., 2019; Gopalakrishnan et al., 2019; Zhou et al., 2018b). In recent years, various advanced techniques have been developed to train grounded dialogue models (Zheng et al., 2020; Cui et al., 2021; Xu et al., 2022; Li et al., 2022a). Despite the notable progress, training these models often requires large amounts of data. However, it is expensive and time-consuming to build a collection\nDialogue Dialogue Flow\nFigure 1: An illustrated example from the Wizard of Wikipedia dataset (Dinan et al., 2019). This example shows the dialogue flow in knowledge-grounded dialogues, i.e., a sequence of knowledge pieces. As each agent response is grounded to a specific piece of knowledge, the dialogue flow implies the outline of the conversation.\nof dialogue data that is naturally grounded on documents or knowledge (Li et al., 2020, 2022b).\nOne solution is to generate grounded dialogue data from unstructured knowledge, by using large pre-trained language models (LMs). Previous work on this topic has explored synthetic dialogue data generation with reinforcement learning (Lin et al., 2022) or user simulation (Wu et al., 2022). However, a key missing component in all these methods is the modeling of dialogue flow.\nDialogue flow can be viewed as the outline of a dialogue. The flow reflects the dialogue’s content and trajectory, i.e., the topics discussed in each session and the topic shifts between sessions. We consider the dialogue flow of a grounded dialogue as the sequence of the grounded knowledge pieces. Figure 1 shows an example dialogue along with its associated dialogue flow. In this example, the grounded knowledge is primarily from a Wikipedia page about “husky” dogs. This dialogue follows a smooth knowledge flow, transitioning from “husky” to “sled dogs” and then to “huskies as pets”. However, if we replace the second knowledge piece with “‘Esquimaux’ or ‘Eskimo’ was a common term for pre-Columbian Arctic inhabitants of North America.”, which is also from the same Wikipedia page, then the flow becomes less consistent. As the backbone guiding the dialogue generation process, a carefully planned dialogue flow is crucial for the coherence and smoothness of the resulting dialogue.\nTo this end, we propose a novel framework named SynDG, to synthetically generate coherent grounded dialogues. The generated dialogues are meant to be used as auxiliary training data. In SynDG, we first determine the dialogue flow by task-specific heuristics, from the unstructured knowledge data (e.g., Wikipedia pages, persona profiles, etc.). Then, we employ T5 (Raffel et al., 2020), a large pre-trained LM, to transform the dialogue flow into a synthetic dialogue, with sequential utterance generation, one at a time. To ensure the quality of the synthetic dialogue, we propose a two-level filtering strategy based on T5: flow-level filtering and utterance-level filtering. The flowlevel filtering is designed to select dialogue flows with higher consistency, whereas the utterancelevel filtering aims to eliminate the synthetic dialogues with poor coherence.\nWe conduct experiments on two grounded dialogue benchmarks, in both full training data and low-resource scenarios. We use the synthetic grounded dialogue data produced by our framework as additional training data for commonly used grounded dialogue models. Both the automatic and human evaluation results show that our synthetic data leads to significant improvement on model performance. Further analysis also reveals that model performance increases along the increase in the number of synthetic dialogues.",
    "conclusion": "In this paper, we propose a framework, SynDG, to automatically construct synthetic training data for the grounded dialogue task. We first construct dialogue flows based on unstructured knowledge, then transform them into synthetic dialogues by large LMs, and finally filter and retain the generated dialogues with high quality. The experimental results demonstrate the effectiveness of our proposed framework in both full training data and lowresource scenarios. Further analysis shows that the model performance tends to increase as the number of synthetic dialogues increases. For future work, we plan to investigate more efficient strategies for determining dialogue flows and take larger LMs to produce synthetic dialogues with higher quality."
  },
  {
    "sample_id": 72,
    "title": "A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation",
    "abstract": "This work studies the widely adopted ancestral sampling algorithms for auto-regressive language models, which is not widely studied in the literature. We use the quality-diversity (QD) trade-off to investigate three popular sampling algorithms (top-k, nucleus and tempered sampling). We focus on the task of open-ended language generation. We first show that the existing sampling algorithms have similar performance. After carefully inspecting the transformations defined by different sampling algorithms, we identify three key properties that are shared among them: entropy reduction, order preservation, and slope preservation. To validate the importance of the identified properties, we design two sets of new sampling algorithms: one set in which each algorithm satisfies all three properties, and one set in which each algorithm violates at least one of the properties. We compare their performance with existing sampling algorithms, and find that violating the identified properties could lead to drastic performance degradation, as measured by the Q-D trade-off. On the other hand, we find that the set of sampling algorithms that satisfies these properties performs on par with the existing sampling algorithms.1",
    "introduction": "A language model (LM) is a central module for natural language generation (NLG) tasks (Young et al., 2018) such as machine translation (Wu et al., 2018), dialogue response generation (Li et al., 2017), image captioning (Lin et al.), and related tasks. Given a trained LM, finding the best way to generate a sample from it has been an important challenge for NLG applications.\nFigure 1: Human evaluation (y-axis: quality, x-axis: diversity, both are the bigger the better) shows that the generation performance of existing sampling algorithms are on par with each other.\nDecoding, i.e., finding the most probable output sequence from a trained model, is a natural principle for generation. The beam-search decoding algorithm approximately finds the most likely sequence by performing breadth-first search over a restricted search space. It has achieved success in machine translation, summarization, image captioning, and other subfields.\nHowever, in the task of open-ended language generation (which is the focus of this work), a significant degree of diversity is required. For example, conditioned on the prompt “The news says that ...”, the LM is expected to be able to generate a wide range of interesting continuations. While the deterministic behavior of decoding algorithms could give high-quality samples, they suffer from a serious lack of diversity.\nThis need for diversity gives rise to a wide adoption of various sampling algorithms. Notably, topk sampling (Fan et al., 2018), nucleus sampling (Holtzman et al., 2020), and tempered sampling (Caccia et al., 2020) have been used in open-ended generation (Radford et al., 2018; Caccia et al., 2020), story generation (Fan et al., 2018), and dialogue response generation (Zhang et al., 2020b). However, the sampling algorithm and the hyperparameter are usually chosen via heuristics, and a comprehensive comparison between existing sampling algorithm is lacking in the literature. More importantly, the underlying reasons behind the success of the existing sampling algorithms still remains poorly understood.\nIn this work, we begin by using the qualitydiversity (Q-D) trade-off (Caccia et al., 2020) to compare the three existing sampling algorithms. For automatic metrics, we use the BLEU score for quality and n-gram entropy for diversity. We also correlate these automatic metrics with human judgements. The first observation we draw is that top-k , nucleus and tempered sampling perform on par in the Q-D trade-off, as shown in Figure 1. Motivated by this result, we extract three key properties by inspecting the transformations defined by the sampling algorithms: (1) entropy reduction, (2) order preservation and (3) slope preservation. We prove all three properties hold for the three existing sampling algorithms.\nWe then set out to systematically validate the importance of the identified properties. To do so, we design two sets of new sampling algorithms in which each algorithm either violates one of the identified properties, or satisfies all properties. Using the Q-D trade-off, we compare their efficacy against existing algorithms, and find that violating these identified properties could result in significant performance degradation. More interestingly, we find that the set of sampling algorithms that satisfies these properties has generation performance that matches the performance of existing sampling algorithms.",
    "conclusion": "This work studies sampling algorithms for the openended language generation task. We show that the existing algorithms, namely top-k, nucleus, and tempered sampling, have similar generation performance as measured by the quality-diversity tradeoff evaluation. Motivated by this result, we identify three key properties that we prove are shared by the existing algorithms. To validate the importance of these identified properties, we design a set of new sampling algorithms, and compare their performance with the existing sampling algorithms. We find that violation of the identified properties may lead to drastic performance degradation. On the other hand, we propose several novel algorithms, namely random top-k and max entropy sampling, that meet the identified properties. We find that their generation performance is on par with the existing algorithms."
  },
  {
    "sample_id": 73,
    "title": "ACL_2020_TaleOfPerplexities (15).pdf",
    "abstract": "In recent years there has been a burgeoning interest in the use of computational methods to distinguish between elicited speech samples produced by patients with dementia, and those from healthy controls. The difference between perplexity estimates from two neural language models (LMs) - one trained on transcripts of speech produced by healthy participants and the other trained on transcripts from patients with dementia - as a single feature for diagnostic classification of unseen transcripts has been shown to produce state-of-the-art performance. However, little is known about why this approach is effective, and on account of the lack of case/control matching in the most widely-used evaluation set of transcripts (DementiaBank), it is unclear if these approaches are truly diagnostic, or are sensitive to other variables. In this paper, we interrogate neural LMs trained on participants with and without dementia using synthetic narratives previously developed to simulate progressive semantic dementia by manipulating lexical frequency. We find that perplexity of neural LMs is strongly and differentially associated with lexical frequency, and that a mixture model resulting from interpolating control and dementia LMs improves upon the current state-of-the-art for models trained on transcript text exclusively.",
    "introduction": "Alzheimer’s Disease (AD) is a debilitating neurodegenerative condition which currently has no cure, and Dementia of the Alzheimer’s Type (DAT) is one of the most prominent manifestations of AD pathology. Prior to availability of diseasemodifying therapies, it is important to focus on reducing the emotional and financial burden of this devastating disease on patients, caregivers, and the healthcare system. Recent longitudinal studies of aging show that cognitive manifestations of future dementia may appear as early as 18 years prior to clinical diagnosis - much earlier than previously believed (Rajan et al., 2015; Aguirre-Acevedo et al., 2016). With 30-40% of healthy adults subjectively reporting forgetfulness on a regular basis (Cooper et al., 2011), there is an urgent need to develop sensitive and specific, easy-to-use, safe, and costeffective tools for monitoring AD-specific cognitive markers in individuals concerned about their cognitive function. Lack of clear diagnosis and prognosis, possibly for an extended period of time (i.e., many years), in this situation can produce uncertainty and negatively impact planning of future care (Stokes et al., 2015), and misattribution of AD symptoms to personality changes can lead to family conflict and social isolation (Boise et al., 1999; Bond et al., 2005). Delayed diagnosis also results in an estimated $7.9 trillion in medical and care costs (Association, 2018) due to high utilization of emergency care, amongst other factors, by patients with undiagnosed AD.\nCognitive status is reflected in spoken language. As manual analysis of such data is prohibitively time-consuming, the development and evaluation of computational methods through which symptoms of AD and other dementias can be identified on the basis of linguistic anomalies observed in transcripts of elicited speech samples have intensified in the last several years (Fraser et al., 2016; Yancheva and Rudzicz, 2016; Orimaye et al., 2017). This work has generally employed a supervised machine learning paradigm, in which a model is trained to distinguish between speech samples produced by patients with dementia and those from controls, using a set of deliberately engineered or computationally identified features. However, on account of the limited training data available, overfitting is a concern. This is particularly problematic in DAT, where the nature of linguistic anomalies varies between patients, and with AD progression (Altmann and McClung, 2008).\nIn the current study we take a different approach, focusing our attention on the perplexity of a speech sample as estimated by neural LMs trained on transcripts of the speech of participants completing a cognitive task. To date, the most successful approach to using LM perplexity as a sole distinguishing feature between narratives by dementia patients and controls was proposed by Fritsch et al. (2019) and replicated by Klumpp et al. (2018). The approach consists of training two recurrent neural LMs - one on transcripts from patients with dementia and the other on transcripts from controls. The difference between the perplexities estimated with these two LMs results in very high classification accuracy (AUC: 0.92) reported by both studies.\nThe explanation for this performance offered by Fritsch et al. (2019) relies on observations that patients with DAT describe the picture in an unforeseen way and their speech frequently diverts from the content of the picture, contains repetitions, incomplete utterances, and refers to objects in the picture using words like “thing” or “something”. This explanation, however, conflicts with the findings by Klumpp et al. (2018) that demonstrate similarly high classification accuracy (AUC: 0.91) with a single hidden layer non-recurrent neural network and bag-of-words input features, suggesting that while word sequences play a role, it may not be as large as previously believed by Fritsch et al. (2019). Klumpp et al.’s (2018) explanation contrasts “local” with “global language properties” of the picture descriptions being captured by recurrent neural LMs vs. the non-recurrent bag-of-words neural network classifier, respectively. Both of these explanations are based on informal qualitative observations of the data and are not entirely satisfying because both fail to explain the fact that it is precisely the difference between the control and dementia LMs that is able to discriminate between patients and controls. The individual LMs are not nearly as good at this categorization task.\nThe objective of the current study is to quantify the extent to which the differences between neural LMs trained on language produced by DAT patients and controls reflect known deficits in language use in this disease - in particular the loss of access to relatively infrequent terms that occurs with disease progression (Almor et al., 1999a). We approach this objective by interrogating trained neural LMs with two methods: interrogation by perturbation in which we evaluate how trained neural LMs respond to text that has been deliberately perturbed to simulate AD progression; and interrogation by interpolation in which we develop and evaluate hybrid LMs by interpolating between neural LMs modeling language use with and without dementia. We find neural LMs are progressively more perplexed by text simulating disease of greater severity, and that this perplexity decreases with increasing contributions of a LM trained on transcripts from patients with AD, but increases again when only this LM is considered. Motivated by these observations, we modify the approach of Fritsch et al. (2019) by incorporating an interpolated model and pre-trained word embeddings, with improvements in performance over the best results reported for models trained on transcript text exclusively.",
    "conclusion": "We offer an empirical explanation for the success of the difference between neural LM perplexities in discriminating between DAT patients and controls, involving lexical frequency effects. Interrogation of control- and dementia-based LMs using synthetic transcripts and interpolation of parameters reveals inconsistencies harmful to model performance that can be remediated by incorporating interpolated models and pre-trained embeddings, with significant performance improvements."
  },
  {
    "sample_id": 74,
    "title": "A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation",
    "abstract": "Large pretrained generative models like GPT3 often suffer from hallucinating non-existent or incorrect content, which undermines their potential merits in real applications. Existing work usually attempts to detect these hallucinations based on a corresponding oracle reference at a sentence or document level. However ground-truth references may not be readily available for many free-form text generation applications, and sentence- or documentlevel detection may fail to provide the finegrained signals that would prevent fallacious content in real time. As a first step to addressing these issues, we propose a novel token-level, reference-free hallucination detection task and an associated annotated dataset named HADES (HAllucination DEtection dataSet) 1. To create this dataset, we first perturb a large number of text segments extracted from English language Wikipedia, and then verify these with crowd-sourced annotations. To mitigate label imbalance during annotation, we utilize an iterative model-in-loop strategy. We conduct comprehensive data analyses and create multiple baseline models.",
    "introduction": "Automatic text generation using neural natural language generation (NLG) systems is increasingly fluent and thus seemingly plausible in many realworld applications. Large-scale pretrained models like GPT-3 (Brown et al., 2020) are proven to be powerful in understanding and performing free form text generation tasks at human-quality level with a few in-context examples, which dramatically reduces the manual labor needed in many text-based applications and services. Despite their great success, however, neural NLG systems using very large pre-trained models struggle to generate factually accurate and trustworthy text (Devlin et al., 2019; Radford et al., 2019), and exhibit a propensity to hallucinate non-existent or incorrect content that is unacceptable in most user-oriented applications. This poses a major challenge for deploying production NLG systems with realtime generation, where post-examination is impossible.\nExisting work has sought to detect hallucination and quantitatively measure generation consistency against a provided reference. Such reference-based hallucination detection has been proposed for abstractive summarization (Maynez et al., 2020), machine translation (Wang and Sennrich, 2020), datato-text generation (Rebuffel et al., 2021), and image caption generation (Rohrbach et al., 2018). For many free-form text generation tasks, however, references are not readily available. For example, in a production NLG system such as a social chatbot using real-time response generation or a document auto-completion system, the generation model often cannot pair its outputs with sufficient reference information, rendering reference-based methods less applicable: i) It may be difficult to even know where to obtain the reference, as obtaining it may be as hard as generating consistent information in the first place; ii) Generation may be at a real-time online setting that demands leveraging only existing context to create new content.\nOne common setup for qualitatively measuring the level of hallucination is performed at sentenceor document-level (Dhingra et al., 2019; Scialom et al., 2019). Related tasks such as fake news detection (Zellers et al., 2019) or fact checking (Thorne and Vlachos, 2018) also adopt this strategy. However, sentence- or document-level detection may not always provide high-resolution signals sufficient to pinpoint the hallucinated text, or can only judge whether a generated sentence or a document\nInput: …. She had a large family and lived\nwith her grandparents …. In 1933 she gave birth to her first child …. In July 1926, many\nof her friends attended her funeral …\nLabel1: grandparents → Not Hallucination\nLabel2: funeral → Hallucination\n(C) Data Format in HADES\nInput: Operation Valkyrie ( german : unternehmen\nwalkure ) was a german world war ii emergency\ncontinuity … civil order of the nation. Failure of the army to assume control of civil order might have been caused by the allied bombing of german cities , or because of\nthe millions of jewish forced laborers employed by\ngerman factories . … modified the plan with the\nintention of using it to take control of german forces , to\ndirectly attack the ss , and arrest the ss leaders …\nLabel: to directly attack → Hallucination\nFigure 1: Overview for reference-free token-level hallucination detection task.\nas a whole is a hallucinated artifact. Consequently, these high-level strategies may be insufficient to avoid hallucinations. As an alternative, at decoding time of an NLG system, we suggest that if the locus of hallucination can be identified at the token level, it may be possible to guide beam search or suppress the probability of certain tokens at real-time.\nTo this end, we propose a reference-free, tokenlevel hallucination detection task and introduce an annotated training and benchmark testing dataset that we call HADES (HAllucination DEtection dataSet). The reference-free property of this task yields greater flexibility in a broad range of generation applications. We expect the token-level property of this task to foster the development of models that can detect fine-grained signals of potential hallucination. In conjunction with consulting context to identify self-contradictory statements and access to commonsense and world knowledge, such fine-grained signals, when detected, should further mitigate real-time hallucination.\nOur contributions include: 1) We propose a reference-free, token-level hallucination detection task for free-form text generation. 2) We support this task with a dataset that we call HADES, with ∼11k instances extracted from English Wikipedia using an iterative data collection strategy to address data imbalance issues. We also present comprehensive analyses on the statistical features to shed light on what is commonly recognized as hallucination in crowd-sourced judgments and its salient characteristics in free-form text generation. 3) We create multiple baselines, including feature based models and pretrained models as a first step towards addressing the proposed task.",
    "conclusion": "We have proposed a token-level reference-free hallucination detection task and introduced a benchmark dataset HADES for identifying fine granularity hallucination in free-form text generation. To create this dataset, we perturbed texts to simulate hallucination in NLG system, and performed an interative model-in-the-loop annotation approach to annotate the perturbed text in an imbalanced label scenario. We have further provided comprehensive analyses of HADES and evaluated several baseline models to establish initial benchmarks. We hope that the proposed task and dataset will shed light on high-resolution hallucination detection in freeform text generation and will eventually lead to real-time hallucination prevention."
  },
  {
    "sample_id": 75,
    "title": "A Trio Neural Model for Dynamic Entity Relatedness Ranking",
    "abstract": "Measuring entity relatedness is a fundamental task for many natural language processing and information retrieval applications. Prior work often studies entity relatedness in static settings and an unsupervised manner. However, entities in real-world are often involved in many different relationships, consequently entity-relations are very dynamic over time. In this work, we propose a neural networkbased approach for dynamic entity relatedness, leveraging the collective attention as supervision. Our model is capable of learning rich and different entity representations in a joint framework. Through extensive experiments on large-scale datasets, we demonstrate that our method achieves better results than competitive baselines.",
    "introduction": "Measuring semantic relatedness between entities is an inherent component in many text mining applications. In search and recommendation, the ability to suggest most related entities to the entity-bearing query has become a standard feature of popular Web search engines (Blanco et al., 2013). In natural language processing, entity relatedness is an important factor for various tasks, such as entity linking (Hoffart et al., 2012) or word sense disambiguation (Moro et al., 2014).\nHowever, prior work on semantic relatedness often neglects the time dimension and consider entities and their relationships as static. In practice, many entities are highly ephemeral (Jiang et al., 2016), and users seeking information related to those entities would like to see fresh information. For example, users looking up the entity Taylor Lautner during 2008–2012 might want to be recommended with entities such as The Twilight Saga, due to Lautner’s well-known performance in the film series; however the same query in August 2016 should be served with entities related to his appearances in more recent films such as “Scream Queens”, “Run the Tide”. In addition, much of previous work resorts to deriving semantic relatedness from co-occurrence -based computations or heuristic functions without direct optimization to the final goal. We believe that desirable framework should see entity semantic relatedness as not separate but an integral part of the process, for instance in a supervised manner.\nIn this work, we address the problem of entity relatedness ranking, that is, designing the semantic relatedness models that are optimized for ranking systems such as top-k entity retrieval or recommendation. In this setting, the goal is not to quantify the semantic relatedness between two entities based on their occurrences in the data, but to optimize the partial order of the related entities in the top positions. This problem differs from traditional entity ranking (Kang et al., 2015) in that the entity rankings are driven by user queries and are optimized to their (ad-hoc) information needs, while entity relatedness ranking also aims to uncover the meanings of the the relatedness from the data. In other words, while conventional entity semantic relatedness learns from data (editors or content providers’ perspectives), and entity ranking learns from the user’s perspective, the entity relatedness ranking takes the tradeoff between these views. Such a hybrid approach can benefit applications such as exploratory entity search (Miliaraki et al., 2015), where users have a specific goal in mind, but at the same time are opened to other related entities.\nWe also tackle the issue of dynamic ranking and design the supervised-learning model that takes into account the temporal contexts of entities, and proposes to leverage collective attention from public sources. As an illustration, when one looks into the Wikipedia page of Taylor Lautner, each naviFigure 1: The dynamics of collective attention for related entities of Taylor Lautner in 2016.\ngation to other Wikipedia pages indicates the user interest in the corresponding target entity given her initial interest in Lautner. Collectively, the navigation traffic observed over time is a good proxy to the shift of public attention to the entity (Figure 1).\nIn addition, while previous work mainly focuses on one aspect of the entities such as textual profiles or linking graphs , we propose a trio neural model that learns the low level representations of entities from three different aspects: Content, structures and time aspects. For the time aspect, we propose a convolutional model to embed and attend to local patterns of the past temporal signals in the Euclidean space. Experiments show that our trio model outperforms traditional approaches in ranking correlation and recommendation tasks. Our contributions are summarized as follows.\n• We present the first study of dynamic entity relatedness ranking using collective attention.\n• We introduce an attention-based convolutional neural networks (CNN) to capture the temporal signals of an entity.\n• We propose a joint framework to incorporate multiple views of the entities, both from content provider and from user’s perspectives, for entity relatedness ranking.",
    "conclusion": "In this work, we presented a trio neural model to solve the dynamic entity relatedness ranking problem. The model jointly learns rich representations of entities from textual content, graph and temporal signals. We also propose an effective CNNbased attentional mechanism for learning the temTable 4: Different top-k rankings for entity Kingsman: The Golden Circle. Italic means irrelevance.\nporal representation of an entity. Experiments on ranking correlations and top-k recommendation tasks demonstrate the effectiveness of our approach over existing baselines. For future work, we aim to incorporate more temporal signals, and investigate on different ‘trainable’ attention mechanisms to go beyond the time-based decay, for instance by incorporating latent topics.\nAcknowledgments. This work is funded by the ERC Advanced Grant ALEXANDRIA (grant no. 339233). We thank the reviewers for the suggestions on the content and structure of the paper."
  },
  {
    "sample_id": 76,
    "title": "A Unified Generative Framework for Various NER Subtasks",
    "abstract": "Named Entity Recognition (NER) is the task of identifying spans that represent entities in sentences. Whether the entity spans are nested or discontinuous, the NER task can be categorized into the flat NER, nested NER, and discontinuous NER subtasks. These subtasks have been mainly solved by the token-level sequence labelling or span-level classification. However, these solutions can hardly tackle the three kinds of NER subtasks concurrently. To that end, we propose to formulate the NER subtasks as an entity span sequence generation task, which can be solved by a unified sequence-to-sequence (Seq2Seq) framework. Based on our unified framework, we can leverage the pre-trained Seq2Seq model to solve all three kinds of NER subtasks without the special design of the tagging schema or ways to enumerate spans. We exploit three types of entity representations to linearize entities into a sequence. Our proposed framework is easy-to-implement and achieves state-of-theart (SoTA) or near SoTA performance on eight English NER datasets, including two flat NER datasets, three nested NER datasets, and three discontinuous NER datasets 1.",
    "introduction": "Named entity recognition (NER) has been a fundamental task of Natural Language Processing (NLP), and three kinds of NER subtasks have been recognized in previous work (Sang and Meulder, 2003; Pradhan et al., 2013a; Doddington et al., 2004; Kim et al., 2003; Karimi et al., 2015), including flat NER, nested NER, and discontinuous NER. As shown in Figure 1, the nested NER contains overlapping entities, and the entity in the discontinuous NER may contain several nonadjacent spans.\nthe Lincoln Memorial\n(a) - (c) illustrate flat NER, nested NER, discontinuous NER, and their corresponding mainstream solutions respectively. (d) Our proposed generative solution to solve all NER subtasks in a unified way.\nThe sequence labelling formulation, which will assign a tag to each token in the sentence, has been widely used in the flat NER field (McCallum and Li, 2003; Collobert et al., 2011; Huang et al., 2015; Chiu and Nichols, 2016; Lample et al., 2016; Strakov´ a et al., 2019; Yan et al., 2019; Li et al., 2020a). Inspired by sequence labelling’s success in the flat NER subtask, Metke-Jimenez and Karimi (2016); Muis and Lu (2017) tried to formulate the nested and discontinuous NER into the sequence labelling problem. For the nested and discontinuous NER subtasks, instead of assigning labels to each token directly, Xu et al. (2017); Wang and Lu (2019); Yu et al. (2020); Li et al. (2020b) tried to enumerate all possible spans and conduct the span-level classification. Another way to efficiently represent spans is to use the hypergraph (Lu and Roth, 2015; Katiyar and Cardie, 2018; Wang and Lu, 2018; Muis and Lu, 2016).\nAlthough the sequence labelling formulation has dramatically advanced the NER task, it has to design different tagging schemas to fit various NER subtasks. One tagging schema can hardly fit for all three NER subtasks2 (Ratinov and Roth, 2009; Metke-Jimenez and Karimi, 2016; Strakov´ a et al., 2019; Dai et al., 2020). While the span-based models need to enumerate all possible spans, which is quadratic to the length of the sentence and is almost impossible to enumerate in the discontinuous NER scenario (Yu et al., 2020). Therefore, span-based methods usually will set a maximum span length (Xu et al., 2017; Luan et al., 2019; Wang and Lu, 2018). Although hypergraphs can efficiently represent all spans (Lu and Roth, 2015; Katiyar and Cardie, 2018; Muis and Lu, 2016), it suffers from the spurious structure problem, and structural ambiguity issue during inference and the decoding is quite complicated (Muis and Lu, 2017). Because the problems lie in different formulations, no publication has tested their model or framework in three NER subtasks simultaneously to the best of our knowledge.\nIn this paper, we propose using a novel and simple sequence-to-sequence (Seq2Seq) framework with the pointer mechanism (Vinyals et al., 2015) to generate the entity sequence directly. On the source side, the model inputs the sentence, and on the target side, the model generates the entity pointer index sequence. Since flat, continuous and discontinuous entities can all be represented as entity pointer index sequences, this formulation can tackle all the three kinds of NER subtasks in a unified way. Besides, this formulation can even solve the crossing structure entity3 and multi-type entity4. By converting the NER task into a Seq2Seq generation task, we can smoothly use the Seq2Seq pre-training model BART (Lewis et al., 2020) to enhance our model. To better utilize the pre-trained BART, we propose three kinds of entity representations to linearize entities into entity pointer index sequences.\nOur contribution can be summarized as follows:\nAlthough this is rare, it exists (Dai et al., 2020).\n• We propose a novel and simple generative solution to solve the flat NER, nested NER, and discontinuous NER subtasks in a unified framework, in which NER subtasks are formulated as an entity span sequence generation problem.\n• We incorporate the pre-trained Seq2Seq model BART into our framework and exploit three kinds of entity representations to linearize entities into sequences. The results can shed some light on further exploration of BART into the entity sequence generation.\n• The proposed framework not only avoids the sophisticated design of tagging schema or span enumeration but also achieves SoTA or near SoTA performance on eight popular datasets, including two flat NER datasets, three nested NER datasets, and three discontinuous NER datasets.",
    "conclusion": "In this paper, we formulate NER subtasks as an entity span sequence generation problem, so that we can use a unified Seq2Seq model with the pointer mechanism to tackle flat, nested, and discontinuous NER subtasks. The Seq2Seq formulation enables us to smoothly incorporate the pre-training Seq2Seq model BART to enhance the performance. To better utilize BART, we test three types of entity representation methods to linearize the entity span into sequences. Results show that the entity representation with a shorter length and more similar to continuous BPE sequences achieves better performance. Our proposed method achieves SoTA or near SoTA performance for eight different NER datasets, proving its generality to various NER subtasks."
  },
  {
    "sample_id": 77,
    "title": "A Unified Neural Network Model for Geolocating Twitter Users",
    "abstract": "Locations of social media users are important to many applications such as rapid disaster response, targeted advertisement, and news recommendation. However, many users do not share their exact geographical coordinates due to reasons such as privacy concerns. The lack of explicit location information has motivated a growing body of research in recent years looking at different automatic ways of determining the user’s primary location. In this paper, we propose a unified user geolocation method which relies on a fusion of neural networks. Our joint model incorporates different types of available information including tweet text, user network, and metadata to predict users’ locations. Moreover, we utilize a bidirectional LSTM network augmented with an attention mechanism to identify the most location indicative words in textual content of tweets. The experiments demonstrate that our approach achieves state-of-the-art performance over two Twitter benchmark geolocation datasets. We also conduct an ablation study to evaluate the contribution of each type of information in user geolocation performance.",
    "introduction": "Knowing physical locations involved in social media data helps us to understand what is happening in real life, to bridge the online and offline worlds, and to develop applications for supporting real-life demands. For example, we can monitor public health of residents (Cheng et al., 2010), recommend local events (Yuan et al., 2013) or attractive places (Noulas et al., 2012) to tourists, identify locations of emergency (Ao et al., 2014) or even disasters (Lingad et al., 2013), and summarize regional topics (Rakesh et al., 2013). Even though platforms such as Twitter allow users to geolocate their posts to reveal their locations either manually or with the help of GPS, it is reported that less than 1% of Twitter data has geo-coordinates provided (Jurgens, 2013). Moreover, location information on Twitter is far from being complete and accurate. For instance, self-declared home information in many user profiles is inaccurate or even invalid (Hecht et al., 2011). The lack of explicit location information in the majority of tweets has motivated a growing body of research in recent years looking at different automatic ways of determining the user’s primary location (i.e.,user geolocation) and/or - as a proxy for the former - the location from which tweets have been posted (Ajao et al., 2015).\nGeolocation methods usually train a model on a small set of users whose locations are known (e.g., through GPS-based geotagging), and predict locations of other users using the resulting model. These models broadly fall into three categories: text-based (Eisenstein et al., 2010; Wing and Baldridge, 2011; Roller et al., 2012), networkbased (Jurgens, 2013; Compton et al., 2014; Jurgens et al., 2015), and hybrid methods that combine text, user network, and metadata information (Rahimi et al., 2015b,a; Jayasinghe et al., 2016; Miura et al., 2016) with the aim of achieving stateof-the-art performance.\nIn this paper, we present a neural network-based system that we developed for user geolocation in Twitter. Our model combines different sources of information including tweet text, metadata, and user network. We employ a neural network model to generate a dense vector representation for each field and then use the concatenation of these representations as the feature for classification. Our main contributions can be summarized as follows:\n1. We propose a unified user geolocation method that relies on a fusion of neural networks, incorporating different types of available information: tweet message, users’ social relationships, and metadata fields embedded in tweets and profiles.\n2. For modeling the tweet text (and textual metadata fields), we use bidirectional Long Short-Term Memory (LSTM) networks augmented with a context-aware attention mechanism (Yang et al., 2016), which helps to identify the most location indicative words.\n3. Through the empirical studies on two standard Twitter datasets, we demonstrate that the proposed method outperforms other state-ofthe-art approaches in addressing the problem of user geolocation.\n4. We train an individual model for each information field, and analyze the contribution of each component in the geolocation process.\nThe rest of the paper is organized as follows. We review the related work in Section 2. Utilized data is described in Section 3. Section 4 explains the proposed approach. The experimental results are given in Section 5, and finally, we conclude the paper and outline possible future work in Section 6.",
    "conclusion": "In this paper, we have proposed a unified user geolocation method which relies on a fusion of neural networks. Our joint model effectively utilizes different sources of information including tweet message, users’ social relationships, and metadata fields embedded in tweets and profiles. In particular, we employed a neural network model to generate a dense vector representation for each information field and then used the concatenation of these representations as the feature for classification. For modeling tweet message and textual metadata fields, we utilized a bidirectional LSTM network augmented with an attention mechanism to identify the most location indicative words.\nWe have conducted comprehensive experiments on two standard Twitter geolocation datasets, and demonstrated that our method achieves the best performance in terms of all three evaluation metrics. In an ablation study, we have also trained individual models to investigate the usefulness of each information field in predicting the locations of Twitter users.\nAs a future work, it would be intriguing to utilize customized scrapers for social media websites (Jayasinghe et al., 2016) to further improve the performance of our geolocation model. It is noteworthy that the proposed model could be modified to infer other user demographic attributes such as gender and age.\nTweet publication time include both date and time, however, only time information is exploited in this work to infer users’ geolocations. A future direction is to leverage tweeting behavior over dates for user geolocation. The intuition is that local residents would occasionally post tweets about their home city in a long-term manner, while tourists tend to tweet a lot while visiting the city. Hence, their different tweeting patterns can be easily revealed using date information from their tweet timestamps."
  },
  {
    "sample_id": 78,
    "title": "Backpropagating through Structured Argmax using a SPIGOT",
    "abstract": "We introduce the structured projection of intermediate gradients optimization technique (SPIGOT), a new method for backpropagating through neural networks that include hard-decision structured predictions (e.g., parsing) in intermediate layers. SPIGOT requires no marginal inference, unlike structured attention networks (Kim et al., 2017) and some reinforcement learning-inspired solutions (Yogatama et al., 2017). Like socalled straight-through estimators (Hinton, 2012), SPIGOT defines gradient-like quantities associated with intermediate nondifferentiable operations, allowing backpropagation before and after them; SPIGOT’s proxy aims to ensure that, after a parameter update, the intermediate structure will remain well-formed.\nWe experiment on two structured NLP pipelines: syntactic-then-semantic dependency parsing, and semantic parsing followed by sentiment classification. We show that training with SPIGOT leads to a larger improvement on the downstream task than a modularly-trained pipeline, the straight-through estimator, and structured attention, reaching a new state of the art on semantic dependency parsing.",
    "introduction": "Learning methods for natural language processing are increasingly dominated by end-to-end differentiable functions that can be trained using gradient-based optimization. Yet traditional NLP often assumed modular stages of processing that formed a pipeline; e.g., text was tokenized, then tagged with parts of speech, then parsed into a phrase-structure or dependency tree, then semantically analyzed. Pipelines, which make “hard” (i.e., discrete) decisions at each stage, appear to be incompatible with neural learning, leading many researchers to abandon earlier-stage processing.\nInspired by findings that continue to see benefit from various kinds of linguistic or domain-specific preprocessing (He et al., 2017; Oepen et al., 2017; Ji and Smith, 2017), we argue that pipelines can be treated as layers in neural architectures for NLP tasks. Several solutions are readily available:\n•\nReinforcement learning (most notably the REINFORCE algorithm; Williams, 1992), and structured attention (SA; Kim et al., 2017). These methods replace argmax with a sampling or marginalization operation. We note two potential downsides of these approaches: (i) not all argmax-able operations have corresponding sampling or marginalization methods that are efficient, and (ii) inspection of intermediate outputs, which could benefit error analysis and system improvement, is more straightforward for hard decisions than for posteriors.\n•\nThe straight-through estimator (STE; Hinton, 2012) treats discrete decisions as if they were differentiable and simply passes through gradients. While fast and surprisingly effective, it ignores constraints on the argmax problem, such as the requirement that every word has exactly one syntactic parent. We will find, experimentally, that the quality of intermediate representations degrades substantially under STE.\nThis paper introduces a new method, the structured projection of intermediate gradients optimization technique (SPIGOT; §2), which defines a\nproxy for the gradient of a loss function with respect to the input to argmax. Unlike STE’s gradient proxy, SPIGOT aims to respect the constraints in the argmax problem. SPIGOT can be applied with any intermediate layer that is expressible as a constrained maximization problem, and whose feasible set can be projected onto. We show empirically that SPIGOT works even when the maximization and the projection are done approximately.\nWe offer two concrete architectures that employ structured argmax as an intermediate layer: semantic parsing with syntactic parsing in the middle, and sentiment analysis with semantic parsing in the middle ( §3). These architectures are trained\nusing a joint objective, with one part using data for the intermediate task, and the other using data for the end task. The datasets are not assumed to overlap at all, but the parameters for the intermediate task are affected by both parts of the training data.\nOur experiments ( §4) show that our architecture\nimproves over a state-of-the-art semantic dependency parser, and that SPIGOT offers stronger performance than a pipeline, SA, and STE. On sentiment classification, we show that semantic parsing offers improvement over a BiLSTM, more so with SPIGOT than with alternatives. Our analysis considers how the behavior of the intermediate parser is affected by the end task ( §5). Our\ncode is open-source and available at https:// github.com/Noahs-ARK/SPIGOT.",
    "conclusion": "We presented SPIGOT, a novel approach to backpropagating through neural network architectures that include discrete structured decisions in intermediate layers. SPIGOT devises a proxy for the gradients with respect to argmax’s inputs, employing a projection that aims to respect the constraints in the intermediate task. We empirically evaluate our method with two architectures: a semantic parser with an intermediate syntactic parser, and a sentiment classifier with an intermediate semantic parser. Experiments show that SPIGOT achieves stronger performance than baselines under both settings, and outperforms stateof-the-art systems on semantic dependency parsing. Our implementation is available at https: //github.com/Noahs-ARK/SPIGOT."
  },
  {
    "sample_id": 79,
    "title": "Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLP",
    "abstract": "Graph neural networks have triggered a resurgence of graph-based text classification methods, defining today’s state of the art. We show that a wide multi-layer perceptron (MLP) using a Bag-of-Words (BoW) outperforms the recent graph-based models TextGCN and HeteGCN in an inductive text classification setting and is comparable with HyperGAT. Moreover, we fine-tune a sequence-based BERT and a lightweight DistilBERT model, which both outperform all state-of-the-art models. These results question the importance of synthetic graphs used in modern text classifiers. In terms of efficiency, DistilBERT is still twice as large as our BoW-based wide MLP, while graph-based models like TextGCN require setting up an O(N2) graph, where N is the vocabulary plus corpus size. Finally, since Transformers need to compute O(L2) attention weights with sequence length L, the MLP models show higher training and inference speeds on datasets with long sequences.",
    "introduction": "Text categorization is the task of assigning topical categories to text units such as documents, social media postings, or news articles. Research on text categorization is a very active field as just the sheer amount of new methods in recent surveys shows (Bayer et al., 2021; Li et al., 2020; Zhou et al., 2020; Kowsari et al., 2019; Kadhim, 2019).\nThere are approaches based on a Bag of Words (BoW) that perform text categorization purely on the basis of a multiset of tokens. Among them are Deep Averaging Networks (DAN) (Iyyer et al., 2015), a deep Multi-Layer Perceptron (MLP) model with n layers that relies on averaging the BoW, Simple Word Embedding Models (SWEM) (Shen et al., 2018) that explores different pooling strategies for pretrained word embeddings, and fastText (Bojanowski et al., 2017), which uses a linear layer on top of pretrained word embeddings. These models count the occurrence of all tokens in the input sequence, while disregarding word position and order, and then rely on word embeddings and fully connected feedforward layer(s). We call these BoW-based models.\nAmong the most popular recent methods for text categorization are graph-based models such as TextGCN (Yao et al., 2019) that first induce a synthetic word-document co-occurence graph over the corpus and subsequently apply a graph neural network (GNN) to perform the classification task. Besides TextGCN, there are follow-up works like HeteGCN (Ragesh et al., 2021), TensorGCN (Liu et al., 2020), and HyperGAT (Ding et al., 2020), which we collectively call graph-based models.\nFinally, there is the well-known Transformer (Vaswani et al., 2017) universe with models such as BERT (Devlin et al., 2019) and its sizereduced variants such as DistilBERT (Sanh et al., 2019). Here, the input is a (fixed-length) sequence of tokens, which is then fed into multiple layers of self-attention. Lightweight versions such as DistilBERT and others (Tay et al., 2020; Fournier et al., 2021) use less parameters but operate on the same type of input. Together with recurrent models such as LSTMs, we call these sequence-based models.\nIn this paper, we hypothesize that text categorization can be very well conducted by simple but effective BoW-based models. We investigate this research question in three steps: First, we conduct an in-depth analysis of the literature. We review the key research in the field of text categorization. From this analysis, we derive the different families of methods, the established benchmark datasets, and identify the top performing methods. We decide for which models we report numbers from the literature and which models we run on our own. Overall, we compare 16 different methods from the families of BoW-based models (8 methods), sequence-based models (3 methods), and graphbased models (5 methods). We run our own experiments for 7 of these methods on 5 text categorization datasets, while we report the results from the literature for the remaining methods.\nThe result is surprising: Our own BoW-based MLP, called the WideMLP, with only one wide hidden layer, outperforms many of the recent graphbased models for inductive text categorization (Yao et al., 2019; Liu et al., 2020; Ragesh et al., 2021). Moreover, we did not find any reported scores for BERT-based methods from the sequence-based family. Thus, we fine-tuned our own BERT (Devlin et al., 2019) and DistilBERT (Sanh et al., 2019). These models set a new state of the art. On a metalevel, our study shows that MLPs have largely been ignored as competitor methods in experiments. It seems as if MLPs have been forgotten as baseline in the literature, which instead is focusing mostly on other advanced Deep Learning architectures. Considering strong baselines is, however, an important means to argue about true scientific advancement (Shen et al., 2018; Dacrema et al., 2019). Simple models are also often preferred in industry due to lower operational and maintenance costs.\nBelow, we introduce our methodology and results from the literature study. Subsequently, we introduce the families of models in Section 3. Thereafter, we describe the experimental procedure in Section 4. We present the results of our experiments in Section 5 and discuss our findings in Section 6, before we conclude.",
    "conclusion": "We argue that a wide multi-layer perceptron enhanced with today’s best practices should be considered as a strong baseline for text classification tasks. In fact, the experiments show that our WideMLP is oftentimes on-par or even better than recently proposed models that synthesize a graph structure from the text.\nThe source code is available online:\nhttps://github.com/lgalke/\ntext-clf-baselines"
  },
  {
    "sample_id": 80,
    "title": "Balanced Adversarial Training: Balancing Tradeoffs between Fickleness and Obstinacy in NLP Models",
    "abstract": "Traditional (fickle) adversarial examples involve finding a small perturbation that does not change an input’s true label but confuses the classifier into outputting a different prediction. Conversely, obstinate adversarial examples occur when an adversary finds a small perturbation that preserves the classifier’s prediction but changes the true label of an input. Adversarial training and certified robust training have shown some effectiveness in improving the robustness of machine learnt models to fickle adversarial examples. We show that standard adversarial training methods focused on reducing vulnerability to fickle adversarial examples may make a model more vulnerable to obstinate adversarial examples, with experiments for both natural language inference and paraphrase identification tasks. To counter this phenomenon, we introduce Balanced Adversarial Training, which incorporates contrastive learning to increase robustness against both fickle and obstinate adversarial examples.",
    "introduction": "Interpreted broadly, an adversarial example is an input crafted intentionally to confuse a model. Most research on adversarial examples, however, focuses on a definition of an adversarial example as an input that is constructed by making minimal perturbations to a normal input that change the model’s output, assuming that the small perturbations preserve the original true label (Goodfellow et al., 2015). Such adversarial examples occur when a model is overly influenced by small changes in the input. Attackers can also target the opposite objective—to find inputs with minimal changes that change the ground truth label but for which the model retains its prior prediction (Jacobsen et al., 2019b).\nVarious names have been used in the research literature for these two types of adversarial examples including perturbation or sensitivity-based and invariance-based examples (Jacobsen et al., 2019b,a), and over-sensitive and over-stable examples (Niu and Bansal, 2018; Kumar and Boulanger, 2020). To avoid confusions associated with these names, we refer them as fickle adversarial examples (the model changes its output too easily) and obstinate adversarial examples (the model doesn’t change its output even though the input has changed in a way that it should).\nIn NLP, synonym-based word substitution is a common method for constructing fickle adversarial examples (Alzantot et al., 2018; Jin et al., 2020) since synonym substitutions are assumed to not change the true label for an input. These methods target a model’s weakness of being invariant to certain types of changes which makes its predictions insufficiently responsive to small input changes. Attacks based on antonyms and negation have been proposed to create obstinate adversarial examples for dialogue models (Niu and Bansal, 2018).\nAdversarial training is considered as the most effective defense strategy yet found against adversarial examples (Madry et al., 2018; Goodfellow et al., 2016). It aims to improve robustness by augmenting the original training set with generated adversarial examples in a way that results in decision boundaries that correctly classify inputs that otherwise would have been fickle adversarial examples. Adversarial training has been shown to improve robustness for NLP models (Yoo and Qi, 2021). Recent works have also studied certified robustness training which gives a stronger guarantee that the model is robust to all possible perturbations of a given input (Jia et al., 2019; Ye et al., 2020).\nWhile prior work on NLP robustness focuses on fickle adversarial examples, we consider both fickle and obstinate adversarial examples. We then further examine the impact of methods designed to improve robustness to fickle adversarial examples on a model’s vulnerability to obstinate adversarial examples. Recent work in the vision domain demonstrated that increasing adversarial robustness of im-\nFigure 1: Distance-oracle misalignment (Tramer et al., 2020). While the model is trained to be robust to ϵbounded perturbation, it becomes too invariant to small changes in the example (obstinate example ˜ x) that lie on the other side of the oracle decision boundary.\nage classification models by training with fickle adversarial examples may increase vulnerability to obstinate adversarial examples (Tramer et al., 2020). Even in cases where the model certifiably guarantees that no adversarial examples can be found within an L p-bounded distance, the norm-bounded perturbation does not align with the ground truth decision boundary. This distance-oracle misalignment makes it possible to have obstinate adversarial examples located within the same perturbation distance, as depicted in Figure 1. In text, fickle examples are usually generated with a cosine similarity constraint to encourage the representations of the original and the perturbed sentence to be close in the embedding space. However, this similarity measurement may not preserve the actual semantics (Morris et al., 2020) and the model may learn poor representations during adversarial training.\nContributions. We study fickle and obstinate adversarial robustness in NLP models with a focus on synonym and antonym-based adversarial examples (Figure 2 shows a few examples). We evaluate both kinds of adversarial robustness on natural language inference and paraphrase identification tasks with BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) models. We find that there appears to be a tradeoff between robustness to synonym-based and antonym-based attacks. We show that while certified robust training increases robustness against synonym-based adversarial examples, it increases vulnerability to antonym-based attacks (Section 3). We propose a modification to robust training, Balanced Adversarial Training (BAT), which uses a contrastive learning objective to help mitigate the distance misalignment problem by learning from both fickle and obstinate examples (Section 4). We implement two versions of BAT with different contrastive learning objectives, and show the effectiveness in improving both fickleness and obstinacy robustness (Section 4.2).",
    "conclusion": "We demonstrate the tradeoff between vulnerability to synonym-based (fickle) and antonym-based (obstinate) adversarial examples for NLP models and show that increasing robustness against synonym based attacks also increases vulnerability to antonym-based attacks. To manage this tension, we introduce a new adversarial training method, BAT, which targets the distance-oracle misalignment problem and can help balance the fickleness and obstinacy in adversarial training."
  },
  {
    "sample_id": 81,
    "title": "BenchIE: A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluation",
    "abstract": "Intrinsic evaluations of OIE systems are carried out either manually—with human evaluators judging the correctness of extractions— or automatically, on standardized benchmarks. The latter, while much more cost-effective, is less reliable, primarily because of the incompleteness of the existing OIE benchmarks: the ground truth extractions do not include all acceptable variants of the same fact, leading to unreliable assessment of the models’ performance. Moreover, the existing OIE benchmarks are available for English only. In this work, we introduce BenchIE: a benchmark and evaluation framework for comprehensive evaluation of OIE systems for English, Chinese, and German. In contrast to existing OIE benchmarks, BenchIE is fact-based, i.e., it takes into account informational equivalence of extractions: our gold standard consists of fact synsets, clusters in which we exhaustively list all acceptable surface forms of the same fact. Moreover, having in mind common downstream applications for OIE, we make BenchIE multi-faceted; i.e., we create benchmark variants that focus on different facets of OIE evaluation, e.g., compactness or minimality of extractions. We benchmark several state-of-the-art OIE systems using BenchIE and demonstrate that these systems are significantly less effective than indicated by existing OIE benchmarks. We make BenchIE (data and evaluation code) publicly available.1",
    "introduction": "Open Information Extraction (OIE) is the task of extracting relations and their arguments from natural language text in a schema-free manner (Banko et al., 2007). Consider the sentence \"Sen. Mitchell, who is from Maine, is a lawyer.\"; an OIE system is expected to extract the triples (\"Sen. Mitchell\"; \"is from\"; \"Maine\") and (\"Sen. Mitchell\"; \"is\"; \"a lawyer\") from the sentence. OIE systems are used in many downstream tasks, including knowledge graph (KG) population (Gashteovski et al., 2020), open link prediction (Broscheit et al., 2020), and question answering (Yan et al., 2018). These downstream tasks lend themselves as natural setups for extrinsic OIE evaluation (Mausam, 2016). While valuable in concrete applications, such extrinsic evaluations do not measure the intrinsic correctness of the extracted facts: for that purpose, several benchmarks for intrinsic OIE evaluation have been proposed (Stanovsky and Dagan, 2016; Lechelle et al., 2019; Bhardwaj et al., 2019).\nAutomated benchmark evaluations are more feasible (i.e., faster and cheaper) than manual OIE evaluations (Hohenecker et al., 2020). The current benchmarks, however, use scoring functions that are based on approximate (token-level) matching of system extractions against ground truth facts, which seems to be substantially less reliable than human judgments of extraction correctness (Zhan and Zhao, 2020). This primarily stems from the incompleteness of existing OIE benchmarks: the gold standard extractions do not include all acceptable surface realizations of the same fact. Consider, for example, a sentence from the recent evaluation framework CaRB (Bhardwaj et al., 2019): “Sen. Mitchell is confident he has sufficient votes to block such a measure with procedural actions”; with the gold triple extraction (“Sen. Mitchell”; “is confident he has”; “sufficient votes to . . . procedural actions”). Intuitively, a system extraction with a more concise object— (“Sen. Mitchell”; “is confident he has”; “sufficient votes”)—could also be accepted, as it still captures the same core piece of knowledge, and would arguably be valuable in most downstream tasks.\nTo account for this, existing benchmarks credit system extractions for per-slot lexical overlap with gold extractions. Such scoring is overly lenient and overestimates the systems’ ability to extract correct knowledge facts. Consider, e.g., a system extraction (“Sen. Mitchell”; “is confident he has”; “procedural actions”) for the above-mentioned sentence. From the factual perspective, this extraction is clearly incorrect (Sen. Mitchell has votes, not actions). However, the popular CaRB benchmark with its token-level metrics would judge the extraction as having (1) perfect precision, since all extracted tokens can be found in corresponding slots of a gold extraction and (2) high recall, as all of the gold subject and predicate tokens as well as two gold object tokens (“procedural” and “actions”) are found within corresponding slots of the system extraction (Table 1). Moreover, by providing a single ground truth extraction per fact, existing OIE benchmarks fail to acknowledge that different downstream applications focus on different facets (i.e., aspects) of OIE extractions: e.g., for text summarization, one may prefer minimal extractions (Ponza et al., 2018), whereas knowledge base population benefits from strict correctness of entities in subject and object slots (Lin et al., 2020).\nIn this work, we depart from lenient OIE evaluations based on per-slot token overlaps and propose BenchIE, a novel fact-centric and multi-faceted OIE evaluation framework and benchmark at the core of which is the following question:\nDoes the system extraction express the same fact (i.e., the same unit of knowledge) as any of the ground truth extractions (and vice versa) w.r.t. the specific aspect of the OIE extraction that is of interest for one or more downstream applications?\nContributions. BenchIE advances the state of the art in OIE evaluation in the following: (1) it is the first fact-centered approach to OIE evaluation: to reliably answer the above question, we exhaustively list all correct extractions of the same fact. In contrast to existing benchmarks, BenchIE specifies complete sets of fact-equivalent extractions (dubbed fact synsets), allowing us to avoid error-prone evaluation based on token overlap measures; (2) BenchIE is the first multi-faceted OIE benchmark, allowing to test systems for different aspects of OIE extractions that may be relevant in concrete downstream applications; (3) BenchIE is a multilingual benchmark, covering English, Chinese, and German, and to the best of our knowledge the first with manually annotated (i.e., gold standard) extractions in all languages;2 (4) finally, as a fact-based and multi-faceted benchmark, BenchIE allows us to perform what we believe to be the most comprehensive profiling and comparative evaluation of OIE systems. BenchIE portrays fact extraction abilities of six state-of-the-art OIE models much less favorably and points to their limitations that cannot be detected with existing benchmarks.",
    "conclusion": "We introduced BenchIE: a benchmark for more reliable fact-level evaluation of OIE systems for English, Chinese and German. Unlike existing benchmarks, BenchIE takes into account fact-level equivalence of extractions: it consists of fact synsets that contain all acceptable surface forms of the same fact. Further, EN BenchIE is multi-faceted – it allows to evaluate OIE extractions w.r.t. several aspects relevant in common downstream tasks. Our experiments show that current benchmarks, with incomplete gold standard and approximate tokenlevel matching, drastically overestimate fact extraction abilities of OIE systems. Currently, the limits of BenchIE are its relatively small size (300 sentences v.s. CaRB’s 1,200) and its time-consuming annotation process. A promising research direction is the investigation of trade-off between the manual effort and completeness of different OIE annotation strategies. In this scenario, BenchIE is an ideal point of reference: it can precisely quantify the completeness of some larger (non-exhaustive) OIE dataset created with limited or no manual effort."
  },
  {
    "sample_id": 82,
    "title": "BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks",
    "abstract": "Transformer-based language models (TLMs), such as BERT, ALBERT and GPT-3, have shown strong performance in a wide range of NLP tasks and currently dominate the field of NLP. However, many researchers wonder whether these models can maintain their dominance forever. Of course, we do not have answers now, but, as an attempt to find better neural architectures and training schemes, we pretrain a simple CNN using a GAN-style learning scheme and Wikipedia data, and then integrate it with standard TLMs. We show that on the GLUE tasks, the combination of our pretrained CNN with ALBERT outperforms the original ALBERT and achieves a similar performance to that of SOTA. Furthermore, on open-domain QA (Quasar-T and SearchQA), the combination of the CNN with ALBERT or RoBERTa achieved stronger performance than SOTA and the original TLMs. We hope that this work provides a hint for developing a novel strong network architecture along with its training scheme. Our source code and models are available at https://github.com/nict-wisdom/bertac.",
    "introduction": "Transformer-based language models (TLMs) such as BERT (Devlin et al., 2019), ALBERT (Lan et al., 2020), and GPT-3 (Brown et al., 2020) have shown that large-scale self-supervised pretraining leads to strong performance on various NLP tasks. Many researchers have used TLMs for various downstream tasks, possibly as subcomponents of their methods, and/or they have focused on scaling up TLMs or improving their pretraining schemes. As a result, other architectures like Recurrent Neural Networks (RNN) (Hochreiter and Schmidhuber, 1997; Cho et al., 2014) and Convolutional Neural Networks (CNN) (LeCun et al., 1999) are fading away. In this work, we propose a method\n!! ! \"\" ! # \"\"\n!\"#$\"#!\"#$%&’()’*+\n%&$\"#!\"!\"#$\"#%\" !#$!\"#$\"#%\" \"\n+34(%%5678!\"\"\n!8;2’(1%;<-18-’;1%13=\n9#$>’6-$’;1’;?’$\n!! !$\" ! #$\"\nFigure 1: Overall architecture of BERTAC under the setting of classification for a two-sentence input (sentence x and sentence y).\nfor improving TLMs by integrating a simple conventional CNN to them. We pretrained this CNN on Wikipedia using a Generative Adversarial Network (GAN) style training scheme (Goodfellow et al., 2014), and then combined it with TLMs. Oh et al. (2019) similarly used GAN-style training to improve a QA model using a CNN, but their training scheme was applicable only to QAspecific datasets. On the other hand, similarly to TLM, our proposed method for training the CNN is independent of specific tasks. We show that the combination of this CNN with TLMs can achieve higher performance than that of the original TLMs on publicly available datasets for several distinct tasks. We hope that this gives an insight into how to develop novel strong network architectures and training schemes.\nWe call our combination of a TLM and a CNN BERTAC (BERT-style TLM with an Adversarially pretrained Convolutional neural network). Its architecture is illustrated in Fig. 1. We do not impose any particular restriction on the TLM in BERTAC, so any TLM, ALBERT (Lan et al.,\n!$%&’()’*+%!,-.,(/0(1\" ![EM] ,2-3+’,4’)562-!’,)-\nFigure 2: GAN-style pretraining of CNNs. The discriminator D takes either a real representation generated by R or a fake representation generated by F as its input and then it predicts whether the input is a real\nand the original sentence (s 1)\n2020) or RoBERTa (Liu et al., 2019) for example, can be used as a subcomponent of BERTAC.\nWe used the CNN to compute representations of a slightly modified version of the input given to a TLM. To integrate these representations with those of the TLM, we stacked on top of the TLM several layers of Transformers for Integrating External Representation (TIERs), which are our modified version of normal transformers (Vaswani et al., 2017). A TIER has the same architecture as that of a normal transformer encoder except for its attention: we replace the transformer’s self-attention with an attention based on the representation provided by the CNN. We expect that, by keeping the basic architecture of transformer encoders, the CNN’s representations can be integrated more effectively with the TLM’s original representations.\nWe pretrained the CNN using a GAN-style training scheme in order to generate representations of sentences rather freely without the constraint of token embedding prediction in the masked language modeling used for TLMs, as we explain later. For the training, we used masked sentences autogenerated from Wikipedia. As in the masked language modeling, neither human intervention nor downstream task-specific hacking is required. As illustrated in Fig. 2, the GANstyle training requires three networks, namely, a discriminator D and two CNN-based generators R and F . Once the training is done, we use the generator F as CNN in BERTAC. The training data consists of pairs of an entity mention and a sentence in which the entity mention is masked with a special token [EM]. For example, the entitymasked sentence m 1 in Table 1 is obtained by masking the entity mention e 1, “Suvarnabhumi Airport,” in the original text s 1. The network F generates a vector representation of the masked sentence (m 1), while R produces a representation of the masked entity (e 1). The discriminator D takes representations generated by either R or F as the input, and it predicts which generator actually gave the representation.\nIn the original GAN, a generator learns to generate an artificial image from random noise so that the resulting artificial image is indistinguishable from given real images. By analogy, we used an entity-masked sentence as “random noise” and a masked entity as a “real image.” In our GAN-style training, we regard the vector representation of a masked entity given by generator R as a real representation of the entity (or the representation of the “real image” in the above analogy). On the other hand, we regard the representation of the masked sentence, generated by F , as a fake representation of the entity (or the representation of the “artificial image” generated from the “random noise” in the above analogy). This representation is deemed fake because the entity is masked in the masked sentence, and F does not know what the entity is exactly. During the training, F should try to deceive the discriminator D by mimicking the real representation and generating a fake representation that is indistinguishable from the real representation of the entity generated by R. On the other hand, R and D, as a team, try to avoid being mimicked by F and also to make the mimic problem harder for F . If everything goes well, once the training is over, F should be able to generate a fake representation of the entity that is similar to its real representation.\nAn interesting point is that F ’s output can be interpreted in two ways: it is a representation of a masked sentence because it is computed from the sentence, and at the same time it is a representation of the masked entity because it is indistinguishable from R’s representation of the entity. This duality suggests that F ’s output can be seen as a representation of the entire sentence.\nWe exploit F as a CNN in BERTAC as follows: first, we use F to compute a representation of a masked version of the sentence originally given as input to a TLM. The entity mention to be masked is chosen by simple rules and, if the input consists of multiple sentences, we generate a representation of each (masked) input sentence and concatenate these together into a single one. Then, this representation is integrated to the output of the TLM through multiple TIER layers.\nOur GAN-style pretraining is conceptually similar to TLM pretraining with masked language modeling (predicting what a masked word in a sentence should be). However, it was designed to pretrain a model that is able to rather freely generate entity representations without strongly sticking to the prediction of token embeddings. Our hypothesis is that such freely generated representations may be useful for improving the performance of downstream tasks. Moreover, we assumed that using multiple text representations computed from different perspectives (i.e., predicting token embeddings and freely generating entity representations) would help to improve the performance of downstream tasks.\nIn our experiments, we show that for the GLUE tasks (Wang et al., 2018), BERTAC’s average performance on the development set was 0.7% higher than that of ALBERT, which was used as a subcomponent of BERTAC, leading to a performance on the test set comparable to that of SOTA (90.3% vs 90.8% (SOTA)). It also outperformed the SOTA method of open-domain QA (Chen et al., 2017) on Quasar-T (Dhingra et al., 2017) and SearchQA (Dunn et al., 2017) using either ALBERT or RoBERTa. We also compared our method with alternative models using a CNN pretrained in a self-supervised (non GAN-style) manner to directly predict embeddings of the entity mentions. Consequently, we confirmed that our method worked better: only the CNN trained by our GAN-style pretraining gave significant performance improvement over base TLMs.\nNote that the computational overhead of BERTAC is reasonably small. It took 20 hours with 16 GPUs to pretrain a single CNN model and 180 hours for the nine models tested with different parameter settings in this work (cf., 480 hours with 96 GPUs for pretraining DeBERTa (He et al., 2021), for example). Moreover, once pretrained, the CNN models can be re-used for various downstream tasks and combined with various TLMs, including potentially future ones. As for the parameter number, BERTAC had just a 14% increase in parameters when ALBERT-xxlarge was used as its base TLM (268 M parameters for BERTAC vs. 235 M for ALBERT-xxlarge). We confirmed from these results that BERTAC could improve pretrained TLMs with reasonably small computational overhead.\nThe code and models of BERTAC are available at https://github.com/nict-wisdom/bertac.",
    "conclusion": "We proposed BERTAC (BERT-style TLM with an Adversarially pretrained Convolutional neural network), a combination of a TLM and a CNN, where the CNN was pretrained using a novel GAN-style training scheme and masked sentences obtained automatically from Wikipedia. Using this CNN, we improved the performance of standard TLMs. We confirmed that BERTAC could achieve comparable performance with the SOTA and outperformed the base TLM used as a subcomponent of BERTAC in the GLUE task. We also show that BERTAC outperformed the SOTA method of open-domain QA on Quasar-T and SearchQA."
  },
  {
    "sample_id": 83,
    "title": "BERT-Based Neural Collaborative Filtering and Fixed-Length Contiguous Tokens Explanation",
    "abstract": "We propose a novel, accurate, and explainable recommender model (BENEFICT) that addresses two drawbacks that most reviewbased recommender systems face. First is their utilization of traditional word embeddings that could influence prediction performance due to their inability to model the word semantics’ dynamic characteristic. Second is their black-box nature that makes the explanations behind every prediction obscure. Our model uniquely integrates three key elements: BERT, multilayer perceptron, and maximum subarray problem to derive contextualized review features, model user-item interactions, and generate explanations, respectively. Our experiments show that BENEFICT consistently outperforms other state-of-the-art models by an average improvement gain of nearly 7%. Based on the human judges’ assessment, the BENEFICT-produced explanations can capture the essence of the customer’s preference and help future customers make purchasing decisions. To the best of our knowledge, our model is one of the first recommender models to utilize BERT for neural collaborative filtering.",
    "introduction": "In recommender systems research, collaborative filtering (CF) is the dominant state-of-the-art recommendation model, which primarily focuses on learning accurate representations of users (user preferences) and items (item characteristics) (Chen et al., 2018; Tay et al., 2018). The earliest recommender models learned these representations based on user-given numeric ratings that each item received (Mnih and Salakhutdinov, 2008; Koren et al., 2009). However, ratings, which are values on a single discrete scale, oversimplify user preferences and item characteristics (Musto et al., 2017). The large amount of users and items in a typical online platform consequently results in a highly sparse rating matrix, making it hard to learn accurate representations (Zheng et al., 2017).\nTo alleviate these issues, review texts have instead been utilized to model such representations for subsequent recommendation and rating prediction, and this approach has attracted growing attention in research (Catherine and Cohen, 2017; Zheng et al., 2017). The main advantage of reviews as the source of features is that they can cover user opinions’ multi-faceted substance. Because users can explain their reasons underlying their given ratings, reviews contain a large amount of latent information that is both rich and valuable, and that cannot be otherwise obtained from ratings alone (Chen et al., 2018; Wang et al., 2019). Recently, models that incorporate user reviews have yielded state-of-the-art performances (Zheng et al., 2017; Chen et al., 2018). These approaches learn user and item representations by using traditional word embeddings (e.g., word2vec, GloVe) to map each word in the review into its corresponding vector. The review is transformed into an embedded matrix before being fed to a convolutional neural network (CNN) (Chen et al., 2018). CNNs have been shown to effectively model reviews and have illustrated outstanding results in numerous natural language processing tasks (Wang et al., 2018a).\nNevertheless, there are drawbacks that most review-based recommender models experience. First is the utilization of traditional or mainstream word embeddings to learn review features. Their static nature is a hindrance, as each word sense is associated with the same embedding regardless of the context. In other words, such embeddings cannot identify the dynamic nature of each word’s semantics. For review-based recommenders, this could be an issue in modeling users and items, which could, in turn, affect recommendation performance (Pilehvar and Camacho-Collados, 2019). Also, once a CNN is fed with the matrix of word embeddings, the word frequency information of contextual features, said to be crucial for modeling reviews, will be lost (Wang et al., 2018a).\nAnother drawback is the inherent black-box nature of deep learning-based models that makes the explanations behind every prediction obscure (Ribeiro et al., 2016; Wang et al., 2018b). The complex architecture of hidden layers has opaqued the models’ internal decision-making processes (Peake and Wang, 2018). Providing explanations could help persuade users to make decisions and develop trust in a recommender system (Zhang et al., 2014; Ribeiro et al., 2016; Costa et al., 2018; Peake and Wang, 2018). However, this leads us to a dilemma, i.e., a trade-off between accuracy and explainability. Usually, the most accurate models are inherently complicated, non-transparent, and unexplainable (Zhang and Chen, 2018). The same is also true for explainable and straightforward methods that sacrifice accuracy. Formulating models that are both explainable and accurate is a challenging yet critical research agenda for the machine learning community to ensure that we derive benefits from machine learning fairly and responsibly (Peake and Wang, 2018).\nIn this paper, we propose a unique model: BERT-Based Neural Collaborative Filtering and Fixed-Length Contiguous Tokens Explanation (BENEFICT). Our model learns user and item representations simultaneously using two parallel networks. To address the first drawback, we incorporate BERT as a key component in each parallel network. BERT affords us to extract more meaningful, contextualized features adaptable to arbitrary contexts; such features cannot be derived from mainstream word embeddings (Pilehvar and CamachoCollados, 2019; Zakbik et al., 2019). BERT can also retain the word frequency information that makes CNN an unnecessary component of our model. Once user and item representations are learned, they are concatenated together in a shared hidden space before being finally fed to an optimal stack of multilayer perceptron (MLP) layers that serve as BENEFICT’s interaction function.\nTo address the second drawback, we introduce a novel component in our model that integrates BERT’s self-attention and an implementation of the fixed-length maximum subarray problem (MSP), which is considered to be a classic computer science problem. BERT applies self-attention in each encoder layer that consequently produces selfattention weights for each token. These are passed to the successive encoder layers through feedforward networks. We argue that these self-attention weights can be the basis for explaining rating predictions. Based on this premise, MSP then selects a segment or subarray of consecutive tokens that has the maximum possible sum of self-attention weights.",
    "conclusion": "We have successfully implemented a novel recommender model that uniquely integrates BERT, MLP, and MSP. BENEFICT’s predictive capability is validated by experiments performed on Amazon and Yelp datasets, consistently outperforming other state-of-the-art models. Moreover, its explanation generation capability is verified by human judges. We argue that our work offers an avenue to help bridge the research gap between accuracy and explainability. In the future, we will consider incorporating other neural components, such as attention mechanisms, in improving the user-item modeling process. We also intend to enhance the expressiveness and the overall quality of the generated explanations."
  },
  {
    "sample_id": 84,
    "title": "BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance",
    "abstract": "Pretraining deep language models has led to large performance gains in NLP. Despite this success, Schick and Sch¨ utze (2020) recently showed that these models struggle to understand rare words. For static word embeddings, this problem has been addressed by separately learning representations for rare words. In this work, we transfer this idea to pretrained language models: We introduce BERTRAM, a powerful architecture based on BERT that is capable of inferring high-quality embeddings for rare words that are suitable as input representations for deep language models. This is achieved by enabling the surface form and contexts of a word to interact with each other in a deep architecture. Integrating BERTRAM into BERT leads to large performance increases due to improved representations of rare and medium frequency words on both a rare word probing task and three downstream tasks.1",
    "introduction": "As word embedding algorithms (e.g. Mikolov et al., 2013) are known to struggle with rare words, several techniques for improving their representations have been proposed. These approaches exploit either the contexts in which rare words occur (Lazaridou et al., 2017; Herbelot and Baroni, 2017; Khodak et al., 2018; Liu et al., 2019a), their surfaceform (Luong et al., 2013; Bojanowski et al., 2017; Pinter et al., 2017), or both (Schick and Sch¨ utze, 2019a,b; Hautte et al., 2019). However, all of this prior work is designed for and evaluated on uncontextualized word embeddings.\nContextualized representations obtained from pretrained deep language models (e.g. Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019b) already handle rare words implicitly using methods such as byte-pair encoding (Sennrich et al., 2016), WordPiece embeddings (Wu et al., 2016) and character-level CNNs (Baevski et al., 2019). Nevertheless, Schick and Sch¨ utze (2020) recently showed that BERT’s (Devlin et al., 2019) performance on a rare word probing task can be significantly improved by explicitly learning representations of rare words using Attentive Mimicking (AM) (Schick and Sch¨ utze, 2019a). However, AM is limited in two important respects:\n• For processing contexts, it uses a simple bagof-words model, making poor use of the available information.\n• It combines form and context in a shallow fashion, preventing both input signals from interacting in a complex manner.\nThese limitations apply not only to AM, but to all previous work on obtaining representations for rare words by leveraging form and context. While using bag-of-words models is a reasonable choice for static embeddings, which are often themselves bagof-words (e.g. Mikolov et al., 2013; Bojanowski et al., 2017), it stands to reason that they are not the best choice to generate input representations for position-aware, deep language models.\nTo overcome these limitations, we introduce BERTRAM (BERT for Attentive Mimicking), a novel architecture for learning rare word representations that combines a pretrained BERT model with AM. As shown in Figure 1, the learned rare word representations can then be used as an improved input representation for another BERT model. By giving BERTRAM access to both surface form and contexts starting at the lowest layer, a deep integration of both input signals becomes possible.\nAssessing the effectiveness of methods like BERTRAM in a contextualized setting is challenging: While most previous work on rare words was evaluated on datasets explicitly focusing on rare words (e.g Luong et al., 2013; Herbelot and Baroni, 2017; Khodak et al., 2018; Liu et al., 2019a), these datasets are tailored to uncontextualized embeddings and thus not suitable for evaluating our model. Furthermore, rare words are not well represented in commonly used downstream task datasets. We therefore introduce rarification, a procedure to automatically convert evaluation datasets into ones for which rare words are guaranteed to be important. This is achieved by replacing task-relevant frequent words with rare synonyms obtained using semantic resources such as WordNet (Miller, 1995). We rarify three common text (or text pair) classification datasets: MNLI (Williams et al., 2018), AG’s News (Zhang et al., 2015) and DBPedia (Lehmann et al., 2015). BERTRAM outperforms previous work on four English datasets by a large margin: on the three rarified datasets and on WNLaMPro (Schick and Sch¨ utze, 2020).\nIn summary, our contributions are as follows:\n• We introduce BERTRAM, a model that integrates BERT into Attentive Mimicking, enabling a deep integration of surface-form and contexts and much better representations for rare words.\n• We devise rarification, a method that transforms evaluation datasets into ones for which rare words are guaranteed to be important.\n• We show that adding BERTRAM to BERT achieves a new state-of-the-art on WNLaMPro (Schick and Sch¨ utze, 2020) and beats all baselines on rarified AG’s News, MNLI and DBPedia, resulting in an absolute improvement of up to 25% over BERT.",
    "conclusion": "We have introduced BERTRAM, a novel architecture for inducing high-quality representations for rare words in BERT’s and RoBERTa’s embedding spaces. This is achieved by employing a powerful pretrained language model and deeply integrating surface-form and context information. By replacing important words with rare synonyms, we created downstream task datasets that are more challenging and support the evaluation of NLP models on the task of understanding rare words, a capability that human speakers have. On all of these datasets, BERTRAM improves over standard BERT and RoBERTa, demonstrating the usefulness of our method.\nOur analysis showed that BERTRAM is beneficial not only for rare words (our main target in this paper), but also for frequent words. In future work, we want to investigate BERTRAM’s potential benefits for such frequent words. Furthermore, it would be interesting to explore more complex ways of incorporating surface-form information – e.g., by using a character-level CNN similar to the one of Kim et al. (2016) – to balance out the potency of BERTRAM’s form and context parts."
  },
  {
    "sample_id": 85,
    "title": "Better Few-Shot Relation Extraction with Label Prompt Dropout",
    "abstract": "Few-shot relation extraction aims to learn to identify the relation between two entities based on very limited training examples. Recent efforts found that textual labels (i.e., relation names and relation descriptions) could be extremely useful for learning class representations, which will benefit the few-shot learning task. However, what is the best way to leverage such label information in the learning process is an important research question. Existing works largely assume such textual labels are always present during both learning and prediction. In this work, we argue that such approaches may not always lead to optimal results. Instead, we present a novel approach called label prompt dropout, which randomly removes label descriptions in the learning process. Our experiments show that our approach is able to lead to improved class representations, yielding significantly better results on the few-shot relation extraction task.1",
    "introduction": "Enabling machines to comprehend sentences and extract relations between entities has been a crucial task in Natural Language Processing (NLP). Conventional methods frame this task as a multiclass classification problem, trying to solve it through large-scale supervised training with LSTM (Hochreiter and Schmidhuber, 1997) or BERT (Devlin et al., 2019) as the backbone (Zhou et al., 2016; Zhang et al., 2017; Yamada et al., 2020). Such an approach has shown great effectiveness. However, one problem left unsolved is to identify novel relations with only a handful of training examples. Therefore, recent studies (Han et al., 2018; Gao et al., 2019b) introduce the task of few-shot relation extraction (FSRE) to study this data scarcity problem.\nAligned with the success of few shot learning in Computer Vision (Sung et al., 2018; Sator-\nFigure 1: An example of 2-way-1-shot learning using label prompt dropout (LPD). Top: Instead of assuming textual labels are always present for support instances, LPD randomly drops out such textual labels. Here the textual label “country of origin” for the second instance is droppoed out. Bottom: LPD directly concatenates the textual label and the context sentence. The textual label serves as a prompt to guide BERT to derive a better class prototype. Note that for simplicity we use the relation names here, while in our implementation we use relation descriptions, which are lengthier and more complex.\nras and Estrach, 2018), most attempts in FSRE adopt a meta learning framework (Santoro et al., 2016; Vinyals et al., 2016) that randomly samples episodes with different label sets from the training data to mimic the few shot scenario in the testing phase. As a meta learning approach, prototypical network (Snell et al., 2017) aims to learn a class-agnostic metric space. A query instance is classified as the class that has the nearest prototype during inference.\nWhile the BERT-based prototypical networks (Baldini Soares et al., 2019; Peng et al., 2020a) have shown impressive performance on FSRE, the class prototypes are only constructed through the average representation of support instances of each class, neglecting the textual labels that may provide additional useful information. Therefore, recent efforts try to modify the prototypical network such that it can use the label information as well. Yang et al. (2020) insert both entity type information and relation descriptions to the model. Dong et al. (2021) use a relation encoder to generate relation representation besides the sentence encoder. Han et al. (2021a) propose a hybrid prototypical network that can generate hybrid prototypes from context sentences and relation descriptions. Nonetheless, these methods largely assume that every support instance is provided with a corresponding textual label in the support set during both learning and prediction. We argue that injecting textual labels to all support instances may render the training task unchallenging, because the model can largely rely on the textual labels during training, and thus results in poor performance during testing when faced with unseen relations and textual labels. Ideally, textual labels should be treated as additional source of information, such that the model can work with or without the textual labels, as shown in the top part in Figure 1.\nIn this work, we propose a novel approach called Label Prompt Dropout (LPD). We directly concatenate the textual label and the context sentence, and feed them together to the Transformer encoder (Vaswani et al., 2017). The textual label serves as a label prompt2 to guide and regularize the Transformer encoder to output a label-aware relation representation through self-attention. During training, we randomly drop out the prompt tokens to create a more challenging scenario, such that the model has to learn to work with and without the relation descriptions. Experiments show our approach achieves significant improvement on two standard FSRE datasets. Extensive ablation studies are conducted to demonstrate the effectiveness of our approach. Furthermore, we highlight a potential issue with the evaluation setup of previous research efforts, in which the pre-training data contains relation types that actually overlap with those in the test set. We argue that this may not be a desirable setup for few-shot learning, and show that the performance gain of existing efforts may be partly due to this “knowledge leakage” issue. We\nWe use relation description to construct a natural language\nThis goal is similar to that of the conventional prompt-based\npropose to filter out all the overlapping relation types in the pre-training data and conduct more rigorous few-shot evaluation. In summary, we make the following contributions:\n• We present LPD, a novel label prompt dropout approach that makes better use of the textual labels in FSRE. This simple design has significantly outperformed previous attempts that fuse the textual label and the context sentence using complex network structures.\n• We identify the limitation of the previous experimental setup in the literature and propose a stricter setup for evaluation in FSRE. For both setups, we show strong improvements over the previous state of the art.",
    "conclusion": "This paper proposes a novel label prompt dropout approach that directly concatenates the label prompt with the context sentence for few-shot relation extraction. The label prompt is randomly dropped out during pre-training and training to create a more challenging learning setup, leading to better use of the relation descriptions. In the experiments, we discover a “knowledge leakage” issue in the previous works’ experimental setup. We propose a stricter setup for more rigorous evaluations in FSRE by filtering out all overlapping relations. Our method has demonstrated significant improvements on both evaluation settings. Ablation studies show that LPD shares some similar and interesting properties to the neural dropout operation and prompt based methods. One possible direction of future work is to generalize this idea to other text classification tasks such as intent classification (Larson et al., 2019)."
  },
  {
    "sample_id": 86,
    "title": "Better than Average: Paired Evaluation of NLP systems",
    "abstract": "Evaluation in NLP is usually done by comparing the scores of competing systems independently averaged over a common set of test instances. In this work, we question the use of averages for aggregating evaluation scores into a final number used to decide which system is best, since the average, as well as alternatives such as the median, ignores the pairing arising from the fact that systems are evaluated on the same test instances. We illustrate the importance of taking the instancelevel pairing of evaluation scores into account and demonstrate, both theoretically and empirically, the advantages of aggregation methods based on pairwise comparisons, such as the Bradley–Terry (BT) model, a mechanism based on the estimated probability that a given system scores better than another on the test set. By re-evaluating 296 real NLP evaluation setups across four tasks and 18 evaluation metrics, we show that the choice of aggregation mechanism matters and yields different conclusions as to which systems are state of the art in about 30% of the setups. To facilitate the adoption of pairwise evaluation, we release a practical tool for performing the full analysis of evaluation scores with the mean, median, BT, and two variants of BT (Elo and TrueSkill), alongside functionality for appropriate statistical testing.",
    "introduction": "Research is driven by evaluation results, with attention and resources being focused on methods identified as state of the art (SotA). The proper design of evaluation methodology is thus crucial to ensure progress in the field. In NLP, evaluation usually consists in comparing the averaged scores of competing systems over a common set of test instances. Indeed, averaging scores independently for each system and declaring the one with the highest average to be best is particularly\nE v a l\nu a t i\no n\ns c o\nr e s\no f\ns y s\nt e m\nSystem B\nFigure 1: Motivating example (synthetic data). Evaluation scores of systems A, B, and C for five test instances. All systems have the same mean. C is better than A on all instances but one, so BT declares C > A Also, B is better than A on all instances but one, so BT declares B > A, whereas the median of A is greater, and the means are the same. Overall, mean and median fail to capture the complex instance-level pairing.\nsimple, well understood, and mirrors the expected risk minimization paradigm used to train systems.\nHere, we critically assess the specific choice of the average to aggregate evaluation scores. In particular, we emphasize that there is a natural instance-level pairing between the evaluation scores of systems, which aggregation mechanisms such as the mean or median fail to take into account: as they produce a score for each system independently, systems that have the same set of scores (but potentially in different order) cannot be distinguished.\nConsider the three systems A, B, and C compared on five test instances in Fig. 1. Despite a complex pairing structure, they all have the same mean score across test instances. Moreover, even though B is better than A on all test instances but one, the median of A is greater than the median of B.\nIn this work, we discuss an alternative aggregation mechanism: the Bradley–Terry (BT) model (Bradley and Terry, 1952). BT compares systems for each test instance and estimates the latent strength of systems based on how frequently one system scores higher than another. Such paired mechanisms have already been successfully used to aggregate human judgments (Novikova et al., 2018; Sedoc and Ungar, 2020); for example, WMT evaluation protocols regularly employ TrueSkill (Herbrich et al., 2007), a Bayesian variant of BT (Sakaguchi et al., 2014).\nContributions. We contribute the first comprehensive analysis of the BT model (especially vis-à-vis mean and median) as an aggregation mechanism for comparing system scores in NLP.\n(i) We illustrate the importance of accounting for instance-level pairing and discuss the conditions under which the mean, median, and BT disagree about the ordering of systems. In Sec. 3, we draw parallels with the field of statistical testing, where paired statistical tests are recommended when comparing paired variables. Thus, we argue that paired aggregation mechanisms such as BT are more robust alternatives to the mean and median. We support this argument with simulations in Sec. 4.\n(ii) We show that the differences between mean, median, and BT matter in practice. By re-evaluating 296 real NLP evaluation setups across four tasks and 18 evaluation metrics, different aggregation mechanisms yield different conclusions as to which systems are SotA in about 30% of the setups (Sec. 5). These results hold when replacing BT by the Elo (Elo, 1978) and TrueSkill variants.\n(iii) We discuss further advantages and potential limitations of BT, alongside possible resolutions, in Sec. 7.\n(iv) We recommend replacing the mean by BT in future evaluations of NLP systems. To ease the adoption of more robust aggregation mechanisms, we release Pairformance,1 a practical tool for performing full analyses of evaluation scores with mean, median, BT, and two variants of BT (Elo and TrueSkill). The tool reports paired evaluation results alongside appropriate statistical testing for all five aggregation mechanisms and various visualization functionalities to elucidate the pairing structure between system scores.\nCode and data for replicating our analyses and experiments is available online.2",
    "conclusion": "We performed a critical assessment of the standard NLP evaluation methodology based on averaged scores, which ignores the natural instance-level pairing of evaluation scores when comparing systems. We showed the importance of the pairing and demonstrated the advantages of paired mechanisms such as Bradley–Terry (BT) over more standard aggregation schemes such as the mean or median. The choice of aggregation mechanism matters in real evaluation setups, and we therefore recommend BT as a robust aggregation mechanism. To facilitate adoption, we release Pairformance, a new tool to perform full analyses of system scores using BT and two of its variants, Elo and TrueSkill."
  },
  {
    "sample_id": 87,
    "title": "Better Together: Jointly Using Masked Latent Semantic Modeling and Masked Language Modeling for Sample Efficient Pre-training",
    "abstract": "In this paper, we demonstrate the benefits of jointly using Masked Latent Semantic Modeling (MLSM) and traditional Masked Language Modeling (MLM) as the pre-training objective of masked language models. The core idea behind MLSM is to modify the pre-training objective in a way which ensures that the language models predict a (latent) semantic distribution for the masked tokens – instead of outputting their exact identity as in MLM. Language models pre-trained with MLSM behave more favorable in terms of fine-tuneability towards downstream tasks, however, their performance lags behind MLM pre-trained language models in evaluations that investigate the linguistic capabilities. In an attempt to combine the strengths of the two different pre-training paradigms, we propose their joint use in a multitask learning setting. Our evaluations that we performed using the BabyLM evaluation framework (Warstadt et al., 2023) demonstrate the synergistic effects of the joint use of the two different kinds of pre-training objectives.",
    "introduction": "Albeit being effective and easy to implement in practice, the highly stochastic batch-based masked language modeling (MLM) objective frequently used for pre-training language models, such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), is not sample efficient and works in a rather unnatural way from a human cognitive perspective. This is caused by the fact that traditional MLM expects the neural models to recover the exact identity of the masked (sub)words within an input sequence. In an attempt to overcome the unnaturalness of MLM, (Berend, 2023) has recently proposed masked latent semantic modeling (MLSM), a sample efficient alternative to traditional masked language modeling.\nMLSM differs from MLM in that its objective is to recover the semantic distribution of masked\n(a) MLM objective\n(b) MLSM objective\nFigure 1: Comparisons of the probability distributions used in MLM (a) and MLSM (b) pre-training.\n(sub)tokens over an unsupervised inventory of latent semantic properties — as opposed to that of a one-hot distribution over the entire vocabulary of the language model. This kind of pre-training is arguably more plausible from a human cognitive perspective, i.e., traditional MLM acts as if there was a single proper substitute for a special [MASK] token (the one that got masked), whereas from a human perspective multiple viable tokens – tokens that share some common semantic properties – can substitute a masked token.\nFor instance, in the sentence ’She picked a delicious [MASK].’, human subjects would agree that any word referring to an edible concept is a viable substitute for the last word of the sentence. In Figure 1, we illustrate the different kinds of outputs that the MLM (Figure 1a) and the MLSM (Figure 1b) objectives could produce for some masked token such as the one in the above example.\nEven though (Berend, 2023) has demonstrated the improved sample efficiency of MLSM, language models pre-trained with it perform poorly in evaluations that test the linguistic capabilities of language models. In this paper, we extend the results from (Berend, 2023) in several important aspects. On the one hand, – instead of using a medium-sized BERT model – we pre-train basesized DeBERTa (He et al., 2021) models, illustrating that the MLSM pre-training objective generalizes across different model types and sizes. On the other hand, we investigate the added value of a multi-task learning setting during pre-training, in which the use of MLSM objective is coupled with traditional MLM. Our empirical results show vast improvements in the performance of the pretrained language models using the joint objective. We release our source code1 and pre-trained models that we created using the strict2 and strict-small3 datasets provided as part of the BabyLM shared task (Warstadt et al., 2023).",
    "conclusion": "Even though MLSM is a cognitively more appealing pre-training objective than MLM, models exclusively pre-trained with MLSM fail at assigning reliable pseudo-log-likelihood scores to sequences (§3.3.1). To this end, we experimented with the coupled use of MLSM loss and the traditional MLM objective.\nOur empirical results suggest that the joint use of masked latent semantic modeling and traditional masked language modeling can boost the performance of the pre-trained language models. This is especially the case for tasks that directly assess the linguistic capabilities of the pre-trained models that were obtained by relying on limited corpus size, i.e., the 10 million token strict-small dataset. Our ablation experiments also revealed that the advantages of MLSM pre-training are more pronounced during the earlier phase of pre-training."
  },
  {
    "sample_id": 88,
    "title": "Better Word Representations with Recursive Neural Networks for Morphology",
    "abstract": "Vector-space word representations have been very successful in recent years at improving performance across a variety of NLP tasks. However, common to most existing work, words are regarded as independent entities without any explicit relationship among morphologically related words being modeled. As a result, rare and complex words are often poorly estimated, and all unknown words are represented in a rather crude way using only one or a few vectors. This paper addresses this shortcoming by proposing a novel model that is capable of building representations for morphologically complex words from their morphemes. We combine recursive neural networks (RNNs), where each morpheme is a basic unit, with neural language models (NLMs) to consider contextual information in learning morphologicallyaware word representations. Our learned models outperform existing word representations by a good margin on word similarity tasks across many datasets, including a new dataset we introduce focused on rare words to complement existing ones in an interesting way.",
    "introduction": "The use of word representations or word clusters pretrained in an unsupervised fashion from lots of text has become a key “secret sauce” for the success of many NLP systems in recent years, across tasks including named entity recognition, part-ofspeech tagging, parsing, and semantic role labeling. This is particularly true in deep neural network models (Collobert et al., 2011), but it is also true in conventional feature-based models (Koo et al., 2008; Ratinov and Roth, 2009).\nDeep learning systems give each word a distributed representation, i.e., a dense lowdimensional real-valued vector or an embedding. The main advantage of having such a distributed representation over word classes is that it can capture various dimensions of both semantic and syntactic information in a vector where each dimension corresponds to a latent feature of the word. As a result, a distributed representation is compact, less susceptible to data sparsity, and can implicitly represent an exponential number of word clusters.\nHowever, despite the widespread use of word clusters and word embeddings, and despite much work on improving the learning of word representations, from feed-forward networks (Bengio et al., 2003) to hierarchical models (Morin, 2005; Mnih and Hinton, 2009) and recently recurrent neural networks (Mikolov et al., 2010; Mikolov et al., 2011), these approaches treat each full-form word as an independent entity and fail to capture the explicit relationship among morphological variants of a word.1 The fact that morphologically complex words are often rare exacerbates the problem. Though existing clusterings and embeddings represent well frequent words, such as “distinct”, they often badly model rare ones, such as “distinctiveness”.\nIn this work, we use recursive neural networks (Socher et al., 2011b), in a novel way to model morphology and its compositionality. Essentially, we treat each morpheme as a basic unit in the RNNs and construct representations for morphologically complex words on the fly from their morphemes. By training a neural language model (NLM) and integrating RNN structures for complex words, we utilize contextual information in an interesting way to learn morphemic semantics and their compositional properties. Our model has the capability of building representations for any new unseen word comprised of known morphemes, giving the model an infinite (if still incomplete) covered vocabulary.\nOur learned representations outperform publicly available embeddings by a good margin on word similarity tasks across many datasets, which include our newly released dataset focusing on rare words (see Section 5). The detailed analysis in Section 6 reveals that our models can blend well syntactic information, i.e., the word structure, and the semantics in grouping related words.2",
    "conclusion": "This paper combines recursive neural networks (RNNs) and neural language models (NLMs) in a novel way to learn better word representations. Each of these components contributes to the learned syntactic-semantic word vectors in a unique way. The RNN explicitly models the morphological structures of words, i.e., the syntactic information, to learn morphemic compositionality. This allows for better estimation of rare and complex words and a more principled way of handling unseen words, whose representations could be constructed from vectors of known morphemes.\nThe NLMs, on the other hand, utilize surrounding word contexts to provide further semantics to the learned morphemic representations. As a result, our context-sensitive morphoRNN embeddings could significantly outperform existing embeddings on word similarity tasks for many datasets. Our analysis reveals that the model could blend well both the syntactic and semantic information in clustering related words. We have also made available a word similarity dataset focusing on rare words to complement existing ones which tend to include frequent words.\nLastly, as English is still considered limited in terms of morphology, our model could potentially yield even better performance when applied to other morphologically complex languages such as Finnish or Turkish, which we leave for future work. Also, even within English, we expect our model to be value to other domains, such as bioNLP with complicated but logical taxonomy."
  },
  {
    "sample_id": 89,
    "title": "Beware of Model Collapse! Fast and Stable Test-time Adaptation for Robust Question Answering",
    "abstract": "Although pre-trained language models (PLM) have achieved great success in question answering (QA), their robustness is still insufficient to support their practical applications, especially in the face of distribution shifts. Recently, testtime adaptation (TTA) has shown great potential for solving this problem, which adapts the model to fit the test samples at test time. However, TTA sometimes causes model collapse, making almost all the model outputs incorrect, which has raised concerns about its stability and reliability. In this paper, we delve into why TTA causes model collapse and find that the imbalanced label distribution inherent in QA is the reason for it. To address this problem, we propose Anti-Collapse Fast test-time adaptation (Anti-CF), which utilizes the source model‘s output to regularize the update of the adapted model during test time. We further design an efficient side block to reduce its inference time. Extensive experiments on various distribution shift scenarios and pre-trained language models (e.g., XLM-RoBERTa, BLOOM) demonstrate that our method can achieve comparable or better results than previous TTA methods at a speed close to vanilla forward propagation, which is 1.8× to 4.4× speedup compared to previous TTA methods. Our code is available at https://github.com/yisunlp/Anti-CF.",
    "introduction": "Pre-trained language models (PLMs) have achieved great success on many NLP tasks (Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020a; Raffel et al., 2020; Brown et al., 2020; OpenAI, 2022, 2023; Touvron et al., 2023). However, their success is based on the assumption that the test distribution is consistent with the training distribution. In many scenarios, this assumption is not true, such as adversarial attack (Wang et al., 2022), cross-lingual (Li et al., 2021), cross-domain (Ramponi and Plank, 2020), and so on. This situation is known as distribution shift. Unfortunately, even the most advanced models currently available, such as ChatGPT, do not perform well under the distribution shift (Ye et al., 2023; Wang et al., 2023).\nTo address this problem, researchers have proposed many approaches such as adversarial training (Zhu et al., 2020; Wang et al., 2021a), data augmentation (Zhou et al., 2021; Chen et al., 2021a). These methods improve the robustness of the model by changing the training strategy, but according to the No Free Lunch Theorem (Wolpert and Macready, 1997), a fixed model still cannot perform perfectly in all distribution-shifted scenarios. Therefore, some works (Wang et al., 2021b; Sun et al., 2020; Niu et al., 2022; Ye et al., 2022) explore how to update the model during the testing phase to adapt it to the distribution shifts of the test samples, called Test Time Adaptation (TTA). A typical approach (Wang et al., 2021b) uses the Shannon entropy of the probability given by the model as the loss to update itself. However, due to the unreliable output of the model, TTA may accumulate erroneous information learned in test samples, leading to model collapse and a sharp decline in model performance, which makes TTA extremely unstable and unreliable in practical applications.\nTo solve this problem, we take QA task as an example and investigate why TTA causes the model collapse. Our experiments indicate that the main reason for the model collapse is the imbalanced label distribution of the test data. In contrast to the direct inference, TTA exacerbates this imbalanced distribution, making all outputs of the model to be a specific class. Therefore, we propose Anti-Collapse Fast test-time adaptation (Anti-CF), which utilizes the output of the source model as a soft label to regularize the update of the adapted model during test time to ensure that the adapted model will not deviate too far from the source model, thus avoiding model collapse.\nHowever, to obtain the output of the source model and the adapted model, we need to keep the parameters of two models and conduct forward propagation twice, which will bring a lot of additional costs in practical applications. Therefore, we freeze the source model and add an efficient side block as the adapted model to reduce the cost of additional forward propagation and back propagation. Extensive experiments on various distribution shift scenarios and PLMs demonstrate that our method can achieve comparable or better results than previous TTA methods at a speed close to vanilla forward propagation, which is 1.8× to 4.4× speedup compared to previous TTA methods.\nOverall, our contributions in this work include:\n• We investigate why TTA causes model collapse in QA and find that the imbalanced label distribution inherent in QA is the reason for it.\n• We propose Anti-Collapse Fast test-time adaptation (Anti-CF) to solve the problem that TTA sometimes leads to model collapse.\n• Experimental results show that Anti-CF can effectively prevent the model from collapsing with a fast inference speed. It improves the stability and reliability of TTA in practical applications.",
    "conclusion": "In this paper, we attempt to improve the robustness of QA models by testing time adaptation (TTA) but find that TTA causes the models collapse. We thoroughly investigate why previous TTA methods cause the model collapse and find that the imbalanced label distribution is the main reason. We address this problem by adding constraints between the source and adapted model during the TTA process. We also design an efficient side block to speed up the inference time. Sufficient experimental results show that our proposed method is effective and efficient, making TTA a big step closer to being applied in real-world scenarios."
  },
  {
    "sample_id": 90,
    "title": "Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer",
    "abstract": "Fine-tuning (FT) pre-trained sentence embedding models on small datasets has been shown to have limitations. In this paper we show that concatenating the embeddings from the pretrained model with those from a simple sentence embedding model trained only on the target data, can improve over the performance of FT for few-sample tasks. To this end, a linear classifier is trained on the combined embeddings, either by freezing the embedding model weights or training the classifier and embedding models end-to-end. We perform evaluation on seven small datasets from NLP tasks and show that our approach with end-to-end training outperforms FT with negligible computational overhead. Further, we also show that sophisticated combination techniques like CCA and KCCA do not work as well in practice as concatenation. We provide theoretical analysis to explain this empirical observation.",
    "introduction": "Fine-tuning (FT) powerful pre-trained sentence embedding models like BERT (Devlin et al., 2018) has recently become the de-facto standard for downstream NLP tasks. Typically, FT entails jointly learning a classifier over the pre-trained model while tuning the weights of the latter. While FT has been shown to improve performance on tasks like GLUE (Wang et al., 2018) having large datasets (QQP, MNLI, QNLI), similar trends have not been observed on small datasets, where one would expect the maximum benefits of using a pre-trained model. Several works (Phang et al., 2018; Garg et al., 2019; Dodge et al., 2020; Lee et al., 2020) have demonstrated that FT with a few target domain samples is unstable with high variance, thereby often leading to sub-par gains. Furthermore, this issue has also been well documented in practice 1.\nLearning with low resources has recently become an active research area in NLP, and arguably one of the most interesting scenarios for which pre-trained models are useful (e.g., (Cherry et al., 2019)). Many practical applications have small datasets (e.g., in social science, medical studies, etc), which are different from large-scale academic benchmarks having hundreds of thousands of training samples (e.g, DBpedia (Lehmann et al., 2015), Sogou News (Wang et al., 2008), etc). This necessitates effective transfer learning approaches using pre-trained sentence embedding models for fewsample tasks.\nIn this work, we show that concatenating sentence embeddings from a pre-trained model and those from a smaller model trained solely on the target data, can improve over the performance of FT. Specifically, we first learn a simple sentence embedding model on the target data. Then we concatenate(C AT) the embeddings from this model with those from a pre-trained model, and train a linear classifier on the combined representation. The latter can be done by either freezing the embedding model weights or training the whole network (classifier plus the two embedding models) end-to-end.\nWe evaluate our approach on seven small datasets from NLP tasks. Our results show that our approach with end-to-end training can significantly improve the prediction performance of FT, with less than a 10% increase in the run time. Furthermore, our approach with frozen embedding models performs better than FT for very small datasets while reducing the run time by 30%−50%, and without the requirement of large memory GPUs.\nWe also conduct evaluations of multiple techniques for combining the pre-trained and domainspecific embeddings, comparing concatenation to CCA and KCCA. We observe that the simplest approach of concatenation works best in practice. Moreover, we provide theoretical analysis to explain this empirical observation.\nFinally, our results also have implications on the semantics learning ability of small domainspecific models compared to large pre-trained models. While intuition dictates that a large pre-trained model should capture the entire semantics learned by a small domain-specific model, our results show that there exist semantic features captured solely by the latter and not by the former, in spite of pretraining on billions of words. Hence combining the embeddings can improve the performance of directly FT the pre-trained model.\nRelated Work Recently, several pre-trained models have been studied, of which some provide explicit sentence embeddings (Conneau et al., 2017; Subramanian et al., 2018), while others provide implicit ones (Howard and Ruder, 2018; Radford et al., 2018). Peters et al. (2019) compare the performance of feature extraction (by freezing the pre-trained weights) and FT. There exists other more sophisticated transferring methods, but they are typically much more expensive or complicated. For example, Xu et al. (2019) “post-train” the pretrained model on the target dataset, Houlsby et al. (2019) inject specifically designed new adapter layers, Arase and Tsujii (2019) inject phrasal paraphrase relations into BERT, Sun et al. (2019) use multi-task FT, and Wang et al. (2019) first train a deep network classifier on the fixed pre-trained embedding and then fine-tune it. Our focus is to propose alternatives to FT with similar simplicity and computational efficiency, and study conditions where it has significant advantages. While the idea of concatenating multiple embeddings has been previously used (Peters et al., 2018), we use it for transfer learning in a low resource target domain.",
    "conclusion": "In this paper we have proposed a simple method for transferring a pre-trained sentence embedding model for text classification tasks. We empirically show that concatenating pre-trained and domain specific sentence embeddings, learned on the target dataset, with or without fine-tuning can improve the classification performance of pre-trained models like BERT on small datasets. We have also provided theoretical analysis identifying the conditions when this method is successful and to explain the experimental results."
  },
  {
    "sample_id": 91,
    "title": "Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation",
    "abstract": "Open Domain dialog system evaluation is one of the most important challenges in dialog research. Existing automatic evaluation metrics, such as BLEU are mostly referencebased. They calculate the difference between the generated response and a limited number of available references. Likert-score based self-reported user rating is widely adopted by social conversational systems, such as Amazon Alexa Prize chatbots. However, selfreported user rating suffers from bias and variance among different users. To alleviate this problem, we formulate dialog evaluation as a comparison task. We also propose an automatic evaluation model CMADE (Comparison Model for Automatic Dialog Evaluation) that automatically cleans self-reported user ratings as it trains on them. Specifically, we first use a self-supervised method to learn better dialog feature representation, and then use KNN and Shapley to remove confusing samples. Our experiments show that CMADE achieves 89.2% accuracy in the dialog comparison task. Our implementation is available at https://github.com/Weixin-Liang/\ndialog_evaluation_CMADE.",
    "introduction": "Open-domain dialog system evaluation is one of the most difficult challenges in the dialog community. Open-domain chatbots have a user-centric goal: to provide human with enjoyable user experience. However, user experience is difficult to quantify due to bias and variance among different users. Previous research has optimized on automatic dialog evaluation metrics such as BLUE (Papineni et al., 2002), which measures the difference between the generated responses and the reference responses. Due to the contrast between the one-tomany nature of open-domain conversations and the limited number of available references, such metrics correlate poorly with human judgments (Liu et al., 2016; Lowe et al., 2017; Novikova et al., 2017). Designing a fully automatic dialog evaluation metric is still an open research problem.\nCurrently, both academia and industry (Ram et al., 2018a; Li et al., 2019b; Liang et al., 2019) rely on human ratings to evaluate open-domain dialogs. Following the ubiquitous application of Likert scores in survey research like online reviews (Godes and Silva, 2012) and consumer satisfaction (Peterson and Wilson, 1992), a common practice of human evaluation on dialogs is to ask either a third-person rater or the chatbot user to report a Likert score. However, concerns have been raised about the validity of Likert score-based ratings. Kulikov et al. (Kulikov et al., 2018) observe high bias and variance of Likert scores. Such issue is more severe in real-world commercial dialog systems like Alexa social chatbot (Ram et al., 2018a; Venkatesh et al., 2018), because the real-world users have neither monetary incentive nor necessary annotation training to calibrate their ratings.\nTo explore the validity of Likert score based dialog evaluation, we first perform a large-scale data analysis of 3,608 collected real-world humanmachine dialogs along with their self-reported Likert scale ratings from Amazon Alexa Prize Challenge (Ram et al., 2018a; Yu et al., 2019; Chen et al., 2018). One noticeable property of the ratings is its J-shape skew distribution: nearly half of the dialogs are rated with the highest Likert score. The prevalence of such extreme distribution of ratings has long been observed by the business research community in variable aspects of reallife (Schoenm¨ uller et al., 2018; Godes and Silva, 2012; Hu et al., 2017; Zervas et al., 2015).\nAlthough we could tell which dialog system is better by running statistical test on a large number of noisy ratings, it is difficult to locate dialogs with bad performance reliably to improve dialog system quality. In this paper, we take on the challenge of calibrating a large number of noisy self-reported user ratings to build better dialog evaluation models. We formulate the task as to first denoise the self-reported user ratings and then train a model on the cleaned ratings. We design CMADE (Comparison Model for Automatic Dialog Evaluation), a progressive three-stage denoising pipeline. We first perform a self-supervised learning to obtain good dialog representations. We then fine-tune CMADE on smoothed self-reported user ratings to improve the dialog representation while preventing the network from overfitting on noisy ratings. Finally, we apply data Shapley to remove noisy training data, and fine-tune the model on the cleaned training set. Our experiments show that CMADE is able to successfully identify noisy training data and achieves 89.2% in accuracy and 0.787 in Kappa on a test set with unseen expert-rated dialog pairs.",
    "conclusion": "The ultimate chatbot evaluation metric should be user-centric, as chatbots are there to provide human with an enjoyable experiences. Previously Likertscore based self-reported rating is the de-facto standard for current dialog evaluation . However, our analysis indicates that self-reported dialog ratings are skewed (J-shape), noisy and insensitive due to bias and variance among different users. We propose a three-stage denoising pipeline CMADE to reduce self-reported ratings and, at the same time, build an automatic comparison-based automatic dialog quality predictor. CMADE’s results highly correlate with expert judgments on pair-wise dia-\nmovie name please?\nUser: all the harry potter series\nSys: ah, i don’t know any movies by that name. sorry if i sometimes confuse you saying something else with a movie name ...\n...\nKappa)."
  },
  {
    "sample_id": 92,
    "title": "Bi-Directional Iterative Prompt-Tuning for Event Argument Extraction",
    "abstract": "Recently, prompt-tuning has attracted growing interests in event argument extraction (EAE). However, the existing prompt-tuning methods have not achieved satisfactory performance due to the lack of consideration of entity information. In this paper, we propose a bidirectional iterative prompt-tuning method for EAE, where the EAE task is treated as a clozestyle task to take full advantage of entity information and pre-trained language models (PLMs). Furthermore, our method explores event argument interactions by introducing the argument roles of contextual entities into prompt construction. Since template and verbalizer are two crucial components in a clozestyle prompt, we propose to utilize the role label semantic knowledge to construct a semantic verbalizer and design three kinds of templates for the EAE task. Experiments on the ACE 2005 English dataset with standard and low-resource settings show that the proposed method significantly outperforms the peer stateof-the-art methods. Our code is available at https://github.com/HustMinsLab/BIP.",
    "introduction": "As a key step of event extraction, event argument extraction refers to identifying event arguments with predefined roles. For example, for an \"Attack\" event triggered by the word \"fired\" in the sentence \"Iraqis have fired sand missiles and AAA at aircraft\", EAE aims to identify that \"Iraqis\", \"missiles\", \"AAA\" and \"aircraft\" are event arguments with the \"Attacker\", \"Instrument\", \"Instrument\" and \"Target\" roles, respectively.\nIn order to exploit the rich linguistic knowledge contained in pre-trained language models, finetuning methods have been proposed for EAE. The paradigm of these methods is to use a pre-trained language model to obtain semantic representations,\n(a) Fine-Tuning for EAE\nFigure 1: Illustration of fine-tuning and prompt-tuning methods for predicting the argument role of the entity mention \"Iraqis\" in the event triggered by the word \"fired\".\nand then feed these representations into a welldesigned neural network to extract event arguments. For example in Figure 1(a), an event trigger representation and an entity mention representation are first obtained through a pre-trained language model, and then input to a designed neural network, such as hierarchical modular network (Wang et al., 2019) and syntax-attending transformer network (Ma et al., 2020), to determine the argument role that the entity mention plays in the event triggered by the trigger. However, there is a significant gap between the EAE task and the objective form of pre-training, resulting in the poor utilization of the prior knowledge in PLMs. Additionally, finetuning methods heavily depend on extensive annotated data and perform poorly in low-resource data scenarios.\nTo bridge the gap between the EAE task and the pre-training task, prompt-tuning methods (Li et al., 2021; Ma et al., 2022; Hsu et al., 2022; Liu et al., 2022) recently have been proposed to formalize the EAE task into a more consistent form with the training objective of generative pre-trained language models. These methods achieve significantly better performance than fine-tuning methods in low-resource data scenarios, but not as good as the state-of-the-art fine-tuning method ONEIE (Lin et al., 2020) in high-resource data scenarios.\nTo achieve excellent performance in both lowresource and high-resource data scenarios, we leverage entity information to model EAE as a clozestyle task and use a masked language model to handle the task. Figure 1(b) shows a typical clozestyle prompt-tuning method for EAE. The typical prompt-tuning method suffers from two challenges: (i) The typical human-written verbalizer (Schick and Schütze, 2021) is not a good choice for EAE. The human-written verbalizer is to manually assign a label word to each argument role. For example in Figure 1(b), we choose the \"attacker\" as the label word of \"Attacker\" role. However, an argument role may have different definitions in different types of events. For example, the \"Entity\" role refers to \"the voting agent\" and \"the agents who are meeting\" in the \"Elect\" and \"MEET\" events, respectively. (ii) Event argument interactions are not explored. Existing work (Sha et al., 2018; Xiangyu et al., 2021; Ma et al., 2022) has demonstrated the usefulness of event argument interactions for EAE. For the \"Attack\" event triggered by the word \"fired\" in Figure 1, given that \"missiles\" is an \"Instrument\", it is more likely to correctly classify \"AAA\" into the \"Instrument\" role.\nIn this paper, we propose a bi-directional iterative prompt-tuning (BIP) method to alleviate the aforementioned challenges. To capture argument interactions, a forward iterative prompt and a backward iterative prompt are constructed to utilize the argument roles of contextual entities to predict the current entity’s role. For the verbalizer, we redefine the argument role types and assign a virtual label word to each argument role, where the initial representation of each virtual label word is generated based on the semantic of the argument role. In addition, we design three kind of templates: hard template, soft template, and hard-soft template, which are further discussed in the experimental section. Extensive experiments on the ACE 2005 English dataset show that the proposed method can achieve the state-of-the-art performance in both low-resource and high-resource data scenarios.",
    "conclusion": "In this paper, we regard event argument extraction as a cloze-style task and propose a bidirectional iterative prompt-tuning method to address this task. The bi-directional iterative prompttuning method contains a forward iterative prompt and a backward iterative prompt, which predict the argument role of each entity in a left-to-right and right-to-left manner respectively. For the template construction in each prompt, the predicted argument role information is introduced to capture argument interactions. In addition, a novel semantical verbalizer is designed based on the semantic of the argument role. And three kinds of templates are designed and discussed. Experiment results have shown the effectiveness of our method in both high-resource and low-resource data scenarios. In the future work, we are interested in the joint prompt-tuning method of event detection and event argument extraction."
  },
  {
    "sample_id": 93,
    "title": "Bi-Phone: Modeling Inter Language Phonetic Influences in Text",
    "abstract": "A large number of people are forced to use the Web in a language they have low literacy in due to technology asymmetries. Written text in the second language (L2) from such users often contains a large number of errors that are influenced by their native language (L1). We propose a method to mine phoneme confusions (sounds in L2 that an L1 speaker is likely to conflate) for pairs of L1 and L2. These confusions are then plugged into a generative model (Bi-Phone) for synthetically producing corrupted L2 text. Through human evaluations, we show that Bi-Phone generates plausible corruptions that differ across L1s and also have widespread coverage on the Web. We also corrupt the popular language understanding benchmark SuperGLUE with our technique (FunGLUE for Phonetically Noised GLUE) and show that SoTA language understating models perform poorly. We also introduce a new phoneme prediction pre-training task which helps byte models to recover performance close to SuperGLUE. Finally, we also release the FunGLUE benchmark to promote further research in phonetically robust language models. To the best of our knowledge, FunGLUE is the first benchmark to introduce L1-L2 interactions in text.",
    "introduction": "We live in a multilingual world with over 7,000 languages spoken across the globe (Eberhard and Fennig, 2022). However, technology asymmetrically supports only a few specific languages. For instance, the internet is mostly in English with over 60% of websites using the language despite just around 16% share of its speaking population around the world1 (Grefenstette and Nioche, 2000). Increasingly, people are forced to navigate and produce content on the web in languages they have not been formally trained on. The English text produced by ESL (English as Second / L2 language) writers is heavily influenced by their native language (L1).\nResearch in the field of second-language acquisition has found evidence of phoneme-shift based misspellings stemming from L1 influence in L2 text for specific language pairs (Ibrahim, 1978; Cook, 1997; Bestgen and Granger, 2011; Sari, 2014; Ogneva, 2018; Motohashi-Saigo and Ishizawa, 2020). Studies in Natural Language Understanding (NLU) have been limited to spelling correction Nagata et al. (2017); Flor et al. (2019) and native language identification Chen et al. (2017); Nicolai et al. (2013) in English learners. These studies predominantly use the TOEFL11 dataset (Blanchard et al., 2013) which deals with very specific demographics such as test-takers who have formal training in the L2 language.\nWe make the following four key observations about prior work in the study of L1-L2 influences in text and speech. First, current models for L1-L2 influence on textual spelling are limited to certain language pairs and tasks. We argue that L1-L2 influence phenomenon is much more broad and is language and task agnostic. Second, there is no large scale study to examine the prevalence of this phenomenon on the open web. Third, given that this is an important problem especially for multilingual, new-to-the-internet communities there is no standardized benchmark to study the robustness of natural language understanding(NLU) and Natural Language Generation (NLG) models to inter-language phonetic noise. Finally, there is very sparse literature on architecture / pre-training strategies to introduce phonetic robustness into large language models. In this paper, we present modeling techniques,data analyses and a new benchmark to address the gaps mentioned above. We summarise our contributions as follows:\n1. We propose a language-agnostic method to mine phoneme confusions that arise due to interference between a native language (L1)\nand second language (L2). Our method exploits the “hidden knowledge\" contained in\nL1\n→\nL2 and L2\n→\nL1 transliteration mod-\nels. We also propose a generative model BiPhone that is able to synthetically produce spelling corruption in accordance with L1-L2 confusions (Sections 3.1, 3.2).\n2. Through human evaluation and coverage analysis we show that Bi-Phone produces spelling corruptions that are not only deemed plausible by native L1 speakers but also have substantial coverage in the open web crawl corpus. To the best of our knowledge no prior work has demonstrated the presence of L1-L2 phonetic corruptions in a large scale, common dataset like Common Crawl (Section 4).\n3. We release a dataset consisting of sentences with L1-L2 phonetic spelling corruptions found in Common Crawl. We also release a benchmark called FunGLUE, an extension of the SuperGLUE benchmark for L1-L2 spelling corruptions. To the best of our knowledge FunGLUE is the first benchmark to measure the robustness of models to L1-L2 interference in text (Section 5).\n4. We show SoTA models do not perform well on FunGLUE. We then introduce a novel pretraining task of phoneme prediction, which together with byte level architectures substantially bridges the gap on the noised benchmark (by up to 11% absolute on certain test sets). This is particularly impressive since this gain is achieved without ever showing the model any noised examples (Section 6).",
    "conclusion": "Language is a significant barrier to technology especially for new internet users. For such users, English often is not their first language. The speech community has made significant progress in making technology (ASR for instance) accessible for such users by making models robust to account for inter-language interactions. We argue that a similar line of effort is needed in the Natural Language Understanding for Text community as well. To this end, we first propose a generative model Bi-Phone that can account for L1-L2 interactions in text. Next we show the inter-language perturbations generated by Bi-Phone are indeed present in non-trival amount in the common crawl corpus. We also release a new benchmark FunGLUE to help further research in this area. We also present our early yet very promising explorations on making natural language understanding models robust to L1-L2 phonetic shifts through a novel phoneme prediction based pre-training."
  },
  {
    "sample_id": 94,
    "title": "Bias Analysis and Mitigation in the Evaluation of Authorship Verification",
    "abstract": "The PAN series of shared tasks is well known for its continuous and high quality research in the field of digital text forensics. Among others, PAN contributions include original corpora, tailored benchmarks, and standardized experimentation platforms. In this paper we review, theoretically and practically, the authorship verification task and conclude that the underlying experiment design cannot guarantee pushing forward the state of the art—in fact, it allows for top benchmarking with a surprisingly straightforward approach. In this regard, we present a “Basic and Fairly Flawed” (BAFF) authorship verifier that is on a par with the best approaches submitted so far, and that illustrates sources of bias that should be eliminated. We pinpoint these sources in the evaluation chain and present a refined authorship corpus as effective countermeasure.",
    "introduction": "When tackling a problem in empirical research, a sound and reliable evaluation of competing solution approaches is a prerequisite to achieve agreement on the state-of-the-art performance. For authorship verification, the PAN series of shared tasks caters for the most important benchmarks to which new approaches refer and compare against. The fundamental problem in authorship verification is to decide whether two given texts were written by the same author. When experimenting within the PAN setting, we learned that one can quickly achieve a competitive performance for this task—with one of the most basic approaches: a TFIDF-weighted character 3-gram model. By extending this model with a few additional features, such as the KullbackLeibler divergence and related measures, we were able to reach the performance of the best verifiers submitted so far.1 However, reality caught up with us when we applied our verifier to other authorship verification problems with little success. To get to the bottom of this rather baffling outcome, we carried out a systematic analysis of the entire evaluation chain, its problem definition, its corpora, its evaluation procedure, and of course our model, in search of any sources of bias that may have artificially inflated the performance of our approach. The paper in hand introduces our “Basic and Fairly Flawed” (BAFF) model and reports on our bias analysis. Moreover, in an attempt to improve the situation and call for better data, we not only contribute a new and carefully curated authorship verification corpus,2 but also collect a few best practices for the creation of such corpora. The outlined situation calls into question a lot of what we believed to know about the state of the art, and future PAN tasks on verification will have to rectify these issues in order to provide for a more valid assessment of the state of the art.",
    "conclusion": "In shared tasks, sometimes basic approaches outperform more sophisticated ones. This is frequently the case when machine learning meets small data. Inadvertent properties of the data act as confounders that a learning algorithm will gladly fit onto if they are not controlled. In the case of authorship verification as per PAN, this was a major part of the problem. As long as much larger corpora remain out of reach for lack of a sufficient source of monographs, extra care needs to be taken in preparing the data, as exemplified for our corpus.\nAnother important take-away message is that model authors in authorship verification need to be extra careful about their feature selection. Fortunately, this will come naturally to researchers in the field as they are already trained to avoid features that encode topic rather than style. In particular, we strongly suggest that future evaluations should adopt a stateless one-case-at-a-time test policy.\nFinally, in a spin-off study on unmasking, we generalized the algorithm to work on short, essaylength texts (Bevendorff et al., 2019): it achieves an accuracy of 0.73, an F 1 of 0.69, and a precision of 0.82, marking the first baseline for our corpus."
  },
  {
    "sample_id": 95,
    "title": "BinaryBERT: Pushing the Limit of BERT Quantization",
    "abstract": "The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution. In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization. We find that a binary BERT is hard to be trained directly than a ternary counterpart due to its complex and irregular loss landscape. Therefore, we propose ternary weight splitting, which initializes BinaryBERT by equivalently splitting from a half-sized ternary network. The binary model thus inherits the good performance of the ternary one, and can be further enhanced by fine-tuning the new architecture after splitting. Empirical results show that our BinaryBERT has only a slight performance drop compared with the full-precision model while being 24× smaller, achieving the state-of-the-art compression results on the GLUE and SQuAD benchmarks.",
    "introduction": "Recent pre-trained language models have achieved remarkable performance improvement in various natural language tasks (Vaswani et al., 2017; Devlin et al., 2019). However, the improvement generally comes at the cost of increasing model size and computation, which limits the deployment of these huge pre-trained language models to edge devices. Various methods have been recently proposed to compress these models, such as knowledge distillation (Sanh et al., 2019; Sun et al., 2019; Jiao et al., 2020), pruning (Michel et al., 2019; Fan et al., 2019), low-rank approximation (Ma et al., 2019; Lan et al., 2020), weightsharing (Dehghani et al., 2019; Lan et al., 2020; Huang et al., 2021), dynamic networks with adaptive depth and/or width (Hou et al., 2020; Xin et al., 2020; Zhou et al., 2020), and quantization (Zafrir\n(a) MRPC. (b) MNLI-m.\nFigure 1: Performance of quantized BERT with varying weight bit-widths and 8-bit activation. We report the mean results with standard deviations from 10 seeds on MRPC and 3 seeds on MNLI-m, respectively.\net al., 2019; Shen et al., 2020; Fan et al., 2020; Zhang et al., 2020).\nAmong all these model compression approaches, quantization is a popular solution as it does not require designing a smaller model architecture. Instead, it compresses the model by replacing each 32-bit floating-point parameter with a low-bit fixedpoint representation. Existing attempts try to quantize pre-trained models (Zafrir et al., 2019; Shen et al., 2020; Fan et al., 2020) to even as low as ternary values (2-bit) with minor performance drop (Zhang et al., 2020). However, none of them achieves the binarization (1-bit). As the limit of quantization, weight binarization could bring at most 32× reduction in model size and replace most floating-point multiplications with additions. Moreover, quantizing activations to 8-bit or 4-bit further replaces the floating-point addition with int8 and int4 addition, decreasing the energy burden and the area usage on chips (Courbariaux et al., 2015).\nIn this paper, we explore to binarize BERT parameters with quantized activations, pushing BERT quantization to the limit. We find that directly training a binary network is rather challenging. According to Figure 1, there is a sharp performance drop when reducing weight bit-width from 2-bit to 1-bit, compared to other bit configurations. To explore the challenges of binarization, we analyze the loss landscapes of models under different precisions both qualitatively and quantitatively. It is found that while the full-precision and ternary (2bit) models enjoy relatively flat and smooth loss surfaces, the binary model suffers from a rather steep and complex landscape, which poses great challenges to the optimization.\nMotivated by the above empirical observations, we propose ternary weight splitting, which takes the ternary model as a proxy to bridge the gap between the binary and full-precision models. Specifically, ternary weight splitting equivalently converts both the quantized and latent full-precision weights in a well-trained ternary model to initialize BinaryBERT. Therefore, BinaryBERT retains the good performance of the ternary model, and can be further refined on the new architecture. While neuron splitting is previously studied (Chen et al., 2016; Wu et al., 2019) for full-precision network, our ternary weight splitting is much more complex due to the additional equivalence requirement of quantized weights. Furthermore, the proposed BinaryBERT also supports adaptive splitting. It can adaptively perform splitting on the most important ternary modules while leaving the rest as binary, based on efficiency constraints such as model size or floating-point operations (FLOPs). Therefore, our approach allows flexible sizes of binary models for various edge devices’ demands.\nEmpirical results show that BinaryBERT split from a half-width ternary network is much better than a directly-trained binary model with the original width. On the GLUE and SQuAD benchmarks, our BinaryBERT has only a slight performance drop compared to the full-precision BERT-base model, while being 24× smaller. Moreover, BinaryBERT with the proposed importance-based adaptive splitting also outperforms other splitting criteria across a variety of model sizes.",
    "conclusion": "In this paper, we propose BinaryBERT, pushing BERT quantization to the limit. As a result of the steep and complex loss landscape, we find directly training a BinaryBERT is hard with a large performance drop. We thus propose a ternary weight splitting that splits a trained ternary BERT to initialize BinaryBERT, followed by fine-tuning for further refinement. Our approach also supports adaptive splitting that can tailor the size of BinaryBERT based on the edge device constraints. Empirical results show that our approach significantly outperforms vanilla binary training, achieving stateof-the-art performance on BERT compression."
  },
  {
    "sample_id": 96,
    "title": "BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric",
    "abstract": "End-to-End speech-to-speech translation (S2ST) is generally evaluated with text-based metrics. This means that generated speech has to be automatically transcribed, making the evaluation dependent on the availability and quality of automatic speech recognition (ASR) systems.\nIn this paper, we propose a text-free evaluation metric for end-to-end S2ST, named BLASER, to avoid the dependency on ASR systems. BLASER leverages a multilingual multimodal encoder to directly encode the speech segments for source input, translation output and reference into a shared embedding space and computes a score of the translation quality that can be used as a proxy to human evaluation. To evaluate our approach, we construct training and evaluation sets from more than 40k human annotations covering seven language directions. The best results of BLASER are achieved by training with supervision from human rating scores. We show that when evaluated at the sentence level, BLASER correlates significantly better with human judgment compared to ASRdependent metrics including ASR-SENTBLEU in all translation directions and ASR-COMET in five of them. Our analysis shows combining speech and text as inputs to BLASER does not increase the correlation with human scores, but best correlations are achieved when using speech, which motivates the goal of our research. Moreover, we show that using ASR for references is detrimental for text-based metrics. 1",
    "introduction": "Speech-to-Speech translation seeks to translate speech segments from one language into another.\nHistorically, it has been implemented and evaluated as a concatenation of three systems: automatic speech recognition (ASR), machine translation (MT) and text-to-speech (TTS) (Lavie et al., 1997; Lazzari, 2006). In recent years, there has been increasing interest in end-to-end approaches (Jia et al., 2019; Lee et al., 2022a). While end-toend S2ST is becoming popular, researchers still rely on text-based metrics to evaluate model performance by automatically transcribing the generated speech segments (Jia et al., 2019). These cascaded metrics rely on ASR systems, which for a given language may not have enough quality or may not even be available (Javed et al., 2022). They are also inappropriate for languages lacking standardized writing systems (Salesky et al., 2021a), like Hokkien or Algerian Arabic.\nIn this work, we propose the text-free metric BLASER for S2ST evaluation, sidestepping the dependency on ASR systems. In particular, we use LASER encoders that support multiple languages and modalities including text (Heffernan et al., 2022) and speech (Duquenne et al., 2021). We use the LASER encoders to directly embed speech segments into vectors and compute a score estimating the quality of generation. We then construct training and evaluation datasets from more than 40k human annotations, covering seven language directions (Spanish ↔English, French ↔English,\nRussian →English, Hokkien →English, and\nEnglish →German). We evaluate BLASER on these\ndatasets on the popular benchmark of MusT-C (Di Gangi et al., 2019). We also benchmark several strong ASR-based metrics, e.g., ASR-SENTBLEU (i.e., sentence-level ASR-BLEU (Jia et al., 2019)) and ASR-COMET (i.e., applying COMET (Rei et al., 2020) on ASR outputs). There is a recent interest of supervised evaluation metrics that are trained on human quality scores (Rei et al., 2020). However, these human quality scores are precious and somehow limited or nonexistent, specially for\nlow-resource languages. Therefore, we propose both an unsupervised and a supervised version of BLASER. The results show that on average both unsupervised and supervised BLASER outperform their corresponding baseline metrics. In particular, BLASER outperforms ASR-COMET significantly in five language directions and obtains comparable results in two other language directions. Our analysis reveals that, while BLASER can use both text and speech, encoding speech data give the most significant benefits. In addition, we show that replacing human-written source input and human-written reference with ASR-generated ones hurts performance of text-based metrics, which motivates the use of modality-agnostic metrics as\nBLASER.",
    "conclusion": "We have introduced BLASER, a text-free metric to evaluate speech-to-speech translation, which avoids the dependency on ASR models required by popular text-based metrics currently used in S2ST.\nWe explored BLASER in both unsupervised and supervised settings. Experimental results in seven language directions show that BLASER outperforms or is comparable to strong text-based metrics in terms of correlation with human scores at the sentencelevel. Moreover, our metric is effective in zero-shot scenarios.\nAs for future work, we want to explore the use of speech references generated by humans and the impact of synthesized references. We also want to evaluate BLASER at the system-level with a much larger number of S2ST systems, and explore different approaches to aggregate the sentence-level scores from BLASER and we want to explore different speech and text representations as alternative to\nLASER."
  },
  {
    "sample_id": 97,
    "title": "BMInf: An Efficient Toolkit for Big Model Inference and Tuning",
    "abstract": "In recent years, large-scale pre-trained language models (PLMs) containing billions of parameters have achieved promising results on various NLP tasks. Although we can pretrain these big models by stacking computing clusters at any cost, it is impractical to use such huge computing resources to apply big models for each downstream task. To address the computation bottleneck encountered in deploying big models in real-world scenarios, we introduce an open-source toolkit for Big Model Inference and tuning (BMInf), which can support big model inference and tuning at extremely low computation cost. More specifically, at the algorithm level, we introduce model quantization and parameter-efficient tuning for efficient model inference and tuning. At the implementation level, we apply model offloading, model checkpointing, and CPU-GPU scheduling optimization to further reduce the computation and memory cost of big models. Based on above efforts, we can efficiently perform big model inference and tuning with a single GPU (even a consumer-level GPU like GTX 1060) instead of computing clusters, which is difficult for existing distributed learning toolkits for PLMs. BMInf is publicly released at\nhttps://github.com/OpenBMB/BMInf.",
    "introduction": "Recent years have witnessed the great success of pre-trained language models (PLMs) (Han et al., 2021) in the NLP community. Various techniques of PLMs enable us to train big models containing billions of parameters from large-scale unlabeled corpora in a self-supervised fashion. Up to now, these big models (with billions of parameters like GPT-3 (Brown et al., 2020)) have achieved promising results on various NLP tasks and gained extensive attention from researchers. Despite the success\nCPU\nHardware\nImplementation\nBMInf convenient for users, the underlying implementation and the hardware adaptation will not be exposed to users, and these modules can be automatically executed.\nof big models, the massive parameters of these big models also bring challenges to their inference and tuning. Since the pre-training process of big models usually requires to be completed once, the cost caused by massive parameters can be handled by stacking computing resources. However, the inference and tuning process of PLMs depends on specific application scenarios and will frequently use big models for computation. If we still stack devices to speed up the inference and tuning of big models, the cost of time, memory, and even money would become unbearable. In this paper, we introduce a toolkit BMInf, aiming at efficiently performing big model inference and tuning.\nAs shown in Figure 1, BMInf is built based on a four-level framework, the most important part of which lies in its algorithm level and implementation level. At the algorithm level, we introduce model quantization to compress big models from highbit floating-point parameters to low-bit fixed-point ones, which can significantly reduce the memory cost of big models. The faster computation speed of low-bit numbers can also accelerate the computation of big models. Besides model quantization, we also introduce parameter-efficient tuning methods (Ding et al., 2022), which freeze the parameters of big models to reduce the computation and memory cost. By inserting additional learnable modules into big models, parameter-efficient tuning can tune these additional modules to help big models handle specific tasks. Some recent works (Lester et al., 2021; Gu et al., 2021; Hu et al., 2021) have shown that applying parameter-efficient tuning on big models can achieve results comparable to finetuning all model weights.\nAt the implementation level, we implement model offloading and model checkpointing, which can make full use of CPU memory to store massive parameters of big models. Moreover, model offloading and checkpointing can drop parameters and computation graphs during both the forward and backward propagation, which can further save GPU memory to operate more data. For the underlying arithmetic operators, we reimplement the mixed-precision CUDA arithmetic operators, which can better utilize the tensor cores of GPUs to further speed up the computation, especially accelerating the mixed-precision computation in model quantization. Considering model offloading and model checkpointing bring extra CPU-GPU communication to load offloaded model weights, we perform CPU-GPU scheduling optimization to synchronously execute weight loading and model computation. This CPU-GPU scheduling optimization can alleviate the time waiting for weight loading. All of model offloading, model checkpointing, and parameter-efficient tuning can benefit from the scheduling optimization.\nDue to the algorithm-level and implementationlevel efficiencies, BMInf can work on various GPUs at the hardware level, including both powerful GPUs (e.g. Tesla V100 and Tesla A100) and consumer GPUs (e.g. GTX 1060 and GTX 1080Ti). In Section 4, we will show that BMInf can run models with more than 10 billion parameters on a consumer GPU GTX 1060, which is quite difficult for existing PLM-related distributed toolkits such as Megatron (Shoeybi et al., 2019) and DeepSpeed (Rasley et al., 2020). At the model level, BMInf supports various possible architectures of\n<latexit sha1_base64=\"ShB6g6w5a0qSHV/nanQZh4mr2T8=\">AAACyXicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdVl0I7ipYB9QRZJ0Wsfm5WQi1uLKH3CrPyb+gf6Fd8YR1CI6IcmZc+85M/dePw15Jh3npWBNTE5NzxRnS3PzC4tL5eWVZpbkImCNIAkT0fa9jIU8Zg3JZcjaqWBe5Ies5Q8OVLx1zUTGk/hEDlN2Fnn9mPd44EmimqeSRyw7L1ecqqOXPQ5cAyowq56Un3GKLhIEyBGBIYYkHMJDRk8HLhykxJ1hRJwgxHWc4Q4l0uaUxSjDI3ZA3z7tOoaNaa88M60O6JSQXkFKGxukSShPEFan2Tqea2fF/uY90p7qbkP6+8YrIlbigti/dJ+Z/9WpWiR62NM1cKop1YyqLjAuue6Kurn9pSpJDilxCncpLggHWvnZZ1trMl276q2n4686U7FqH5jcHG/qljRg9+c4x0Fzq+ruVN3j7Upt34y6iDWsY5PmuYsaDlFHg7wv8YBHPFlH1pV1Y91+pFoFo1nFt2XdvwP+iZG7</latexit>\n⇥\n<latexit sha1_base64=\"M772Qne/5PHLvGLHujZaViOyIGw=\">AAACxHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdSMUBXHZgn1ALZJMp3VoXiQToRT9Abf6beIf6F94Z5yCWkQnJDlz7j1n5t7rJ4HIpOO8Fqy5+YXFpeJyaWV1bX2jvLnVyuI8ZbzJ4iBOO76X8UBEvCmFDHgnSbkX+gFv+6NzFW/f8TQTcXQlxwnvhd4wEgPBPElU4/SmXHGqjl72LHANqMCselx+wTX6iMGQIwRHBEk4gIeMni5cOEiI62FCXEpI6DjHPUqkzSmLU4ZH7Ii+Q9p1DRvRXnlmWs3olIDelJQ29kgTU15KWJ1m63iunRX7m/dEe6q7jenvG6+QWIlbYv/STTP/q1O1SAxwomsQVFOiGVUdMy657oq6uf2lKkkOCXEK9ymeEmZaOe2zrTWZrl311tPxN52pWLVnJjfHu7olDdj9Oc5Z0DqoukdVt3FYqZ2ZURexg13s0zyPUcMl6mhq70c84dm6sAIrs/LPVKtgNNv4tqyHD+Dlj0g=</latexit>\n=\nFloat32\nFloat32\nFigure 2: The illustration of model quantization. To balance both efficiency and effectiveness, we use 8-bit fixed-point numbers to represent the weights of all linear layers and higher-bit floating-point numbers (16-bit or 32-bit numbers) to represent hidden states. Here we use 32-bit floating-point numbers as an example. The dotted parts are only used for the low-bit adaptation training.\nTransformer-based PLMs, and users can choose their own model architectures for inference and tuning. To make BMInf more convenient for users, the underlying implementation and the hardware adaptation are automatically executed and will not be exposed to users. In the following sections, we will show more details about BMInf, especially at the algorithm level and implementation level.",
    "conclusion": "In this paper, we introduce an efficient toolkit BMInf to provide a way to use large-scale PLMs. By applying model quantization, parameterefficient tuning, model offloading, model checkpointing, CPU-GPU scheduling optimization, as well as the reimplementation of mixed-precision arithmetic operators, BMInf can perform big model inference and tuning with less than 1/30 of the GPU memory and 10 times speedup, as compared with existing open-source distributed toolkits for pretraining and fine-tuning PLMs.\nIn the future, our work to improve BMInf will focus on the following three directions:\n(1) At the model level, we will gradually support more models;\n(2) At the algorithm level, we will continue to improve our model quantization methods to achieve better performance, and work with other toolkits such as OpenPrompt (Ding et al., 2021) to explore more effective ways to tune big models;\n(3) At the implementation level, we will provide long-term maintenance for this toolkit.\nWe hope this toolkit can help researchers utilize big models for their own works and advance the adaption of big models in the NLP community."
  },
  {
    "sample_id": 98,
    "title": "BOLT: Fast Energy-based Controlled Text Generation with Tunable Biases",
    "abstract": "Energy-based models (EBMs) have gained popularity for controlled text generation due to their high applicability to a wide range of constraints. However, sampling from EBMs is non-trivial, as it often requires a large number of iterations to converge to plausible text, which slows down the decoding process and makes it less practical for real-world applications. In this work, we propose BOLT, which relies on tunable biases to directly adjust the language model’s output logits. Unlike prior work, BOLT maintains the generator’s autoregressive nature to assert a strong control on token-wise conditional dependencies and overall fluency, and thus converges faster. When compared with state-of-the-arts on controlled generation tasks using both soft constraints (e.g., sentiment control) and hard constraints (e.g., keyword-guided topic control), BOLT demonstrates significantly improved efficiency and fluency. On sentiment control, BOLT is 7x faster than competitive baselines, and more fluent in 74.4% of the evaluation samples according to human judges.",
    "introduction": "Generating text using pre-trained language models (PLMs) to satisfy user-specified constraints is an important task to allow practical usage of PLMs. Common controlled text generation methods include training conditional language models (Keskar et al., 2019; Zhang et al., 2020) or attribute-based fine-tuning of PLMs (Liu et al., 2020; Zhang and Song, 2022). Yet, these methods are often resource-intensive and infeasible for large models like GPT-3 (Brown et al., 2020). Furthermore, these methods assume access to large amounts of attribute-specific data and are inflexible for new constraints. On the contrary, inference-time methods (Qin et al., 2022; Kumar et al., 2022; Mireshghallah et al., 2022) directly steer the generations without model re-training or\nSpeed (tokens/s)\nS\ne\nn t i\nm\ne\nn t\nC\no\nn t\nr o l\nFigure 1: Sentiment controllability (i.e., % of generations with a given sentiment, as estimated by a classifier) against sampling speed for different energy-based methods. BOLT shows a pronounced improvement in decoding speed with comparable or better control.\nfine-tuning. In particular, energy-based models (EBMs) (LeCun et al., 2006) have demonstrated greater flexibility, since they can accommodate arbitrary energy functions (Khalifa et al., 2021; Qin et al., 2022; Kumar et al., 2022).\nDespite their benefits, sampling from EBMs presents profound challenges. Notably, the sampling process, which is often done through Langevin Dynamics (Welling and Teh, 2011) or Gibbs Sampling (Goyal et al., 2022), requires a substantial number of iterations to converge to readable sequences of text. This can significantly slow down the decoding process, rendering the methods unusable in real-world applications.\nIn this paper, we propose BOLT1, that uses a sequence of tunable Biases Over LogiTs of the PLM’s output layer, to steer the generation towards specified constraints. The biases are tuned through a gradient-based process, with the goal of minimizing the energy of the generated sequences. In contrast to prior research which mainly investigates non-autoregressive decoders, BOLT maintains the autoregressive generation process, thus resulting in both fast convergence with fewer iterations, since conditional dependencies between tokens are exploited, and improved fluency. Fig. 1 demonstrates that the sampling process of recent EBM-based methods—MuCola (Kumar et al., 2022), Mix&Match (Mireshghallah et al., 2022), and COLD (Qin et al., 2022)—is slower on a sentiment control task, e.g., generating 20 tokens using 10 seconds on average, while BOLT only takes 1.4 seconds.\nWe conduct controlled generation experiments over three tasks: sentiment control, toxicity avoidance, and keyword-guided topic control, encompassing both soft and hard constraint-based generation problems. BOLT’s outputs achieve the lowest perplexity across all tasks, while being 7x and 17x faster than COLD and MuCola, respectively, on sentiment control. Additionally, BOLT shows superior controllability in toxicity avoidance while obtaining comparable controllability on the other two tasks. Lastly, according to human evaluation, 74.4% and 51.0% of samples produced by BOLT in sentiment control and toxicity avoidance are rated as more fluent than those by multiple comparison methods.",
    "conclusion": "We introduce BOLT, an energy-based model for controlled text generation. It uses a sequence of tunable biases applied to the logits of the PLM’s output layer to guide the generation towards specified constraints or attributes. Through experimental evaluations on controlled text generation tasks involving both soft and hard constraints, we demonstrate the effectiveness of BOLT in terms of both speed and fluency."
  },
  {
    "sample_id": 99,
    "title": "BRAINTEASER: Lateral Thinking Puzzles for Large Language Models",
    "abstract": "The success of language models has inspired the NLP community to attend to tasks that require implicit and complex reasoning, relying on human-like commonsense mechanisms. While such vertical thinking tasks have been relatively popular, lateral thinking puzzles have received little attention. To bridge this gap, we devise BRAINTEASER: a multiple-choice Question Answering task designed to test the model’s ability to exhibit lateral thinking and defy default commonsense associations. We design a three-step procedure for creating the first lateral thinking benchmark, consisting of data collection, distractor generation, and generation of reconstruction examples, leading to 1,100 puzzles with high-quality annotations. To assess the consistency of lateral reasoning by models, we enrich BRAINTEASER based on a semantic and contextual reconstruction of its questions. Our experiments with state-ofthe-art instruction- and commonsense language models reveal a significant gap between human and model performance, which is further widened when consistency across reconstruction formats is considered. We make all of our code and data available to stimulate work on developing and evaluating lateral thinking models.",
    "introduction": "Human reasoning processes comprise two types of thinking: vertical and lateral (Waks, 1997). Vertical thinking, also known as linear, convergent, or logical thinking, is a sequential analytical process that is based on rationality, logic, and rules, typically associated with the left-brain hemisphere. Vertical thinking, as illustrated in Figure 1 (top), is needed to create a reasoning path from flooding a room to filling it with water for physical reasoning, and from inanimate objects with five fingers to gloves in riddles. Meanwhile, lateral thinking (or “thinking\nFigure 1: Contrasting existing Vertical Thinking tasks (PIQA (Bisk et al., 2020) and RiddleSense (Lin et al., 2021)) to our novel lateral thinking task called BRAINTEASER. While prior tasks require commonsense to be injected, BRAINTEASER’s lateral thinking puzzles require default commonsense thinking to be deprecated.\noutside the box”) is a divergent and creative process that involves looking at a problem from a new perspective and defying preconceptions, associated with the right-brain hemisphere (De Bono, 1970; Waks, 1997). Lateral thinking is required to solve the puzzle in Figure 1 (bottom), by overwriting the commonsense associations of man shaves to he shaves himself, and regarding the man as somebody who shaves others all day (e.g., a barber).\nThe development of natural language processing (NLP) models and their evaluation has achieved much progress in vertical thinking. In particular, large language models (LLMs) (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020b) have achieved strong performance across a variety of complex reasoning tasks (Talmor et al., 2019; Bisk et al., 2020; Sap et al., 2019b), even with the complete absence (zero-shot) (Sanh et al., 2022) or limited provision (few-shot) of training time exemplars (Chung et al., 2022).1 To perform well on tasks such as reasoning over physical interactions (Bisk et al., 2020) and social implications (Sap et al., 2019b), LLMs exhibit better vertical thinking capabilities, including commonsense association (Wei et al., 2022) and inference ability (Bosselut et al., 2019). While the extent to which these models possess common sense is heavily discussed (Marcus, 2022; Bubeck et al., 2023; Wei et al., 2023), we note that prior work has not considered the lateral thinking ability of LLMs. Creative thinking problems in benchmarks and knowledge bases are often filtered out as noise during preprocessing (Vajjala and Meurers, 2012; Speer et al., 2017; Sap et al., 2019a), and only kept if their resolution can be supported by commonsense associations, as in the case of riddles (Figure 1) (Lin et al., 2021; Gao et al., 2018). As many situations are novel, we expect that lateral thinking puzzles like those in Figure 1-bottom will be hindered by default commonsense associations and cannot be easily solved by further adaptation and scaling of the existing LLM methods.\nTo bridge this gap, we propose to study the ability of state-of-the-art LLMs to reason on lateral thinking puzzles. We formulate lateral thinking puzzles as multiple-choice Question Answering (QA) tasks, making them intuitive to answer by humans and easy to evaluate automatically. Following our task definition, we create a novel BRAINTEASER benchmark with two tasks of different granularity: Sentence Puzzles and Word Puzzles (cf. Figure 1). To construct the dataset, we design a data collection procedure, which crawls relevant puzzles from several publicly available websites, performs semiautomatic filtering of irrelevant question categories (e.g., pun, dad jokes), and ensures high data quality. To ensure fair and informative questions, we construct distractors semi-automatically by manual annotation of the explicit and implicit (commonsense) premises that arise from each puzzle. To address concerns of possible LLM memorization (Carlini et al., 2022) and their lack of consistency (Goldberg, 2023), we enrich BRAINTEASER with two reconstruction strategies: semantic reconstruction and context reconstruction, which create variants of each puzzle without changing its original way of defying default commonsense associations. This systematic procedure results in a novel BRAINTEASER benchmark with 1.1K high-quality data points and nearly 100% human evaluation results. Using BRAINTEASER as the benchmark, we conduct comprehensive experiments involving different model structures, model sizes, and prompting strategies. The results reveal a huge gap between human performance and current LLMs, indicating the great need to improve lateral thinking in LLMs.\nWe summarize our contributions as follows: 1) We introduce lateral thinking puzzles, a multiplechoice QA task designed to test the model’s ability to exhibit lateral thinking and defy default commonsense associations. 2) We design a three-step procedure for creating the first lateral thinking benchmark, BRAINTEASER, consisting of data collection, distractor generation, and generation of reconstruction examples, leading to 1,100 highquality puzzles. 3) We conduct comprehensive experiments with state-of-the-art LLMs. We make all of our code and data available to stimulate work on developing and evaluating lateral thinking models.2",
    "conclusion": "We defined the task of lateral thinking for LLMs, formulated as a multiple-choice QA with a sentence- and word-level puzzles. We developed BRAINTEASER, a 1.1K lateral thinking benchmark that combines original puzzles and their reconstruction variants. Our experiments showed that ChatGPT’s performance on this task is halfway between random and humans, whereas other models often perform close to random. While scaling up model size improved performance, enriching with common sense or providing few-shot demonstrations yielded limited benefits. Meanwhile, all models tend to solve the variants of the same puzzle inconsistently. Our error analysis showed that the models’ lateral thinking is often hindered by memorization and misleading commonsense associations. In the future, we intend to develop lateral thinking models, create additional lateral thinking evaluation tasks (e.g., relating to alteration (De Bono, 1970)), and investigate flexible ways to combine lateral and vertical thinking."
  }
]