Vi. How Does The Pre-Training Escalate Target Task Performance?

To analyze the representations in neural networks consisting of different weights, we employed Central Kernel Alignment (CKA) #REFR as an index of measuring similarity between two representations from different neural networks.

Throughout experimental analyses in section IV and V-B, we discovered pre-training methods concretely contribute to the escalated target task performance in every image recognition task and problem settings.

Based on the aforementioned discoveries, we aim to scrutinize an underlying reason behind the effectiveness of pre-training by analyzing learned knowledge at the neural networks.

Suppose we provide image samples into two neural networks (N 1 , N 2 ) trained from different pre-trained weights.

Then, we can extract a pair of representation vectors from any layers of N 1 and N 2 denoted as (R 1 , R 2 ).

Given these representation vectors R 1 and R 2 , CKA effectively measures the similarity between layers in the same neural networks with different weights and across entirely different architectures.

The CKA yields a similarity metric lying between 0 to 1, where 0 implies less similarity and 1 illustrates high similarity.

Due to its convenience and effectiveness in measuring similarity between two representations, we utilized it.

Discovering The Effectiveness Of Pre-Training In A Large-Scale Car-Sharing Platform

Abstract: Recent progress of deep learning has empowered various intelligent transportation applications, especially in carsharing platforms. While the traditional operations of the carsharing service highly relied on human engagements in fleet management, modern car-sharing platforms let users upload car images before and after their use to inspect the cars without a physical visit. To automate the aforementioned inspection task, prior approaches utilized deep neural networks. They commonly employed pre-training, a de-facto technique to establish an effective model under the limited number of labeled datasets. As candidate practitioners who deal with car images would presumably get suffered from the lack of a labeled dataset, we analyzed a sophisticated analogy into the effectiveness of pretraining is important. However, prior studies primarily shed a little spotlight on the effectiveness of pre-training. Motivated by the aforementioned lack of analysis, our study proposes a series of analyses to unveil the effectiveness of various pre-training methods in image recognition tasks at the car-sharing platform. We set two real-world image recognition tasks in the car-sharing platform in a live service, established them under the many-shot and few-shot problem settings, and scrutinized which pre-training method accomplishes the most effective performance in which setting. Furthermore, we analyzed how does the pre-training and fine-tuning convey different knowledge to the neural networks for a precise understanding.

Similarity Of Neural Network Representations Revisited

Abstract: Recent work has sought to understand the behavior of neural networks by comparing representations between layers and between different trained models. We examine methods for comparing neural network representations based on canonical correlation analysis (CCA). We show that CCA belongs to a family of statistics for measuring multivariate similarity, but that neither CCA nor any other statistic that is invariant to invertible linear transformation can measure meaningful similarities between representations of higher dimension than the number of data points. We introduce a similarity index that measures the relationship between representational similarity matrices and does not suffer from this limitation. This similarity index is equivalent to centered kernel alignment (CKA) and is also closely connected to CCA. Unlike CCA, CKA can reliably identify correspondences between representations in networks trained from different initializations.

neural networks
Central Kernel Alignment


It has been shown, both in practice and in theory #REFR , to provide an advantage over fixed parameter strategies.

It is surprising that an evolutionary algorithm on bit vectors similar to CMA-ES is missing.

One may object to this observation that PBIL is a natural candidate.

However, its parameters are all continuous and, in the form of a probability vector, do not belong to the search space in the same way as, in CMA-ES, the mean of the search distribution does.

Furthermore, it is expected that such a speculative evolutionary algorithm will be able to control the mutation rate, just as CMA-ES provides an optimal control of correlated mutations.

Adaptive parameter control in evolutionary algorithms has been the subject of sustained efforts for decades #OTHEREFR .

However, update rules are often introduced as heuristics and do not derive from first principles.

A famous update rule is the so called one-fifth success rule [25] which adjusts the mutation rate so as to keep the success rate of mutations equal or close to one fifth (a mutation is successful if it produces an offspring of increased fitness).

If the success rate is larger (smaller) than one fifth then the mutation rate should be increased (decreased).

The one-fifth constant can be derived from theoretical considerations about the (1 + 1) evolution strategy applied to the sphere problem in Euclidean spaces.

It has been proved that the (1 + 1) evolutionary algorithm equipped with a similar success-based update rule achieves the same performance on LeadingOnes as the (1+1) EA with an optimal fitness-dependent mutation rate #OTHEREFR .

An Information-Theoretic Evolutionary Algorithm *

Abstract: We propose a novel evolutionary algorithm on bit vectors which derives from the principles of information theory. The informationtheoretic evolutionary algorithm (it-EA) iteratively updates a search distribution with two parameters, the center, that is the bit vector at which standard bit mutation is applied, and the mutation rate. The mutation rate is updated by means of information-geometric optimization and the center is updated by means of a maximum likelihood principle. Standard elitist and non elitist updates of the center are also considered. Experiments illustrate the dynamics of the mutation rate and the influence of hyperparameters. In an empirical runtime analysis, on OneMax and LeadingOnes, the elitist and non elitist it-EAs obtain promising results.

Theory Of Parameter Control For Discrete Black-Box Optimization: Provable Performance Gains Through Dynamic Parameter Choices

Abstract: Parameter control is aimed at realizing performance gains through a dynamic choice of the parameters which determine the behavior of the underlying optimization algorithm. In the context of evolutionary algorithms, this research line has for a long time been dominated by empirical approaches. With the significant advances in running-time analysis achieved in the last ten years, the parameter control question has become accessible to theoretical investigations. A number of running-time results for a broad range of different parameter control mechanisms have been obtained in recent years. This chapter surveys these results, and puts them into context by proposing an updated classification scheme for parameter control.


From a gender point of view, our results converge with previous research that reported that male scholars have a greater propensity to self-cite than female counterparts #REFR .

These 'mechanical' factors affecting self-citations suggest that researchers from different cohorts cannot be held to the same standards in terms of self-citations.

Third, at the individual researcher level, our results show that direct self-citations have, on average, little effect on the h-index, but can have a considerable impact for researchers that have high rates of self-citations.

This suggests that, while self-citations are, in most cases, not affecting the reliability of citation analysis for assessing research impact, some researchers with unusually high levels of self-citations do.

Fourth, it provides evidence that self-citations are correlated with external citations.

That is, researchers who have high levels of self-citations are also highly cited by their colleagues. Women are also self-citing their work less than men.

These results are congruent with studies that show that women are relatively less prone to engage in selfpromoting behavior at work #OTHEREFR , and that women tend to face relatively greater social penalties for self-promotion than men #OTHEREFR .

In line with those, our results suggest that women require a greater threshold of relevancy in order to self-cite.

This "higher bar" results in lower self-citation rates for women vis-à-vis men.

Finally, or results show that self-citations are more highly related to the citing paper than external citations, which suggest that, in most cases, self-citations are a normal feature of knowledge accumulation.

However, such finding can be nuanced by the inverse relationship observed between text similarity and the extent of researchers' self-refencing practices: the higher researchers' percentage of self-references, the lower the similarity between citing and cited articles.

Title: Are Self-Citations A Normal Feature Of Knowledge Accumulation?

Abstract: Science is a cumulative activity, which can manifest itself through the act of citing. Citations are also central to research evaluation, thus creating incentives for researchers to cite their own work. Using a dataset containing more than 63 million articles and 51 million disambiguated authors, this paper examines the relative importance of self-citations and selfreferences in the scholarly communication landscape, their relationship with the age and gender of authors, as well as their effects on various research evaluation indicators. Results show that self-citations and self-references evolve in different directions throughout researchers' careers, and that men and older researchers are more likely to self-cite. Although self-citations have, on average, a small to moderate effect on author's citation rates, they highly inflate citations for a subset of researchers. Comparison of the abstracts of cited and citing papers to assess the relatedness of different types of citations shows that self-citations are more similar to each other than other types of citations, and therefore more relevant. However, researchers that selfreference more tend to include less relevant citations. The paper concludes with a discussion of the role of self-citations in scholarly communication. One-Sentence Summary: This study provides evidence of career and gender effects in selfcitations, and of a higher similarity of citing and cited papers in the case of self-citations than external citations. Main Text: Citation analysis has been used in research evaluation for almost five decades (1-3). What started as a tool to help researchers and librarians find relevant literature more efficiently (4-6) slowly became, after the creation Science Citation Index in 1963, a means to assess research and various levels, from individuals to institutions and countries (7). In this context, citation analysis has come under scrutiny from researchers across all disciplines. Several authors criticized bibliometrics and citation analysis for their limitations (8). Those limitations can be divided into those that relate to the coverage of the database (9-11), accuracy of citation data (12), adverse effects (13-16), overabundance of indicators (17), and citations being questionable indicators of research impact (18-19). Literature on citation analysis has highlighted the diversity of roles of citations in scholarly papers. Based on papers published in high-energy physics, the classic study by Moravcsik and Murugesan (20) provides a typology of functions of citations, based on four non-exclusive dichotomies. Citations can be conceptual, related to theories or concepts contained in the cited

Men Set Their Own Cites High: Gender And Self-Citation Across Fields And Over Time

Abstract: How common is self-citation in scholarly publication, and does the practice vary by gender? Using novel methods and a data set of 1.5 million research papers in the scholarly database JSTOR published between 1779 and 2011, the authors find that nearly 10 percent of references are self-citations by a paper’s authors. The findings also show that between 1779 and 2011, men cited their own papers 56 percent more than did women. In the last two decades of data, men self-cited 70 percent more than women. Women are also more than 10 percentage points more likely than men to not cite their own previous work at all. While these patterns could result from differences in the number of papers that men and women authors have published rather than gender-specific patterns of self-citation behavior, this gender gap in self-citation rates has remained stable over the last 50 years, despite increased representation of women in academia. The authors break down self-citation patterns by academic field and number of authors ...


In order to avoid the "unreliable tail" of the distribution we use Nucleus Sampling (topp) #REFR .

PaSST has recently achieved state-of-the-art performance in audio classification tasks #OTHEREFR .

Using a pre-trained PaSST model, we infer AudioSet class labels from the input audio.

Each word in the label is embedded in the input space using trainable embeddings and concatenated with the extracted patches.

Similarly to #OTHEREFR , in order to make our system more robust to PaSST's prediction errors, we sample each label from the output distribution.

During inference, we select the most probable output label instead of sampling.

Since we add word-level information to our model, we want the input label to be semantically similar to the ground truth caption, functioning as a guiding text.

We observe that PaSST tends to output labels that capture the general, high-level information in the audio and not labels that are more infrequent and specific.

Such labels are more likely to be semantically similar or even be present verbatim in the ground truth captions.

For example, an audio clip with the caption "A short distance away, a group of people engage in indistinguishable chatter." is classified as Speech when in fact the AudioSet class Chatter would have higher semantic accuracy.

Efficient Audio Captioning Transformer With Patchout And Text Guidance

Abstract: Automated audio captioning is multi-modal translation task that aim to generate textual descriptions for a given audio clip. In this paper we propose a full Transformer architecture that utilizes Patchout as proposed in [1], significantly reducing the computational complexity and avoiding overfitting. The caption generation is partly conditioned on textual AudioSet tags extracted by a pre-trained classification model which is fine-tuned to maximize the semantic similarity between AudioSet labels and ground truth captions. To mitigate the data scarcity problem of Automated Audio Captioning we introduce transfer learning from an upstream audio-related task and an enlarged in-domain dataset. Moreover, we propose a method to apply Mixup augmentation for AAC. Ablation studies are carried out to investigate how Patchout and text guidance contribute to the final performance. The results show that the proposed techniques improve the performance of our system and while reducing the computational complexity. Our proposed method received the Judges Award at the Task6A of DCASE Challenge 2022.

The Curious Case Of Neural Text Degeneration

Abstract: Despite considerable advances in neural language modeling, it remains an open question what the best decoding strategy is for text generation from a language model (e.g. to generate a story). The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration — output text that is bland, incoherent, or gets stuck in repetitive loops. To address this we propose Nucleus Sampling, a simple but effective method to draw considerably higher quality text out of neural language models. Our approach avoids text degeneration by truncating the unreliable tail of the probability distribution, sampling from the dynamic nucleus of tokens containing the vast majority of the probability mass. To properly examine current maximization-based and stochastic decoding methods, we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition. Our results show that (1) maximization is an inappropriate decoding objective for open-ended text generation, (2) the probability distributions of the best current language models have an unreliable tail which needs to be truncated during generation and (3) Nucleus Sampling is the best decoding strategy for generating long-form text that is both high-quality — as measured by human evaluation — and as diverse as human-written text.


These methods are based on the clever solution approximations that result in a time-stepping scheme requiring only a small number of previous solution states for the next state evaluation. With some exceptions ( e.g. #REFR ), these methods are also O(h p ).

The methods from the first class are sequential in nature and have algebraic convergence order that typically does not exceed 2, even for the multi-step methods #OTHEREFR , because of the intrinsic fractional-kernel singularity #OTHEREFR .

In addition, at each time-step these methods need to query the entire solution history in order to evaluate ∂ α t or J α , numerically.

In the consequence of that, they are computationally costly and memory constrained.

Nonetheless, the methods from this class are popular due to their simplicity, numerical stability #OTHEREFR and the ability to handle non-smooth initial data #OTHEREFR .

The second class of numerical methods represented by the works #OTHEREFR , to name a few.

Spectral methods from #OTHEREFR deserve a separate mention.

Although formally belonging to the second class, they make use of the exponentially convergent contour based propagator approximation, which permits to evaluate the transient component of the solution to linear problem without time-stepping.

Authors of these works, however, do not apply it to (1.1), (1.4) directly.

Instead they consider a special proxy problem ∂ t u + I 1−α Au = g where I α is a non-local operator equal to ∂ α t , if α < 1, or to J α , otherwise.

It was shown in #OTHEREFR , that the existing methodology for parabolic problems #OTHEREFR can be transferred to the mild solution of such proxy problem with all important numerical features of the solution algorithms preserved, including uniform exponential convergence for t ∈ [0, T ] and the capacity for multi-level parallelism.

Exponentially Convergent Numerical Method For Abstract Cauchy Problem With Fractional Derivative Of Caputo Type

Abstract: We present an exponentially convergent numerical method to approximate the solution of the Cauchy problem for the inhomogeneous fractional differential equation with an unbounded operator coefficient and Caputo fractional derivative in time. The numerical method is based on the newly obtained solution formula that consolidates the mild solution representations of sub-parabolic, parabolic and sub-hyperbolic equations with sectorial operator coefficient A and non-zero initial data. The involved integral operators are approximated using the sinc-quadrature formulas that are tailored to the spectral parameters of A, fractional order α and the smoothness of the first initial condition, as well as to the properties of the equation's right-hand side f (t). The resulting method possesses exponential convergence for positive sectorial A, any finite t, including t = 0, and the whole range α ∈ (0, 2). It is suitable for a practically important case, when no knowledge of f (t) is available outside the considered interval t ∈ [0, T ]. The algorithm of the method is capable of multi-level parallelism. We provide numerical examples that confirm the theoretical error estimates.

Efficient Multistep Methods For Tempered Fractional Calculus: Algorithms And Simulations

Abstract: In this work, we extend the fractional linear multistep methods in [C. Lubich, SIAM J. Math. Anal., 17 (1986), pp.704--719] to the tempered fractional integral and derivative operators in the sense that the tempered fractional derivative operator is interpreted in terms of the Hadamard finite-part integral. We develop two fast methods, Fast Method I and Fast Method II, with linear complexity to calculate the discrete convolution for the approximation of the (tempered) fractional operator. Fast Method I is based on a local approximation for the contour integral that represents the convolution weight. Fast Method II is based on a globally uniform approximation of the trapezoidal rule for the integral on the real line. Both methods are efficient, but numerical experimentation reveals that Fast Method II outperforms Fast Method I in terms of accuracy, efficiency, and coding simplicity. The memory requirement and computational cost of Fast Method II are $O(Q)$ and $O(Qn_T)$, respectively, where $n_T$ is the number of the final time steps and $Q$ is the number of quadrature points used in the trapezoidal rule. The effectiveness of the fast methods is verified through a series of numerical examples for long-time integration, including a numerical study of a fractional reaction-diffusion model.


In particular, Menda and Watrous #REFR showed that for oracle A, a problem L A in QSZK A if there exists a reduction from L A to QSD A . Theorem 3.6 ([44, Theorem 1]).

The completeness follows from the fact that the states's trace distance is negligibly close to one, and this implies there exists a measurement that perfectly distinguishes the states.

For the soundness, since the states are negligibly close in trace distance, the prover does not succeed with non-negligibly advantage over random guessing.

To show the protocol is zero-knowledge, the quantum simulator applies the verifier's quantum operation first.

After receiving the reponse b ′ from the prover, it sets b ′ = b.

The QSZK-completeness of QSD relativizes.

For alphabet Σ, Γ, let L ⊆ Γ * be a language and A ⊆ Σ * be an oracle.

The language L A is contained in QSZK A if and only if there exists a polynomial-time uniform family of pairs of relativized quantum circuits (Q A 0 , Q A 1 ) with the following properties:

• If x ∈ L A , then (Q A 0 , Q A 1 ) ∈ QSD A 1 .

• If x / ∈ L A , then (Q A 0 , Q A 1 ) ∈ QSD A 0 .

Ben-David and Kothari #OTHEREFR studied independently the so-called QSZK complexity of function f , denoted QSZK(f ), which is defined as the minimum number k made by a pair of query algorithms A, B given oracle access to x such that for every x such that (i) if f (x) = 1, then A x − B x tr ≥ 2/3, and (ii) if f (x) = 0, then A x − B x tr ≤ 1/3.

Certified Randomness From Quantum Supremacy

Abstract: We propose an application for near-term quantum devices: namely, generating cryptographically certified random bits, to use (for example) in proof-of-stake cryptocurrencies. Our protocol repurposes the existing "quantum supremacy" experiments, based on random circuit sampling, that Google and USTC have successfully carried out starting in 2019. We show that, whenever the outputs of these experiments pass the now-standard Linear Cross-Entropy Benchmark (LXEB), under plausible hardness assumptions they necessarily contain Ω(n) min-entropy, where n is the number of qubits. To achieve a net gain in randomness, we use a small random seed to produce pseudorandom challenge circuits. In response to the challenge circuits, the quantum computer generates output strings that, after verification, can then be fed into a randomness extractor to produce certified nearly-uniform bits-thereby "bootstrapping" from pseudorandomness to genuine randomness. We prove our protocol sound in two senses: (i) under a hardness assumption called Long List Quantum Supremacy Verification, which we justify in the random oracle model, and (ii) unconditionally in the random oracle model against an eavesdropper who could share arbitrary entanglement with the device. (Note that our protocol's output is unpredictable even to a computationally unbounded adversary who can see the random oracle.) Currently, the central drawback of our protocol is the exponential cost of verification, which in practice will limit its implementation to at most n ∼ 60 qubits, a regime where attacks are expensive but not impossible. Modulo that drawback, our protocol appears to be the only practical application of quantum computing that both requires a QC and is physically realizable today.

Oracle Separations For Quantum Statistical Zero-Knowledge

Abstract: This paper investigates the power of quantum statistical zero knowledge interactive proof systems in the relativized setting. We prove the existence of an oracle relative to which quantum statistical zero-knowledge does not contain UP intersect coUP, and we prove that quantum statistical zero knowledge does not contain UP relative to a random oracle with probability 1. Our proofs of these statements rely on a bound on output state discrimination for relativized quantum circuits based on the quantum adversary method of Ambainis, following a technique similar to one used by Ben-David and Kothari to prove limitations on a query complexity variant of quantum statistical zero-knowledge.


The concatenated key-points of stage-one X #REFR 1 and X 1 2 are then fed .After the loop in 1, the green, yellow, and red cluster will be gathered, while the pink one and the blue one will be rejected.

As shown in Figure 4 , The two-stages pipeline support plugging any image matching models or recipe of models into both stages, such that the models set for stage 1 is

Each of the models in stage-one produce a set of key-points for both images.

If the pink one is included, more areas without useful information will be included, making rejected the pink cluster makes sense.

Based on the two images I 1 and I 2 and the key-points of stage-one X 1 1 and X #OTHEREFR 2 , The MKPC generates the cropped critical regions for both images (I 1 and I 2 ).

With the cropped regions of both images, each of the models in stage-two outputs the matched key-points for stage-two (X 2 1 and X 2 2 ).

Afterwards, the models in stage-two matches on the cropped area, producing the key-points for the second stage X 2 1 and X 2 2 . Concatenating the key-points of both stages, i.e.

Efficient And Accurate Co-Visible Region Localization With Matching Key-Points Crop (Mkpc): A Two-Stage Pipeline For Enhancing Image Matching Performance

Abstract: Image matching is a classic and fundamental task in computer vision. In this paper, under the hypothesis that the areas outside the co-visible regions carry little information, we propose a matching key-points crop (MKPC) algorithm. The MKPC locates, proposes and crops the critical regions, which are the co-visible areas with great efficiency and accuracy. Furthermore, building upon MKPC, we propose a general two-stage pipeline for image matching, which is compatible to any image matching models or combinations. We experimented with plugging SuperPoint + SuperGlue into the two-stage pipeline, whose results show that our method enhances the performance for outdoor pose estimations. What's more, in a fair comparative condition, our method outperforms the SOTA on Image Matching Challenge 2022 Benchmark, which represents the hardest outdoor benchmark of image matching currently. * denotes contributing equally to this work. Preprint. Under review.

Superpoint: Self-Supervised Interest Point Detection And Description

Abstract: This paper presents a self-supervised framework for training interest point detectors and descriptors suitable for a large number of multiple-view geometry problems in computer vision. As opposed to patch-based neural networks, our fully-convolutional model operates on full-sized images and jointly computes pixel-level interest point locations and associated descriptors in one forward pass. We introduce Homographic Adaptation, a multi-scale, multi-homography approach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g., synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able to repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other traditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches when compared to LIFT, SIFT and ORB.


Models based on the T5 #REFR model architecture have achieved state-of-the-art results in various natural language processing challenges and can be fine-tuned for specific tasks.

One of the applications of T5 can be lemmatization, the process of reducing a word or phrase to its basic form (lemma).

In Slavic languages such as Polish, Czech and Russian, lemmatization is particularly important due to the complex inflection of these languages.

We approached the lemmatization task as a textto-text problem.

The input to the model is an inflected phrase or named entity, which can consist of several word forms.

For example, it can consist of nouns in singular or plural form, or verbs in different tenses.

Exploring The Use Of Foundation Models For Named Entity Recognition And Lemmatization Tasks In Slavic Languages

Abstract: This paper describes Adam Mickiewicz University's (AMU) solution for the 4th Shared Task on SlavNER. The task involves the identification, categorization, and lemmatization of named entities in Slavic languages. Our approach involved exploring the use of foundation models for these tasks. In particular, we used models based on the popular BERT and T5 model architectures. Additionally, we used external datasets to further improve the quality of our models. Our solution obtained promising results, achieving high metrics scores in both tasks. We describe our approach and the results of our experiments in detail, showing that the method is effective for NER and lemmatization in Slavic languages.

Exploring The Limits Of Transfer Learning With A Unified Text-To-Text Transformer

Abstract: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.


However, these designs suffer from high energy consumption and low computation resource utilization because of temporal data dependencies. #REFR Lack of parallelism between GNN and RNN.

Integrated DGNN GCRN-M2 #OTHEREFR , GC-LSTM #OTHEREFR LRGCN #OTHEREFR , RE-Net #OTHEREFR • Data dependencies between GNN and RNN in adjacent time steps.

• Dependent GNN at different time steps.

Weights-evolved DGNN EvolveGCN #OTHEREFR • Weights for GNN are evolved by RNN.

• Independent GNN at different time steps.

However, there still remain some challenges on DGNN hardware deployment. #OTHEREFR High energy consumption and low computation resource utilization. Previous works primarily focus on deploying DGNNs on GPUs.

Previous research focuses on treating GNN and RNN as separate parts, which limits parallelism.

#OTHEREFR Lack of integrating GNN and RNN optimizations together into a single system.

Previous research usually optimizes GNN and RNN individually, which limits achieving optimal hardware efficiency.

Dgnn-Booster: A Generic Fpga Accelerator Framework For Dynamic Graph Neural Network Inference

Abstract: Dynamic Graph Neural Networks (DGNNs) are becoming increasingly popular due to their effectiveness in analyzing and predicting the evolution of complex interconnected graph-based systems. However, hardware deployment of DGNNs still remains a challenge. First, DGNNs do not fully utilize hardware resources because temporal data dependencies cause low hardware parallelism. Additionally, there is currently a lack of generic DGNN hardware accelerator frameworks, and existing GNN accelerator frameworks have limited ability to handle dynamic graphs with changing topologies and node features. To address the aforementioned challenges, in this paper, we propose DGNN-Booster, which is a novel Field-Programmable Gate Array (FPGA) accelerator framework for real-time DGNN inference using High-Level Synthesis (HLS). It includes two different FPGA accelerator designs with different dataflows that can support the most widely used DGNNs. We showcase the effectiveness of our designs by implementing and evaluating two representative DGNN models on ZCU102 board and measuring the end-to-end performance. The experiment results demonstrate that DGNN-Booster can achieve a speedup of up to 5.6× compared to the CPU baseline (6226R), 8.4× compared to the GPU baseline (A6000) and 2.1× compared to the FPGA baseline without applying optimizations proposed in this paper. Moreover, DGNN-Booster can achieve over 100× and over 1000× runtime energy efficiency than the CPU and GPU baseline respectively. Our implementation code and on-board measurements are publicly

Neural Message Passing For Quantum Chemistry

Abstract: Supervised learning on molecules has incredible potential to be useful in chemistry, drug discovery, and materials science. Luckily, several promising and closely related neural network models invariant to molecular symmetries have already been described in the literature. These models learn a message passing algorithm and aggregation procedure to compute a function of their entire input graph. At this point, the next step is to find a particularly effective variant of this general approach and apply it to chemical prediction benchmarks until we either solve them or reach the limits of the approach. In this paper, we reformulate existing models into a single common framework we call Message Passing Neural Networks (MPNNs) and explore additional novel variations within this framework. Using MPNNs we demonstrate state of the art results on an important molecular property prediction benchmark; these results are strong enough that we believe future work should focus on datasets with larger molecules or more accurate ground truth labels.


In this study, we extend the shareability network approach #REFR to quantify the potential ridesharing efficiency.

The shareability network transforms the spatiotemporal distribution of trips into a theoretic graph G = (V, E), where V is the set of trips and E the set of edges that indicates the shareability between trips.

Following the original work, we assume that at most two trips can be shared.

Two trips are shareable if serving them together with one vehicle can (1) generate total travel duration savings and (2) only incur an acceptable delay for the involved passengers.

Each trip x ∈ V can be represented as a vector (o

Quantifying The Uneven Efficiency Benefits Of Ridesharing Market Integration

Abstract: Ridesharing is recognized as one of the key pathways to sustainable urban mobility. With the emergence of Transportation Network Companies (TNCs) such as Uber and Lyft, the ridesharing market has become increasingly fragmented in many cities around the world, leading to efficiency loss and increased traffic congestion. While an integrated ridesharing market (allowing sharing across TNCs) can improve the overall efficiency, how such benefits may vary across TNCs based on actual market characteristics is still not well understood. In this study, we extend a shareability network framework to quantify and explain the efficiency benefits of ridesharing market integration using available TNC trip records. Through a case study in Manhattan, New York City, the proposed framework is applied to analyze a real-world ridesharing market with 3 TNCs-Uber, Lyft, and Via. It is estimated that a perfectly integrated market in Manhattan would improve ridesharing efficiency by 13.3%, or 5% of daily TNC vehicle hours traveled. Further analysis reveals that (1) the efficiency improvement is negatively correlated with the overall demand density and inter-TNC spatiotemporal unevenness (measured by network modularity), (2) market integration would generate a larger efficiency improvement in a competitive market, and (3) the TNC with a higher intra-TNC demand concentration (measured by clustering coefficient) would benefit less from market integration. As the uneven benefits may deter TNCs from collaboration, we also illustrate how to quantify each TNC's marginal contribution based on the Shapley value, which can be used to ensure equitable profit allocation. These results can help market regulators and business alliances to evaluate and monitor market efficiency and dynamically adjust their strategies, incentives, and profit allocation schemes to promote market integration and collaboration.

Quantifying The Benefits Of Vehicle Pooling With Shareability Networks

Abstract: Taxi services are a vital part of urban transportation, and a considerable contributor to traffic congestion and air pollution causing substantial adverse effects on human health. Sharing taxi trips is a possible way of reducing the negative impact of taxi services on cities, but this comes at the expense of passenger discomfort quantifiable in terms of a longer travel time. Due to computational challenges, taxi sharing has traditionally been approached on small scales, such as within airport perimeters, or with dynamical ad hoc heuristics. However, a mathematical framework for the systematic understanding of the tradeoff between collective benefits of sharing and individual passenger discomfort is lacking. Here we introduce the notion of shareability network, which allows us to model the collective benefits of sharing as a function of passenger inconvenience, and to efficiently compute optimal sharing strategies on massive datasets. We apply this framework to a dataset of millions of taxi trips taken in New York City, showing that with increasing but still relatively low passenger discomfort, cumulative trip length can be cut by 40% or more. This benefit comes with reductions in service cost, emissions, and with split fares, hinting toward a wide passenger acceptance of such a shared service. Simulation of a realistic online system demonstrates the feasibility of a shareable taxi service in New York City. Shareability as a function of trip density saturates fast, suggesting effectiveness of the taxi sharing system also in cities with much sparser taxi fleets or when willingness to share is low.


We used the HuggingFace implementation of the Flan-T5 model (google/ Flan-t5-base) with the SentencePiece tokenizer #REFR .

The distance between the resulting embeddings is measured using the cosine similarity.

Flan-T5 (2022) is a family of models based on T5 (2019) #OTHEREFR , an encoder-decoder transformer architecture trained on multiple language tasks.

The Flan-T5 models have undergone instruction-finetuning on over 1,800 language tasks, leading to a significant enhancement in their reasoning skills and promptability.

However, it is worth noting that the Flan-T5 models were not trained to perform spam detection tasks.

Our experimentation included the small version of the Flan-T5 model (80M parameters), but it demonstrated limited generalization capabilities, which is why it was excluded from this study.

The Flan-T5 model is a Seq2Seq model that is capable of generating textual outputs, as opposed to binary labels or probabilities.

To leverage the capabilities of this model for spam detection, we fine-tuned it as a new task, introducing a dedicated prefix of "classify as ham or spam:" to every sample.

As a result, the model was trained to correctly output either "ham" or "spam" based on the input text.

To obtain numerical values for classification metrics, a postprocessing step was utilized to map the textual labels to 0 and 1.

Spam-T5: Benchmarking Large Language Models For Few-Shot Email Spam Detection

Abstract: This paper investigates the effectiveness of large language models (LLMs) in email spam detection by comparing prominent models from three distinct families: BERT-like, Sentence Transformers, and Seq2Seq. Additionally, we examine well-established machine learning techniques for spam detection, such as Naïve Bayes and LightGBM, as baseline methods. We assess the performance of these models across four public datasets, utilizing different numbers of training samples (full training set and few-shot settings). Our findings reveal that, in the majority of cases, LLMs surpass the performance of the popular baseline techniques, particularly in few-shot scenarios. This adaptability renders LLMs uniquely suited to spam detection tasks, where labeled samples are limited in number and models require frequent updates. Additionally, we introduce Spam-T5, a Flan-T5 model that has been specifically adapted and fine-tuned for the purpose of detecting email spam. Our results demonstrate that Spam-T5 surpasses baseline models and other LLMs in the majority of scenarios, particularly when there are a limited number of training samples available. Our code is publicly available at https://github.com/jpmorganchase/emailspamdetection.

Sentencepiece: A Simple And Language Independent Subword Tokenizer And Detokenizer For Neural Text Processing

Abstract: This paper describes SentencePiece, a language-independent subword tokenizer and detokenizer designed for Neural-based text processing, including Neural Machine Translation. It provides open-source C++ and Python implementations for subword units. While existing subword segmentation tools assume that the input is pre-tokenized into word sequences, SentencePiece can train subword models directly from raw sentences, which allows us to make a purely end-to-end and language independent system. We perform a validation experiment of NMT on English-Japanese machine translation, and find that it is possible to achieve comparable accuracy to direct subword training from raw sentences. We also compare the performance of subword training and segmentation with various configurations. SentencePiece is available under the Apache 2 license at https://github.com/google/sentencepiece.


This is an active method based on the Follow-the-Leader algorithm #REFR , each validation iteration is an online learning example.

2) Direct Policy Learning: Direct Policy Learning (DPL), a training method based on BC, evaluates the current policy and then obtains more suitable training data for selfoptimization.

Compared with BC, the main advantage of DPL leverages expert trajectories to instruct the agent how to recover from current errors #OTHEREFR .

In this way, DPL alleviates the limitation of BC due to insufficient data.

In this section, we summarize a series of DPL methods. Ross et al.

#OTHEREFR construct a classical online IL method named Dataset Aggregation (DAgger) method.

The method modifies the main classifier or regressor on all state−action pairs experienced by the agent.

DAgger is a novel solution for sequential prediction problems, however, its learning efficiency might be suppressed by the far distance between policy space and learning space. In reply, He et al.

#OTHEREFR propose a DAgger by coaching algorithm which employs a coach to demonstrate easy-to-learn policies for the learner and the demonstrated policies gradually converge to label.

To better instruct the agent, the coach establishes a compromised policy which is not much worse than a ground truth control signal and much better than novice predicted action. As shown in Fig.

3 , π is the predicted command, π * shows the expert trajectory, and π presents the compromised trajectory.

Motion Planning For Autonomous Driving: The State Of The Art And Future Perspectives

Abstract: Thanks to the augmented convenience, safety advantages, and potential commercial value, Intelligent vehicles (IVs) have attracted wide attention throughout the world. Although a few autonomous driving unicorns assert that IVs will be commercially deployable by 2025, their implementation is still restricted to small-scale validation due to various issues, among which precise computation of control commands or trajectories by planning methods remains a prerequisite for IVs. This paper aims to review state-of-the-art planning methods, including pipeline planning and end-to-end planning methods. In terms of pipeline methods, a survey of selecting algorithms is provided along with a discussion of the expansion and optimization mechanisms, whereas in end-to-end methods, the training approaches and verification scenarios of driving tasks are points of concern. Experimental platforms are reviewed to facilitate readers in selecting suitable training and validation methods. Finally, the current challenges and future directions are discussed. The sideby-side comparison presented in this survey not only helps to gain insights into the strengths and limitations of the reviewed methods but also assists with system-level design choices. Index Terms-Pipeline planning, end-to-end planning, imitation learning, reinforcement learning, parallel learning. I. INTRODUCTION I NTELLIGENT vehicles (IVs) have gained considerable attention from government, industry, academia, and the

Global Overview Of Imitation Learning

Abstract: Imitation Learning is a sequential task where the learner tries to mimic an expert's action in order to achieve the best performance. Several algorithms have been proposed recently for this task. In this project, we aim at proposing a wide review of these algorithms, presenting their main features and comparing them on their performance and their regret bounds.

online learning example
Follow-the-Leader algorithm


Yet organisationfocused tools to provide information on points in the AI lifecycle (such as [40, #REFR ) are of limited help where information about interconnections between actors is needed.

Expanding the accountability horizon. The accountability horizon thus poses major problems for accountability.

Interventions are needed to help expand the accountability horizon and better place actors to (i) know more about their own supply chains, and (ii) support others in knowing more about theirs.

Alternatively, tracking data flow between actors could help understand interconnections beyond the first few steps #OTHEREFR , as could legal and institutional mechanisms requiring information about arrangements.

'Know your customer' requirements around customer on-boarding for AI services (common in financial services) could help providers understand customers' purposes and intentions [26] (though these may only give some visibility over one or two steps in the chain).

Moreover, recent CJEU data protection jurisprudence regarding transparency rights confirms that data subjects have the right to know the identity of any recipients of their personal data [21], which may help understand data flows.

However, where the data controller does not know the recipients' identity-which may be common due to the accountability horizon-data subjects can instead be told about the categories of recipients of the data [21] (significantly less useful information).

A particular difficulty, however, is that accountability is contextual #OTHEREFR .

Understanding Accountability In Algorithmic Supply Chains

Abstract: Academic and policy proposals on algorithmic accountability often seek to understand algorithmic systems in their socio-technical context, recognising that they are produced by 'many hands'. Increasingly, however, algorithmic systems are also produced, deployed, and used within a supply chain comprising multiple actors tied together by flows of data between them. In such cases, it is the working together of an algorithmic supply chain of different actors who contribute to the production, deployment, use, and functionality that drives systems and produces particular outcomes. We argue that algorithmic accountability discussions must consider supply chains and the difficult implications they raise for the governance and accountability of algorithmic systems. In doing so, we explore algorithmic supply chains, locating them in their broader technical and political economic context and identifying some key features that should be understood in future work on algorithmic governance and accountability (particularly regarding general purpose AI services). To highlight ways forward and areas warranting attention, we further discuss some implications raised by supply chains: challenges for allocating accountability stemming from distributed responsibility for systems between actors, limited visibility due to the accountability horizon, service models of use and liability, and cross-border supply chains and regulatory arbitrage. CCS CONCEPTS • Social and professional topics → Computing / technology policy; Socio-technical systems.

Model Cards For Model Reporting

Abstract: Trained machine learning models are increasingly used to perform high-impact tasks in areas such as law enforcement, medicine, education, and employment. In order to clarify the intended use cases of machine learning models and minimize their usage in contexts for which they are not well suited, we recommend that released models be accompanied by documentation detailing their performance characteristics. In this paper, we propose a framework that we call model cards, to encourage such transparent model reporting. Model cards are short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups (e.g., race, geographic location, sex, Fitzpatrick skin type [15]) and intersectional groups (e.g., age and race, or sex and Fitzpatrick skin type) that are relevant to the intended application domains. Model cards also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information. While we focus primarily on human-centered machine learning models in the application fields of computer vision and natural language processing, this framework can be used to document any trained machine learning model. To solidify the concept, we provide cards for two supervised models: One trained to detect smiling faces in images, and one trained to detect toxic comments in text. We propose model cards as a step towards the responsible democratization of machine learning and related artificial intelligence technology, increasing transparency into how well artificial intelligence technology works. We hope this work encourages those releasing trained machine learning models to accompany model releases with similar detailed evaluation numbers and other relevant documentation.


Transforming Problem P 1 Into Parametric Convex Optimization Problems

With ( * , * , * ) denoting a globally optimal solution to Problem P 2 (and hence ( * , * ) denoting a globally optimal solution to Problem P 1 ), clearly setting ( , ) as ( * , * ) of #REFR and (18) satisfies( , ) = 0.

With Lemma 5.1 presented above, we now describe how to solve Problem P 2 using P 3 ( , ).

Let [ # ( , ), # ( , )] denote a globally optimal solution to P 3 ( , ), where

(23) Based on the above, solving Problem P 2 and hence P 1 can be transformed into solving (23) to obtain P 3 ( * , * ), and then setting

, a globally optimal solution to P 3 ( * , * ), according to Lemma 5.1.

Based on the above idea, we present Algorithm 1 next, where it will become clear that solving P 1 becomes solving a series of parametric convex optimization P 3 ( ( ) , ( ) ), with denoting the iteration index.

(24) Readers may notice that our Lemma 5.1 provides just a necessary condition for a global optimum of Problem P 2 .

Optimizing Utility-Energy Efficiency For The Metaverse Over Wireless Networks Under Physical Layer Security

Abstract: The Metaverse, an emerging digital space, is expected to offer various services mirroring the real world. Wireless communications for mobile Metaverse users should be tailored to meet the following user characteristics: 1) emphasizing application-specific perceptual utility instead of simply the transmission rate, 2) concerned with energy efficiency due to the limited device battery and energy intensiveness of some applications, and 3) caring about security as the applications may involve sensitive personal data. To this end, this paper incorporates application-specific utility, energy efficiency, and physical-layer security (PLS) into the studied optimization in a wireless network for the Metaverse. Specifically, after introducing utility-energy efficiency (UEE) to represent each Metaverse user's application-specific objective under PLS, we formulate an optimization to maximize the network's weighted sum-UEE by deciding users' transmission powers and communication bandwidths. The formulated problem belongs to the sum-of-ratios optimization, for which prior studies have demonstrated its difficulty. Nevertheless, our proposed algorithm 1) obtains the global optimum for the weighted sum-UEE optimization, via a transform to parametric convex optimization problems, 2) applies to any utility function which is concave, increasing, and twice differentiable, and 3) achieves a linear time complexity in the number of users (the optimal complexity in the order sense). Simulations confirm the superiority of our algorithm over other approaches. We explain that our technique for solving the sum-of-ratios optimization is applicable to other optimization problems in wireless networks and mobile computing.

Throughput Optimal Policies For Energy Harvesting Wireless Transmitters With Non-Ideal Circuit Power

Abstract: Characterizing the fundamental tradeoffs for maximizing energy efficiency (EE) versus spectrum efficiency (SE) is a key problem in wireless communication. In this paper, we address this problem for a point-to-point additive white Gaussian noise (AWGN) channel with the transmitter powered solely via energy harvesting from the environment. In addition, we assume a practical on-off transmitter model with non-ideal circuit power, i.e., when the transmitter is on, its consumed power is the sum of the transmit power and a constant circuit power. Under this setup, we study the optimal transmit power allocation to maximize the average throughput over a finite horizon, subject to the time-varying energy constraint and the non-ideal circuit power consumption. First, we consider the off-line optimization under the assumption that the energy arrival time and amount are a priori known at the transmitter. Although this problem is non-convex due to the non-ideal circuit power, we show an efficient optimal solution that in general corresponds to a two-phase transmission: the first phase with an EE-maximizing on-off power allocation, and the second phase with a SE-maximizing power allocation that is non-decreasing over time, thus revealing an interesting result that both the EE and SE optimizations are unified in an energy harvesting communication system. We then extend the optimal off-line algorithm to the case with multiple parallel AWGN channels, based on the principle of nested optimization. Finally, inspired by the off-line optimal solution, we propose a new online algorithm under the practical setup with only the past and present energy state information (ESI) known at the transmitter.


The problem of detecting interactions between humans and objects was originally introduced in #REFR and has drawn immense attention in the computer vision community since then.

Most of the research efforts on this topic #OTHEREFR use a two-stage solution in which human/object locations are extracted along with their semantic labels by an off-the-shelf object detector first, and an interaction classification model is learnt on pairwise human-object features.

Apart from human/object appearances, there exist models that make use of contextual features #OTHEREFR , spatial layouts #OTHEREFR and human pose estimations #OTHEREFR .

Inspired by one-stage object detection efforts, researchers lately try to formulate end-to-end HOI detection approaches where human/instance localization and interaction classification are performed in parallel #OTHEREFR . These methods are analogous to CNN-based (e.g. YOLO #OTHEREFR ) and Transformer-based (e.g. DETR #OTHEREFR ) end-to-end object detectors.

PPDM #OTHEREFR takes a step forward and drops the need for heuristically created "anchors", formulating HOI detection as a point matching problem between human and object locations.

Regardless of being one-stage or two-stage, these methods rely on strong supervision which is costly to acquire.

Weakly-Supervised Hoi Detection From Interaction Labels Only And Language/Vision-Language Priors

Abstract: Human-object interaction (HOI) detection aims to extract interacting human-object pairs and their interaction categories from a given natural image. Even though the labeling effort required for building HOI detection datasets is inherently more extensive than for many other computer vision tasks, weakly-supervised directions in this area have not been sufficiently explored due to the difficulty of learning human-object interactions with weak supervision, rooted in the combinatorial nature of interactions over the object and predicate space. In this paper, we tackle HOI detection with the weakest supervision setting in the literature, using only image-level interaction labels, with the help of a pretrained vision-language model (VLM) and a large language model (LLM). We first propose an approach to prune non-interacting human and object proposals to increase the quality of positive pairs within the bag, exploiting the grounding capability of the vision-language model. Second, we use a large language model to query which interactions are possible between a human and a given object category, in order to force the model not to put emphasis on unlikely interactions. Lastly, we use an auxiliary weaklysupervised preposition prediction task to make our model explicitly reason about space. Extensive experiments and ablations show that all of our contributions increase HOI detection performance.

Visual Semantic Role Labeling

Abstract: In this paper we introduce the problem of Visual Semantic Role Labeling: given an image we want to detect people doing actions and localize the objects of interaction. Classical approaches to action recognition either study the task of action classification at the image or video clip level or at best produce a bounding box around the person doing the action. We believe such an output is inadequate and a complete understanding can only come when we are able to associate objects in the scene to the different semantic roles of the action. To enable progress towards this goal, we annotate a dataset of 16K people instances in 10K images with actions they are doing and associate objects in the scene with different semantic roles for each action. Finally, we provide a set of baseline algorithms for this task and analyze error modes providing directions for future work.


We considered RandLA-Net #REFR , one of the most famous point-based architectures as a reference, in order to frame the problem in a perspective different from images (i.e., using MLPs in place of convolutional setting).

RandLA-Net is an efficient point-based lightweight network composed of an MLP based encoder-decoder structure that achieves remarkably high efficiency in terms of memory and computation.

In addition, we evaluate on Cylin-der3D #OTHEREFR voxel based architecture for comparison.

Nonetheless, the framework can be applied on top of any architecture for point cloud semantic segmentation.

SemanticKITTI #OTHEREFR has been chosen as a reference dataset, since it is one of the most popular benchmarks for LiDAR semantic segmentation in autonomous driving.

Se-manticKITTI consists of 43, 552 densely annotated LiDAR scans, 19, 130 for training, and 4, 071 for validating (that we used for testing, as done by all competing works being the test labels not publicly available).

Continual Learning For Lidar Semantic Segmentation: Class-Incremental And Coarse-To-Fine Strategies On Sparse Data

Abstract: During the last few years, Continual Learning (CL) strategies for image classification and segmentation have been widely investigated designing innovative solutions to tackle catastrophic forgetting, like knowledge distillation and selfinpainting. However, the application of continual learning paradigms to point clouds is still unexplored and investigation is required, especially using architectures that capture the sparsity and uneven distribution of LiDAR data. The current paper analyzes the problem of class incremental learning applied to point cloud semantic segmentation, comparing approaches and state-of-the-art architectures. To the best of our knowledge, this is the first example of classincremental continual learning for LiDAR point cloud semantic segmentation. Different CL strategies were adapted to LiDAR point clouds and tested, tackling both classic finetuning scenarios and the Coarse-to-Fine learning paradigm. The framework has been evaluated through two different architectures on SemanticKITTI [2, 16], obtaining results in line with state-of-the-art CL strategies and standard offline learning.

Randla-Net: Efficient Semantic Segmentation Of Large-Scale Point Clouds

Abstract: We study the problem of efficient semantic segmentation for large-scale 3D point clouds. By relying on expensive sampling techniques or computationally heavy pre/post-processing steps, most existing approaches are only able to be trained and operate over small-scale point clouds. In this paper, we introduce RandLA-Net, an efficient and lightweight neural architecture to directly infer per-point semantics for large-scale point clouds. The key to our approach is to use random point sampling instead of more complex point selection approaches. Although remarkably computation and memory efficient, random sampling can discard key features by chance. To overcome this, we introduce a novel local feature aggregation module to progressively increase the receptive field for each 3D point, thereby effectively preserving geometric details. Extensive experiments show that our RandLA-Net can process 1 million points in a single pass with up to 200X faster than existing approaches. Moreover, our RandLA-Net clearly surpasses state-of-the-art approaches for semantic segmentation on two large-scale benchmarks Semantic3D and SemanticKITTI.

famous point-based architectures
RandLA-Net


We have the following result for the Gaussian RKHS, which parallels Maurer's result in #REFR Section 3.2] if we impose the same assumption that T is a projection of another RKHS function class onto a set of samples.

so our improvement consists in removing the logarithmic factor in front of the second term on the right-hand side.

Our second example involves functions in a Reproducing Kernel Hilbert Space (RKHS).

Due to space limitations, we can only give a brief sketch; the reader is invited to consult [3, Chaps. 2 and 4] for the background.

Let (H K , •, • K ) be an RKHS associated with a Mercer kernel K : X × X → R; then we consider F = I K (B K (̺)), where B K (̺) = {f ∈ H K : f K ≤ ̺} is the zero-centered closed ball of radius ̺ in H K and I K is the embedding map from H K into the space C(X) of continuous real-valued functions on X equipped with the uniform norm • X .

Consider the Gaussian kernel K(x, y) = exp − 1 2σ 2 x − y 2 2 , where σ 2 > 0 is the kernel bandwidth. Then, for any T

See the Remark following Theorem 2 for the motivation behind our choices of X and T .

A Chain Rule For The Expected Suprema Of Bernoulli Processes *

Abstract: We obtain an upper bound on the expected supremum of a Bernoulli process indexed by the image of an index set under a uniformly Lipschitz function class in terms of properties of the index set and the function class, extending an earlier result of Maurer for Gaussian processes. The proof makes essential use of recent results of Bednorz and Latała on the boundedness of Bernoulli processes. * This research was suppoorted in part by DARPA under the Learning with Less Labels (LwLL) program.

A Chain Rule For The Expected Suprema Of Gaussian Processes

Abstract: The expected supremum of a Gaussian process indexed by the image of an index set under a function class is bounded in terms of separate properties of the index set and the function class. The bound is relevant to the estimation of nonlinear transformations or the analysis of learning algorithms whenever hypotheses are chosen from composite classes, as is the case for multi-layer models.


To visualize the effect of feature decomposition, we adopt deep neural network interpretability methods in #REFR .

To illustrate each model's ability for vision tasks, we choose smiling, wavy hair, and attractive to form three binary classification tasks.

As shown in Figure 5, Fair-CDA achieves SOTA performance followed by two mixup methods.

It is worth mentioning that the DP and EO gap of these methods on the smiling recognition task is smaller compared with other tasks, which is a relatively fair scenario, but Fair-CDA can still improve the fairness.

Also, Fair-CDA is the only method that achieves considerable accuracy given high fairness requirements on both tasks.

We draw the saliency map on the wavy hair recognition task, as shown in Figure 4 .

Sensitive features are those strongly related to gender, while non-sensitive features are those strongly related to wavy hair.

It can be seen that the saliency maps of sensitive features focus more on the whole face, while those of non-sensitive features focus more on the hair of a man/woman.

Additionally, we evaluate Fair-CDA on more sensitive features on CelebA dataset.

We implement Fair-CDA on the same task as that in #OTHEREFR

Fair-Cda: Continuous And Directional Augmentation For Group Fairness

Abstract: In this work, we propose Fair-CDA, a fine-grained data augmentation strategy for imposing fairness constraints. We use a feature disentanglement method to extract the features highly related to the sensitive attributes. Then we show that group fairness can be achieved by regularizing the models on transition paths of sensitive features between groups. By adjusting the perturbation strength in the direction of the paths, our proposed augmentation is controllable and auditable. To alleviate the accuracy degradation caused by fairness constraints, we further introduce a calibrated model to impute labels for the augmented data. Our proposed method does not assume any data generative model and ensures good generalization for both accuracy and fairness. Experimental results show that Fair-CDA consistently outperforms state-of-the-art methods on widely-used benchmarks, e.g., Adult, CelebA and Movie-Lens. Especially, Fair-CDA obtains an 86.3% relative improvement for fairness while maintaining the accuracy on the Adult dataset. Moreover, we evaluate Fair-CDA in an online recommendation system to demonstrate the effectiveness of our method in terms of accuracy and fairness.

Sanity Checks For Saliency Maps

Abstract: Saliency methods have emerged as a popular tool to highlight features in an input deemed relevant for the prediction of a learned model. Several saliency methods have been proposed, often guided by visual appeal on image data. In this work, we propose an actionable methodology to evaluate what kinds of explanations a given method can and cannot provide. We find that reliance, solely, on visual assessment can be misleading. Through extensive experiments we show that some existing saliency methods are independent both of the model and of the data generating process. Consequently, methods that fail the proposed tests are inadequate for tasks that are sensitive to either data or model, such as, finding outliers in the data, explaining the relationship between inputs and outputs that the model learned, and debugging the model. We interpret our findings through an analogy with edge detection in images, a technique that requires neither training data nor model. Theory in the case of a linear model and a single-layer convolutional neural network supports our experimental findings.

feature decomposition
interpretability methods


This pattern library extends the initial pattern catalog of 10-15 patterns originally identified by #REFR both quantitatively with additional patterns observed during the SMS and qualitatively, by offering the patterns in a machine-actionable rather than graphical representation.

While a number of papers about systems that learn and reason were collected as a basis for the analysis described in #OTHEREFR , these were not offered as a corpus of annotated papers to the community.

We addressed both challenges by conducting a large-scale Systematic Mapping Study (SMS #OTHEREFR ) on SWeMLS #OTHEREFR , through which we (i) proposed a set of characteristics for describing SWeMLS and (ii) systematically collected, selected and extracted data from nearly 500 papers describing such systems.

This led to the following artifacts which together are offered as one resource:

the SWeMLS ontology that describes the main aspects of SWeMLS including their internal workflow in terms of boxology patterns as shown in Fig. 1 .

The ontology schema (i.e., capturing important SWeMLS characteristics, e.g., StatisticalM odel) and relevant instances (e.g., DeepLearningM odel) were derived systematically during the scoping and analysis phases of the SMS, the SWeMLS-KG: a knowledge graph containing the machine-actionable description of almost 500 systems in terms of the SWeMLS ontology, and the SWeMLS Pattern Library containing the machine-actionable description of 45 SWeMLS patterns and their associated SHACL-based validation constraints.

This resource is timely considering the recent trend in the SW community (and beyond) to create systems that leverage both SW and ML components.

To the best of our knowledge, it is also novel by (i) providing the first ontology (and associated pattern library) for describing SWeMLS in a machine-actionable way and (ii) a methodologically collected corpus of SWeMLS and their semantic description.

The resource is of immediate benefit for (SW) researchers that aim to explore trends in the SWeMLS field by analysing the data in the SWeMLS-KG and as such promises to have an impact on the understanding of the status-quo in this emerging field.

Furthermore, the resource provides a semantic framework for describing SWeMLS and their internal details, thus potentially strongly influencing this field in terms of being well-documented, data-driven, and transparent.

We continue by discussing the impact of this resource (Sect.

Describing And Organizing Semantic Web And Machine Learning Systems In The Swemls-Kg

Abstract: The overall AI trend of creating neuro-symbolic systems is reflected in the Semantic Web community with an increased interest in the development of systems that rely on both Semantic Web resources and Machine Learning components (SWeMLS, for short). However, understanding trends and best practices in this rapidly growing field is hampered by a lack of standardized descriptions of these systems and an annotated corpus of such systems. To address these gaps, we leverage the results of a large-scale systematic mapping study collecting information about 470 SWeMLS papers and formalize these into one resource containing: (i) the SWeMLS ontology, (ii) the SWeMLS pattern library containing machine-actionable descriptions of 45 frequently occurring SWeMLS workflows, and (iii) SWEMLS-KG, a knowledge graph including machine-actionable metadata of the papers in terms of the SWeMLS ontology. This resource provides the first framework for semantically describing and organizing SWeMLS thus making a key impact in (1) understanding the status quo of the field based on the published paper corpus and (2) enticing the uptake of machine-processable system documentation in the SWeMLS area.

A Boxology Of Design Patterns For Hybrid Learning And Reasoning Systems

Abstract: We propose a set of compositional design patterns to describe a large variety of systems that combine statistical techniques from machine learning with symbolic techniques from knowledge representation. As in other areas of computer science (knowledge engineering, software engineering, ontology engineering, process mining and others), such design patterns help to systematize the literature, clarify which combinations of techniques serve which purposes, and encourage re-use of software components. We have validated our set of compositional design patterns against a large body of recent literature.

machine-actionable rather
additional patterns


A. Pointr-C: 3D Shape Completion Network

The features are used as tokens to a transformer #REFR which captures the long-range relations among them and predicts the centers for the missing point cloud.

Given the current set of observations v o ∈ V, we predict the complete volume using a learning-based predictor g, i.e., V = g(v o ).

To obtainV, we use PoinTr #OTHEREFR , a transformer-based architecture that uses 3D point clouds as the input and output. PoinTr works in multiple steps for shapes completion.

First, a k-nearest neighbor (kNN) algorithm is applied to the partial point cloud to find the cluster centers that represent geometric relationships at a low resolution.

A DGCNN #OTHEREFR adds the local features around the these center points.

Lastly, FoldingNet #OTHEREFR performs a coarse-to-fine transformation over the predicted centers to predict the missing point cloud.

This model was trained on the ShapeNet #OTHEREFR dataset and outperforms the previous methods on a range of objects.

However, PoinTr was trained with implicit knowledge of the center of the object.

Moving the partially observed point cloud to its center results in incorrect prediction from PoinTr.

To improve the predictions, we fine-tune PoinTr using the curriculum framework, which dictates training the network over easy to hard tasks by increasing the difficulty in steps during learning #OTHEREFR .

Pred-Nbv: Prediction-Guided Next-Best-View Planning For 3D Object Reconstruction

Abstract: Prediction-based active perception has shown the potential to improve the navigation efficiency and safety of the robot by anticipating the uncertainty in the unknown environment. The existing works for 3D shape prediction make an implicit assumption about the partial observations and therefore cannot be used for real-world planning and do not consider the control effort for next-best-view planning. We present Pred-NBV, a realistic object shape reconstruction method consisting of PoinTr-C, an enhanced 3D prediction model trained on the ShapeNet dataset, and an information and control effort-based next-best-view method to address these issues. Pred-NBV shows an improvement of 25.46% in object coverage over the traditional methods in the AirSim simulator, and performs better shape completion than PoinTr, the stateof-the-art shape completion model, even on real data obtained from a Velodyne 3D LiDAR mounted on DJI M600 Pro.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


We used CQT transform as a spectrogram function for all experiments. We used nnAudio library #REFR for spectrogram calculation.

Split for training, validation and testing was datasetwise, which means that data from the training set for MAPS and the training set for GuitarSet were present in the training set for SynthesizedInstruments.

We are using only randomly chosen fixed-size sequences of each composition for training, so it should not make this model overfit to traits specific to datasets distributions.

Achieved recordings contain the clean version of each instrument, which is often not desired in analyzing noisy real-world data recorded by modern microphones.

Each recording generated by the software synthesizer was later sampled with a 16 kHz sample rate and transformed into the CQT spectrogram.

Data processing Each experiment was focused on training on a specific dataset.

To avoid recalculating CQT transform each time, we saved data on disk once it was calculated and loaded it in the subsequent experiments to not perform it again.

Experimental protocol Datasets were split into training, validation, and testing sets.

We checked the model using the validation set after every ten learning epochs.

After training, all datasets were tested using corresponding testing sets for all available datasets.

In the discussion of results, only datasets containing real-world recordings were considered (MAPS and GuitarSet).

Transfer Of Knowledge Among Instruments In Automatic Music Transcription

Abstract: Automatic music transcription (AMT) is one of the most challenging tasks in the music information retrieval domain. It is the process of converting an audio recording of music into a symbolic representation containing information about the notes, chords, and rhythm. Current research in this domain focuses on developing new models based on transformer architecture or using methods to perform semi-supervised training, which gives outstanding results, but the computational cost of training such models is enormous. This work shows how to employ easily generated synthesized audio data produced by software synthesizers to train a universal model. It is a good base for further transfer learning to quickly adapt transcription model for other instruments. Achieved results prove that using synthesized data for training may be a good base for pretraining general-purpose models, where the task of transcription is not focused on one instrument.

Nnaudio: An On-The-Fly Gpu Audio To Spectrogram Conversion Toolbox Using 1D Convolution Neural Networks

Abstract: Converting time domain waveforms to frequency domain spectrograms is typically considered to be a prepossessing step done before model training. This approach, however, has several drawbacks. First, it takes a lot of hard disk space to store different frequency domain representations. This is especially true during the model development and tuning process, when exploring various types of spectrograms for optimal performance. Second, if another dataset is used, one must process all the audio clips again before the network can be retrained. In this paper, we integrate the time domain to frequency domain conversion as part of the model structure, and propose a neural network based toolbox, nnAudio, which leverages 1D convolutional neural networks to perform time domain to frequency domain conversion during feed-forward. It allows on-the-fly spectrogram generation without the need to store any spectrograms on the disk. This approach also allows back-propagation on the waveforms-to-spectrograms transformation layer, which implies that this transformation process can be made trainable, and hence further optimized by gradient descent. nnAudio reduces the waveforms-to-spectrograms conversion time for 1,770 waveforms (from the MAPS dataset) from $10.64$ seconds with librosa to only $0.001$ seconds for Short-Time Fourier Transform (STFT), $18.3$ seconds to $0.015$ seconds for Mel spectrogram, $103.4$ seconds to $0.258$ for constant-Q transform (CQT), when using GPU on our DGX work station with CPU: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz Tesla v100 32Gb GPUs. (Only 1 GPU is being used for all the experiments.) We also further optimize the existing CQT algorithm, so that the CQT spectrogram can be obtained without aliasing in a much faster computation time (from $0.258$ seconds to only $0.001$ seconds).


These results corroborate the previous observation in #REFR that, parametric filters offer a more robust alternative to the particle filter in case of parameter uncertainty.

It can be seen that overall the EKF approximation is poor and has the worst performance.

UKF and ENKF improve upon this, where UKF is more robust. However PF outperforms all three of them, as expected.

As for EFKF, for low and high values of α the results are comparable to PF where EFKF performs better in some cases and PF better in the others. The same also applies to SKF.

On the other hand, for mid range of α, EFKF is significantly better than all the other filters, including PF.

The best value is highlighted in boldface, which occurs at α = 0.7.

Interesting, unlike the previous case, we see that EFKF also provides better performance even when there is no parameter uncertainty.

For this "Match" we have the values in the rightmost column.

Firstly, we can see that the PF outperforms all competitors including SKF and MKF, except for EFKF.

For the case of α = 0.5 and α = 0.7, EFKF gives significantly lower RMSE.

The best is once again highlighted in boldface and occurs at α = 0.7.

Nonlinear Kalman Filtering With Reparametrization Gradients

Abstract: We introduce a novel nonlinear Kalman filter that utilizes reparametrization gradients. The widely used parametric approximation is based on a jointly Gaussian assumption of the state-space model, which is in turn equivalent to minimizing an approximation to the Kullback-Leibler divergence. It is possible to obtain better approximations using the alpha divergence, but the resulting problem is substantially more complex. In this paper, we introduce an alternate formulation based on an energy function, which can be optimized instead of the alpha divergence. The optimization can be carried out using reparametrization gradients, a technique that has recently been utilized in a number of deep learning models.

Nonlinear Kalman Filtering With Divergence Minimization

Abstract: We consider the nonlinear Kalman filtering problem using Kullback–Leibler (KL) and $\alpha$ -divergence measures as optimization criteria. Unlike linear Kalman filters, nonlinear Kalman filters do not have closed form Gaussian posteriors because of a lack of conjugacy due to the nonlinearity in the likelihood. In this paper, we propose novel algorithms to approximate this posterior by optimizing the forward and reverse forms of the KL divergence, as well as the $\alpha$ -divergence that contains these two as limiting cases. Unlike previous approaches, our algorithms do not make approximations to the divergences being optimized, but use Monte Carlo techniques to derive unbiased algorithms for direct optimization. We assess performance on radar and sensor tracking, and options pricing, showing general improvement over the extended, unscented, and ensemble Kalman filters, as well as competitive performance with particle filtering.


To address the issue, we introduce the multi-scale network, which was firstly proposed in computer vision tasks #REFR .

If only a certain part of an IPT is considered, it tends to misjudge one IPT as another.

For example, a portamento note will be misjudged as a normal pluck note if its inflection point in the spectrogram is out of the receptive field of the model.

As solutions, the receptive field is usually enlarged by directly stacking multiple convolution layers or use large-size convolution kernels.

But these methods both result in an excess of parameters.

Simply increasing the receptive field will also lead to a loss in details for the subtle change of short IPTs.

As shown in Fig.2(a) , our proposed model is composed of three horizontal branches for different scales in the time axis.

The resolution of the feature in the branches from top to bottom is from high to low.

The middle branch with the medium resolution is used as a transition for the fusing between high-resolution features and long-range features.

By downsampling/upsampling the feature to different scales, longrange features can be fused with high-resolution features repeatedly.

To convert the spectral information into the channel domain, we first process the CQT input (1, F, T) into a sequence with the shape of (88, T, 1) by reshaping and batch normalization.

Frame-Level Multi-Label Playing Technique Detection Using Multi-Scale Network And Self-Attention Mechanism

Abstract: Instrument playing technique (IPT) is a key element of musical presentation. However, most of the existing works for IPT detection only concern monophonic music signals, yet little has been done to detect IPTs in polyphonic instrumental solo pieces with overlapping IPTs or mixed IPTs. In this paper, we formulate it as a framelevel multi-label classification problem and apply it to Guzheng, a Chinese plucked string instrument. We create a new dataset, Guzheng Tech99, containing Guzheng recordings and onset, offset, pitch, IPT annotations of each note. Because different IPTs vary a lot in their lengths, we propose a new method to solve this problem using multi-scale network and self-attention. The multi-scale network extracts features from different scales, and the self-attention mechanism applied to the feature maps at the coarsest scale further enhances the long-range feature extraction. Our approach outperforms existing works by a large margin, indicating its effectiveness in IPT detection.

Deep High-Resolution Representation Learning For Visual Recognition

Abstract: High-resolution representations are essential for position-sensitive vision problems, such as human pose estimation, semantic segmentation, and object detection. Existing state-of-the-art frameworks first encode the input image as a low-resolution representation through a subnetwork that is formed by connecting high-to-low resolution convolutions \emph{in series} (e.g., ResNet, VGGNet), and then recover the high-resolution representation from the encoded low-resolution representation. Instead, our proposed network, named as High-Resolution Network (HRNet), maintains high-resolution representations through the whole process. There are two key characteristics: (i) Connect the high-to-low resolution convolution streams \emph{in parallel}; (ii) Repeatedly exchange the information across resolutions. The benefit is that the resulting representation is semantically richer and spatially more precise. We show the superiority of the proposed HRNet in a wide range of applications, including human pose estimation, semantic segmentation, and object detection, suggesting that the HRNet is a stronger backbone for computer vision problems. All the codes are available at~{\url{this https URL}}.


Edge-AI optimizes the response time and accelerates AI computation when data is generated and processed for inference on-device #REFR .

In the past, AI approaches including machine learning, computer vision, and robotics along with data mining and image processing techniques are utilized to address the SHM challenges #OTHEREFR .

Especially deep learning approaches are broadly utilized in the past for detecting and segmenting bridge components #OTHEREFR .

Results obtained through these computational approaches have shown immense potential to support health monitoring tasks such as crack detection, segmentation of bridge components, UAV inspection of bridges, and damage detection for bridge structure #OTHEREFR .

However, real-time inferences in SHM, especially crack detection is challenging based on several factors like a wide range of various complex backgrounds and crack-like features.

Edge-AI allows a process for artificial intelligence inference and computation on-device rather than on cloud servers or network connections.

Zhou et al 2019 #OTHEREFR suggests that this ability of edge-AI platforms has advantages as follows- In this research study, drawing from artificial intelligence (AI), neural network process acceleration, and edge computing literature, the study aims to investigate edge-AI integration in structural health monitoring tasks.

The objective is to develop a lightweight neural network model and to optimize neural network response time performance in real time.

To accomplish this, we are utilizing Kneron KL520 platform which includes a neural processing unit (NPU), an AI system-on-chip(SoC) hardware.

To address the inherent challenges for real-time inference an edge-AI framework for SHM domain is proposed.

This paper introduces a novel framework to generate lightweight optimized models on the edge within the SHM domain. Figure 1 describes the edge-AI-SHM framework.

Integrating Edge-Ai In Structural Health Monitoring Domain *

Abstract: Structural health monitoring (SHM) tasks like damage detection are crucial for decision-making regarding maintenance and deterioration. For example, crack detection in SHM is crucial for bridge maintenance as crack progression can lead to structural instability. However, most AI/ML models in the literature have low latency and late inference time issues while performing in real-time environments. This study aims to explore the integration of edge-AI in SHM domain for real-time bridge inspections. Based on edge-AI literature, its capabilities will be valuable integration for a real-time decision support system in SHM tasks such that real-time inferences can be performed on physical sites. This study will utilize commercial edge-AI platforms, such as Google Coral Dev Board or Kneron KL520, to develop and analyze the effectiveness of edge-AI devices. Thus, this study proposes an edge AI framework for the structural health monitoring domain. An edge-AIcompatible deep learning model is developed to validate the framework to perform real-time crack classification. The effectiveness of this model will be evaluated based on its accuracy, the confusion matrix generated, and the inference time observed in a real-time setting.

Edge Intelligence: Paving The Last Mile Of Artificial Intelligence With Edge Computing

Abstract: With the breakthroughs in deep learning, the recent years have witnessed a booming of artificial intelligence (AI) applications and services, spanning from personal assistant to recommendation systems to video/audio surveillance. More recently, with the proliferation of mobile computing and Internet of Things (IoT), billions of mobile and IoT devices are connected to the Internet, generating zillions bytes of data at the network edge. Driving by this trend, there is an urgent need to push the AI frontiers to the network edge so as to fully unleash the potential of the edge big data. To meet this demand, edge computing, an emerging paradigm that pushes computing tasks and services from the network core to the network edge, has been widely recognized as a promising solution. The resulted new interdiscipline, edge AI or edge intelligence (EI), is beginning to receive a tremendous amount of interest. However, research on EI is still in its infancy stage, and a dedicated venue for exchanging the recent advances of EI is highly desired by both the computer system and AI communities. To this end, we conduct a comprehensive survey of the recent research efforts on EI. Specifically, we first review the background and motivation for AI running at the network edge. We then provide an overview of the overarching architectures, frameworks, and emerging key technologies for deep learning model toward training/inference at the network edge. Finally, we discuss future research opportunities on EI. We believe that this survey will elicit escalating attentions, stimulate fruitful discussions, and inspire further research ideas on EI.


We learn a joint vocabulary for glosses and texts via byte pair encoding (BPE) #REFR .

Datasets We work on three SLT datasets: PHOENIX-2014T, CSL-Daily, and DGS3-T.

PHOENIX-2014T and DGS3-T focus on German Sign Language, CSL-Daily on Chinese Sign Language.

All three datasets provide triplet samples, each consisting of a sign language video, a sentence-level gloss annotation and their corresponding text translation. Detailed statistics are listed in Table 1 .

We employ MuST-C English-German (En-De, 229K samples) and English-Chinese (En-Zh, 185K samples) #OTHEREFR as the augmented MT data for PHOENIX-2014T/DGS3-T and CSL-Daily, respectively.

We employ 1K BPE operations when MT data is not used, and increase it to 8K/8K/10K for PHOENIX-2014T/DGS3-T/CSL-Daily otherwise.

Model Settings We experiment with Transformer and start our analysis with a Baseline system optimized on Sign2Text alone with the following configurations: encoder and decoder layers of N S enc = 2, N P enc = 0 and N dec = 2 respectively, model dimension of d = 512, feed-forward dimension of d f f = 2048, attention head of h = 8, and no CTC regularization.

We adopt the SMKD model (Hao et al., 2021) 1 to extract sign embeddings, and pretrain the model on each benchmark separately on the Sign2Gloss task considering the large difference of sign videos across benchmarks.

More details about datasets and model settings are given in Appendix A.1.

Evaluation We report results mainly on the SLT task.

Sltunet: A Simple Unified Model For Sign Language Translation

Abstract: Despite recent successes with neural models for sign language translation (SLT), translation quality still lags behind spoken languages because of the data scarcity and modality gap between sign video and text. To address both problems, we investigate strategies for cross-modality representation sharing for SLT. We propose SLTUNET, a simple unified neural model designed to support multiple SLTrelated tasks jointly, such as sign-to-gloss, gloss-to-text and sign-to-text translation. Jointly modeling different tasks endows SLTUNET with the capability to explore the cross-task relatedness that could help narrow the modality gap. In addition, this allows us to leverage the knowledge from external resources, such as abundant parallel data used for spoken-language machine translation (MT). We show in experiments that SLTUNET achieves competitive and even state-of-theart performance on PHOENIX-2014T and CSL-Daily when augmented with MT data and equipped with a set of optimization techniques. We further use the DGS Corpus for end-to-end SLT for the first time. It covers broader domains with a significantly larger vocabulary, which is more challenging and which we consider to allow for a more realistic assessment of the current state of SLT than the former two. Still, SLTUNET obtains improved results on the DGS Corpus. Code is available at https://github.com/bzhangGo/sltunet.

Neural Machine Translation Of Rare Words With Subword Units

Abstract: Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character ngram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English!German and English!Russian by up to 1.1 and 1.3 BLEU, respectively.