Cityscapes dataset #REFR is another benchmark dataset for SSS, which focuses on urban scenarios and it consists of 2,975 annotated training images, 500 validation images and 1,525 testing images from 19 classes.

Pascal VOC 2012 dataset #OTHEREFR is a standard semisupervised semantic segmentation (SSS) benchmark dataset, which consists of over 13,000 images from 21 classes.

It contains 1,464 fully annotated images for training, 1,449 images for validation and 1,456 images for testing.

Previous works use SBD #OTHEREFR to render the labelled images and extend the number of labelled data to 10,582.

The rendered labelled images are of low quality and some of them are accompanied by noise.

Therefore, most of the previous works validate their SSS methods with sampled labelled images from the high-quality training images and rendered training images, respectively.

Conflict-Based Cross-View Consistency For Semi-Supervised Semantic Segmentation

Abstract: Semi-supervised semantic segmentation (SSS) has recently gained increasing research interest as it can reduce the requirement for large-scale fully-annotated training data. The current methods often suffer from the confirmation bias from the pseudo-labelling process, which can be alleviated by the co-training framework. The current co-training-based SSS methods rely on hand-crafted perturbations to prevent the different sub-nets from collapsing into each other, but these artificial perturbations cannot lead to the optimal solution. In this work, we propose a new conflict-based cross-view consistency (CCVC) method based on a two-branch co-training framework which aims at enforcing the two sub-nets to learn informative features from irrelevant views. In particular, we first propose a new cross-view consistency (CVC) strategy that encourages the two sub-nets to learn distinct features from the same input by introducing a feature discrepancy loss, while these distinct features are expected to generate consistent prediction scores of the input. The CVC strategy helps to prevent the two sub-nets from stepping into the collapse. In addition, we further propose a conflict-based pseudo-labelling (CPL) method to guarantee the model will learn more useful information from conflicting predictions, which will lead to a stable training process. We validate our new CCVC approach on the SSS benchmark datasets where our method achieves new state-of-the-art performance. Our code is available at https://github.com/xiaoyao3302/CCVC.

The Cityscapes Dataset For Semantic Urban Scene Understanding

Abstract: Visual understanding of complex urban street scenes is an enabling factor for a wide range of applications. Object detection has benefited enormously from large-scale datasets, especially in the context of deep learning. For semantic urban scene understanding, however, no current dataset adequately captures the complexity of real-world urban scenes. To address this, we introduce Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling. Cityscapes is comprised of a large, diverse set of stereo video sequences recorded in streets from 50 different cities. 5000 of these images have high quality pixel-level annotations, 20 000 additional images have coarse annotations to enable methods that leverage large volumes of weakly-labeled data. Crucially, our effort exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity. Our accompanying empirical study provides an in-depth analysis of the dataset characteristics, as well as a performance evaluation of several state-of-the-art approaches based on our benchmark.


It coincides with the performance of centralized benchmark (44) and the asymptotic performance of #REFR .

The proof revolves around introducing a long-term model, analogous to #OTHEREFR , which can be shown to be accurate for small step-sizes and under conditions 3-4. Details are omitted due to space limitations. We verify the accuracy of numerically in Section 5.

We note that for moderately large step-sizes µ, expression (59) will yield more accurate estimates of the steadystate performance than (62). Relation (62) on the other hand is more tractable.

We can interpret U T H U in (62) as the projection of the Hessian onto the space spanned by U, while U T R s U is the projection of the noise covariance onto the same space.

In this sense, Tr U T H U −1 U T R s U is a measure of the inverse signal-to-noise ratio after restricting the signal to the space of feasible solutions in #OTHEREFR .

We will verify in Section 5 that (59) is accurate for finite step-sizes µ, and that the proposed bias-corrected algorithm outperforms the approximate solution of #OTHEREFR .

Exact Subspace Diffusion For Decentralized Multitask Learning

Abstract: Classical paradigms for distributed learning, such as federated or decentralized gradient descent, employ consensus mechanisms to enforce homogeneity among agents. While these strategies have proven effective in i.i.d. scenarios, they can result in significant performance degradation when agents follow heterogeneous objectives or data. Distributed strategies for multitask learning, on the other hand, induce relationships between agents in a more nuanced manner, and encourage collaboration without enforcing consensus. We develop a generalization of the exact diffusion algorithm for subspace constrained multitask learning over networks, and derive an accurate expression for its mean-squared deviation when utilizing noisy gradient approximations. We verify numerically the accuracy of the predicted performance expressions, as well as the improved performance of the proposed approach over alternatives based on approximate projections.

Adaptation And Learning Over Networks Under Subspace Constraints—Part I: Stability Analysis

Abstract: This paper considers optimization problems over networks where agents have individual objectives to meet, or individual parameter vectors to estimate, subject to subspace constraints that require the objectives across the network to lie in low-dimensional subspaces. This constrained formulation includes consensus optimization as a special case, and allows for more general task relatedness models such as smoothness. While such formulations can be solved via projected gradient descent, the resulting algorithm is not distributed. Starting from the centralized solution, we propose an iterative and distributed implementation of the projection step, which runs in parallel with the stochastic gradient descent update. We establish that, for small step-sizes $\mu$, the proposed distributed adaptive strategy leads to small estimation errors on the order of $\mu$. We also examine steady-state performance. The results reveal explicitly the influence of the gradient noise, data characteristics, and subspace constraints, on the network performance. The results also show that in the small step-size regime, the iterates generated by the distributed algorithm achieve the centralized steady-state performance. Finally, we apply the proposed strategy to distributed adaptive beamforming.


Besides, we apply the co-activity similarity loss L cas 1 #REFR on the enhanced feature E a and the suppressed coarse T-CAS S coarse to model better definitephase feature representation.

The loss L supp mil is defined as the cross-entropy loss between the video label and the prediction p(j),

where y f (j) contains only foreground activities, i.e., the background class y f (C + 1) = 0 for suppressing the background activity.

And the final suppressed MIL loss can be formulated as,

Following the work #OTHEREFR , we utilize a L1-norm loss L norm to make foreground weights more polarized, denoted as

In addition, we introduce a guidance loss L guide to make the distribution of action-ness scores A ness opposite to the background class probability in S, i.e.,

L guide = T i=1 |1 − A ness i − s c+1 |,

Jcdnet: Joint Of Common And Definite Phases Network For Weakly Supervised Temporal Action Localization

Abstract: Weakly-supervised temporal action localization aims to localize action instances in untrimmed videos with only video-level supervision. We witness that different actions record common phases, e.g., the run-up in the HighJump and LongJump. These different actions are defined as conjoint actions, whose rest parts are definite phases, e.g., leaping over the bar in a HighJump. Compared with the common phases, the definite phases are more easily localized in existing researches. Most of them formulate this task as a Multiple Instance Learning paradigm, in which the common phases are tended to be confused with the background, and affect the localization completeness of the conjoint actions. To tackle this challenge, we propose a Joint of Common and Definite phases Network (JCDNet) by improving feature discriminability of the conjoint actions. Specifically, we design a Class-Aware Discriminative module to enhance the contribution of the common phases in classification by the guidance of the coarse definite-phase features. Besides, we introduce a temporal attention module to learn robust action-ness scores via modeling temporal dependencies, distinguishing the common phases from the background. Extensive experiments on three datasets (THUMOS14, ActivityNetv1.2, and a conjoint-action subset) demonstrate that JCDNet achieves competitive performance against the state-of-the-art methods.

W-Talc: Weakly-Supervised Temporal Activity Localization And Classification

Abstract: Most activity localization methods in the literature suffer from the burden of frame-wise annotation requirement. Learning from weak labels may be a potential solution towards reducing such manual labeling effort. Recent years have witnessed a substantial influx of tagged videos on the Internet, which can serve as a rich source of weakly-supervised training data. Specifically, the correlations between videos with similar tags can be utilized to temporally localize the activities. Towards this goal, we present W-TALC, a Weakly-supervised Temporal Activity Localization and Classification framework using only video-level labels. The proposed network can be divided into two sub-networks, namely the Two-Stream based feature extractor network and a weakly-supervised module, which we learn by optimizing two complimentary loss functions. Qualitative and quantitative results on two challenging datasets - Thumos14 and ActivityNet1.2, demonstrate that the proposed method is able to detect activities at a fine granularity and achieve better performance than current state-of-the-art methods.


In 2018, a work called sparse manifold transform #REFR builds upon the above two areas.

Our work has also been inspired by the classical approaches before deep learning, especially sparse modeling and manifold learning.

Some earlier works approach unsupervised learning mainly from the perspective of sparsity #OTHEREFR .

In particular, a work focuses on lossy coding #OTHEREFR has inspired many of the recent SSL learning methods #OTHEREFR , as well as our work to promote covariance in the representation of data through maximizing the coding rate.

Manifold learning #OTHEREFR and spectral clustering #OTHEREFR propose to model the geometric structure of high dimensional objects in the signal space.

The work proposes to use sparsity to handle locality in the data space to build support and construct representations that assign similar values to similar points on the support.

One may note that this work already shares a similar idea with the current joint-embedding self-supervised learning in the deep-learning community.

Emp-Ssl: Towards Self-Supervised Learning In One Training Epoch Under Review

Abstract: Recently, self-supervised learning (SSL) has achieved tremendous success in learning image representation. Despite the empirical success, most self-supervised learning methods are rather "inefficient" learners, typically taking hundreds of training epochs to fully converge. In this work, we show that the key towards efficient self-supervised learning is to increase the number of crops from each image instance. Leveraging one of the state-of-the-art SSL method, we introduce a simplistic form of self-supervised learning method called Extreme-Multi-Patch Self-Supervised-Learning (EMP-SSL) that does not rely on many heuristic techniques for SSL such as weight sharing between the branches, feature-wise normalization, output quantization, and stop gradient, etc, and reduces the training epochs by two orders of magnitude. We show that the proposed method is able to converge to 85.1% on CIFAR-10, 58.5% on CIFAR-100, 38.1% on Tiny ImageNet and 58.5% on ImageNet-100 in just one epoch. Furthermore, the proposed method achieves 91.5% on CIFAR-10, 70.1% on CIFAR-100, 51.5% on Tiny ImageNet and 78.9% on ImageNet-100 with linear probing in less than ten training epochs. In addition, we show that EMP-SSL shows significantly better transferability to out-of-domain datasets compared to baseline SSL methods. We will release the code in https://github.com/tsb0601/EMP-SSL. * Equal contribution 2 For example, all representations collapse to the same point.

The Sparse Manifold Transform

Abstract: We present a signal representation framework called the sparse manifold transform that combines key ideas from sparse coding, manifold learning, and slow feature analysis. It turns non-linear transformations in the primary sensory signal space into linear interpolations in a representational embedding space while maintaining approximate invertibility. The sparse manifold transform is an unsupervised and generative framework that explicitly and simultaneously models the sparse discreteness and low-dimensional manifold structure found in natural scenes. When stacked, it also models hierarchical composition. We provide a theoretical description of the transform and demonstrate properties of the learned representation on both synthetic data and natural videos.


It is also evident there is diversity in fact-checking practices and that robustness of the workflow depends on the collaborative and ongoing efforts of the team members, confirming findings from our earlier study #REFR .

In this paper we have reported findings from a series of semi-structured interviews with news professionals working in fact-checking.

We have seen how the fact-checking workflow consists of series of stages.

Each stage has its particular challenges for fact-checkers working under pressure to determine quickly the veracity of a claim and publish the result.

Interviewees identified ways in which computational tools could assist throughout the fact-checking workflow: surfacing claims, deciding on which are check-worthy and assembling the evidence.

Interviewees were skeptical about the prospects for automating the process.

They take for granted that the fact-checker is the human-in-the-loop and that they must be the final arbiter when a claim is assessed.

As with the introduction of advanced decision-making tools in other fields, there are important professional and organisational reasons why this should remain the case in fact-checking #OTHEREFR .

Hence, like several recent studies (e.g., #OTHEREFR , #OTHEREFR and #OTHEREFR

Some Observations On Fact-Checking Work With Implications For Computational Support

Abstract: Social media and user-generated content (UGC) have become increasingly important features of journalistic work in a number of different ways. However, the growth of misinformation means that news organisations have had devote more and more resources to determining its veracity and to publishing corrections if it is found to be misleading. In this work, we present the results of interviews with eight members of fact-checking teams from two organisations. Team members described their fact-checking processes and the challenges they currently face in completing a fact-check in a robust and timely way. The former reveals, inter alia, significant differences in fact-checking practices and the role played by collaboration between team members. We conclude with a discussion of the implications for the development and application of computational tools, including where computational tool support is currently lacking and the importance of being able to accommodate different fact-checking practices.

Supporting The Use Of User Generated Content In Journalistic Practice

Abstract: Social media and user-generated content (UGC) are increasingly important features of journalistic work in a number of different ways. However, their use presents major challenges, not least because information posted on social media is not always reliable and therefore its veracity needs to be checked before it can be considered as fit for use in the reporting of news. We report on the results of a series of in-depth ethnographic studies of journalist work practices undertaken as part of the requirements gathering for a prototype of a social media verification 'dashboard' and its subsequent evaluation. We conclude with some reflections upon the broader implications of our findings for the design of tools to support journalistic work.


In addition to the methods of CHECKMATE and MOCCASIN, we include results for the rounding algorithm proposed in #REFR under the "LP+rounding" column of the table.

Table 2 provides numerical results for a range of different computation graphs.

In Table 2 we have selected the memory budget values for each graph to be the 80% and 90% #OTHEREFR to evaluate CHECKMATE, where CM 1 is FCN with VGG layers and CM 2 is the ResNet50 model, n: number of nodes, m: number of edges, M : memory budget, TDI: total duration increase in percentage, peak mem: peak memory of the resulting rematerialization sequence.

The column 'Time (s)' indicates the elapsed time in seconds until the best solution. Dashes "-" indicate that no solution is found.

The best solution in each row is shown in bold font.

Full version of this of the initial peak memory without rematerialization.

This method consists of relaxing the MILP into a linear program (LP) and then rounding the solution (see #OTHEREFR for further details on this algorithm).

Note that solution produced by the rounding algorithm is not guaranteed to satisfy the memory budget constraint.

This could be seen in Table 2 where in most cases the peak memory for the relaxation and rounding approach is higher than the memory budget M .

Table 2 shows that the random layered graphs and real-world graphs are the most challenging ones among the graph set.

The solve times for these graphs are higher than the CM graphs, which is consistent with higher edge densities and more complex edge connectivities of the RL and RW graphs.

Moccasin: Efficient Tensor Rematerialization For Neural Networks

Abstract: The deployment and training of neural networks on edge computing devices pose many challenges. The low memory nature of edge devices is often one of the biggest limiting factors encountered in the deployment of large neural network models. Tensor rematerialization or recompute is a way to address high memory requirements for neural network training and inference. In this paper we consider the problem of execution time minimization of compute graphs subject to a memory budget. In particular, we develop a new constraint programming formulation called MOCCASIN with only O(n) integer variables, where n is the number of nodes in the compute graph. This is a significant improvement over the works in the recent literature that propose formulations with O(n 2) Boolean variables. We present numerical studies that show that our approach is up to an order of magnitude faster than recent work especially for large-scale graphs.

Checkmate: Breaking The Memory Wall With Optimal Tensor Rematerialization

Abstract: Modern neural networks are increasingly bottlenecked by the limited capacity of on-device GPU memory. Prior work explores dropping activations as a strategy to scale to larger neural networks under memory constraints. However, these heuristics assume uniform per-layer costs and are limited to simple architectures with linear graphs, limiting their usability. In this paper, we formalize the problem of trading-off DNN training time and memory requirements as the tensor rematerialization optimization problem, a generalization of prior checkpointing strategies. We introduce Checkmate, a system that solves for optimal schedules in reasonable times (under an hour) using off-the-shelf MILP solvers, then uses these schedules to accelerate millions of training iterations. Our method scales to complex, realistic architectures and is hardware-aware through the use of accelerator-specific, profile-based cost models. In addition to reducing training cost, Checkmate enables real-world networks to be trained with up to 5.1$\times$ larger input sizes.


It has been found that movement execution (ME) generates stronger amplitude co-relates in EEG signals, hence decoding ME gave promising results when compared to decoding movement imagination (MI) #REFR .

For instance, it does not require surgical intervention, it is painless, cheap, portable, and accurate #OTHEREFR .

Hence EEG is widely used in various biomedical applications including seizure detection #OTHEREFR , stress assessment #OTHEREFR , depression disorder detection #OTHEREFR , and schizophrenia #OTHEREFR .

The interaction of the human body with its environment depends significantly on controlled movements of the upper limbs.

However, spinal cord injury (SCI) and other neuro-muscular diseases can effect this control and limb movement.

It is crucial to restoring upper limb movements for people with SCI so they can independently take care of their daily activities.

Deep learning (DL) methods are getting a lot of attention recently in a diverse set of data processing tasks.

The same holds for upper limb movement classification using EEG data.

Features like minimal pre-processing of data, automatic low-and high-level feature extraction, and superior learning capabilities make DL an attractive tool in the machine learning domain #OTHEREFR .

Classification of the MI of two movements was performed using a convolutional neural network (CNN), along with common spatial patterns (CSP) as a feature extraction mechanism beforehand.

The strategy was able to classify palm extension and hand grasp movements #OTHEREFR .

Upper Limb Movement Execution Classification Using Electroencephalography For Brain Computer Interface

Abstract: An accurate classification of upper limb movements using electroencephalography (EEG) signals is gaining significant importance in recent years due to the prevalence of brain-computer interfaces. The upper limbs in the human body are crucial since different skeletal segments combine to make a range of motion that helps us in our trivial daily tasks. Decoding EEG-based upper limb movements can be of great help to people with spinal cord injury (SCI) or other neuro-muscular diseases such as amyotrophic lateral sclerosis (ALS), primary lateral sclerosis, and periodic paralysis. This can manifest in a loss of sensory and motor function, which could make a person reliant on others to provide care in day-today activities. We can detect and classify upper limb movement activities, whether they be executed or imagined using an EEG-based brain-computer interface (BCI). Toward this goal, we focus our attention on decoding movement execution (ME) of the upper limb in this study. For this purpose, we utilize a publicly available EEG dataset that contains EEG signal recordings from fifteen subjects acquired using a 61-channel EEG device. We propose a method to classify four ME classes for different subjects using spectrograms of the EEG data through pre-trained deep learning (DL) models. Our proposed method of using EEG spectrograms for the classification of ME has shown significant results, where the highest average classification accuracy (for four ME classes) obtained is 87.36%, with one subject achieving the best classification accuracy of 97.03%. Clinical relevance-This research shows that movement execution of upper limbs is classified with significant accuracy by employing a spectrogram of the EEG signals and a pre-trained deep learning model which is fine-tuned for the downstream task.

Decoding Movement Imagination And Execution From Eeg Signals Using Bci-Transfer Learning Method Based On Relation Network

Abstract: A brain-computer interface (BCI) is used not only to control external devices for healthy people but also to rehabilitate motor functions for motor-disabled patients. Decoding movement intention is one of the most significant aspects for performing arm movement tasks using brain signals. Decoding movement execution (ME) from electroencephalogram (EEG) signals have shown high performance in previous works, however movement imagination (MI) paradigm-based intention decoding has so far failed to achieve sufficient accuracy. In this study, we focused on a robust MI decoding method with transfer learning for the ME and MI paradigm. We acquired EEG data related to arm reaching for 3D directions. We proposed a BCI-transfer learning method based on a Relation network (BTRN) architecture. Decoding performances showed the highest performance compared to conventional works. We confirmed the possibility of the BTRN architecture to contribute to continuous decoding of MI using ME datasets.


To better capture long range spatial-temporal dependencies in users' historical check-in sequences, we stack Transformer encoder layers #REFR for constructing the history encoder.

All these vectors are then linearly projected into -dimensional embeddings ∈ R , ∈ R , and ∈ R .

In this way, for the user , the historical check-in sequence { } =1 can be further denoted as

Note that since checkin data requires a certain order of precedence, learnable positional embedding is also added into inputs for history encoder.

Compared with previous RNN-based methods, Transformer architecture #OTHEREFR can not only avoid recurrence, allowing parallel computing to reduce training time, but also migrate performance degradation problem with regard to long-term dependencies in RNNs.

Each Transformer encoder layer involves a multi-head self-attention module and a point-wise feed-forward network.

We also keep the residual connection and layer normalization employed in Transformer encoder layers.

Dividing the attention mechanism into multiple heads to form multiple sub-spaces allows the model to focus on different aspects of information.

For each attention head, self-attention result for a check-in can be computed as

Timestamps As Prompts For Geography-Aware Location Recommendation

Abstract: Figure 1: An illustration of how TPG performs the next location recommendation (denoted by the purple line) and interval predictions (denoted by red lines) by using temporal prompts. Different colored markers denote different categories of POIs.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

history encoder
Transformer encoder layers


A similar limitation has been reported for the simultaneous gradient descent-ascent (SGDA) approach #REFR .

First, we discuss the slow convergence issue on smooth strongly convex-concave problems highlighted in #OTHEREFR .

For instance, consider a convex-concave quadratic problem ( , ) = ( /2) 2 + − ( /2) 2 . The worst-case scenario isˆ( ) = ( / )

for each and the optimal solution isˆ( ) = −( / ) for each .

It is intuitive that bothˆ( ) andˆ( ) should not be too sensitive to follow their change by (3).

In fact, it has been theoretically derived that, for linear convergence, the learning rate must be set as , ∈ ( /( + 2 )) and the required number of iterations to find near-optimal solution is Ω(1 + 2 /( )); refer to #OTHEREFR for details.

The same limitation is expected to exist in ZO-Min-Max because it is regarded as an approximation of the SGDA approach.

The adaptation of the learning rates in ADV-CMA-ES can mitigate the difficulty in tuning learning rates. However, it cannot avoid the slow convergence problem.

The situation is worse if the objective function is convex-concave but not strongly convex-concave.

For example, consider 7 with = = 1 and = .

This objective function is similar to , but the coefficients are regarded as = (1/2) 2 and = (1/2) 2 , i.e., decreasing as the solution approaches the global min-max saddle point ( * , * =ˆ( * )).

Covariance Matrix Adaptation Evolutionary Strategy With Worst-Case Ranking Approximation For Min-Max Optimization And Its Application To Berthing Control Tasks

Abstract: In this study, we consider a continuous min-max optimization problem min ∈X max ∈Y (,) whose objective function is a black-box. We propose a novel approach to minimize the worst-case objective function () = max (,) directly using a covariance matrix adaptation evolution strategy (CMA-ES) in which the rankings of solution candidates are approximated by our proposed worst-case ranking approximation (WRA) mechanism. We develop two variants of WRA combined with CMA-ES and approximate gradient ascent as numerical solvers for the inner maximization problem. Numerical experiments show that our proposed approach outperforms several existing approaches when the objective function is a smooth strongly convex-concave function and the interaction between and is strong. We investigate the advantages of the proposed approach for problems where the objective function is not limited to smooth strongly convex-concave functions. The effectiveness of the proposed approach is demonstrated in the robust berthing control problem with uncertainty. CCS Concepts: • Mathematics of computing → Continuous optimization.

Interaction Matters: A Note On Non-Asymptotic Local Convergence Of Generative Adversarial Networks

Abstract: Motivated by the pursuit of a systematic computational and algorithmic understanding of Generative Adversarial Networks (GANs), we present a simple yet unified non-asymptotic local convergence theory for smooth two-player games, which subsumes several discrete-time gradient-based saddle point dynamics. The analysis reveals the surprising nature of the off-diagonal interaction term as both a blessing and a curse. On the one hand, this interaction term explains the origin of the slow-down effect in the convergence of Simultaneous Gradient Ascent (SGA) to stable Nash equilibria. On the other hand, for the unstable equilibria, exponential convergence can be proved thanks to the interaction term, for four modified dynamics proposed to stabilize GAN training: Optimistic Mirror Descent (OMD), Consensus Optimization (CO), Implicit Updates (IU) and Predictive Method (PM). The analysis uncovers the intimate connections among these stabilizing techniques, and provides detailed characterization on the choice of learning rate. As a by-product, we present a new analysis for OMD proposed in Daskalakis, Ilyas, Syrgkanis, and Zeng [2017] with improved rates.


Cubature Rule For Unitary Jacobi Ensembles

The corresponding specialization of the cubature rule in Theorem 2 was presented in turn in #REFR Section 9] .

(3.1d), (3.1e) that the boundary value ξ (m+n) 0 = 0 is attained iff ǫ − =ǫ − = 0 and the boundary value ξ (m+n) m+n−1 = π is attained iff ǫ + =ǫ + = 0 (since π 0 u a (θ)dθ = π for |a| < 1). Remark 4.

In the above integration formulas the degree of exactness is optimal if D (3.2b) reaches the Gaussian value 2m+1, which is achieved whend = 0 andǫ ± = 1. This special case of the quadrature rule in Eqs.

(3.1a)-(3.1g) can be inferred from #OTHEREFR for ǫ ± = 0, and from #OTHEREFR for general ǫ ± ∈ {0, 1} (cf. also #OTHEREFR Section 8] ).

(2π) n n! π 0 • • • π 0 f cos(ξ) ρ ǫ (ξ)dξ 1 • • • dξ n = (3.4a) 1 N (m,n) ǫ λ∈Λ (m,n) 1 2 (1−ǫ+)(1−ǫ+)δ m−λ 1 +(1−ǫ−)(1−ǫ−)δ λn f cos ξ (m,n) λ ρ ǫ ξ (m,n) λ

, x n ) = f (x) ∈ P (D,n) with D = 2m +ǫ + +ǫ − − 1 and

Cubature Rules For Unitary Jacobi Ensembles

Abstract: We present Chebyshev type cubature rules for the exact integration of rational symmetric functions with poles on prescribed coordinate hyperplanes. Here the integration is with respect to the densities of unitary Jacobi ensembles stemming from the Haar measures of the orthogonal and the compact symplectic Lie groups.

Exact Cubature Rules For Symmetric Functions

Abstract: We employ a multivariate extension of the Gauss quadrature formula, originally due to Berens, Schmid and Xu [BSX95], so as to derive cubature rules for the integration of symmetric functions over hypercubes (or infinite limiting degenerations thereof) with respect to the densities of unitary random matrix ensembles. Our main application concerns the explicit implementation of a class of cubature rules associated with the Bernstein-Szego polynomials, which permit the exact integration of symmetric rational functions with prescribed poles at coordinate hyperplanes against unitary circular Jacobi distributions stemming from the Haar measures on the symplectic and the orthogonal groups.


HDRNet #REFR obtains transformations on the low-resolution input and applies upsampled transformations to the full-resolution image with bilateral grid processing.

StarEnhancer #OTHEREFR introduces multiple-style enhancement with the ability to transform inputs to an unseen style.

CSRNet #OTHEREFR designs a lightweight framework containing a base network and a conditioning network for extracting global features and performing photo retouching, respectively.

Pienet #OTHEREFR constructs preference vectors with metric learning and adaptively enhances images according to user-provided preferable styles.

DeepLPF #OTHEREFR regresses the parameters of spatially localized filters and automatically applies those filters to enhance inputs.

Enhancement curve learning methods #OTHEREFR , #OTHEREFR , #OTHEREFR estimate retouching curves to tone global properties of inputs rather than directly mapping.

RCTNet #OTHEREFR first estimates the transformation of the representative colors and then enhances inputs according to the similarity between inputs and representative colors.

3D LUT based methods #OTHEREFR , #OTHEREFR utilize variants of 3D lookup tables (3D LUTs) with deep learning, achieving real-time and flexible photo enhancement performance.

In contrast to most of the methods above, 3D LUT HRP [2] focuses on portrait retouching and achieves region awareness through the human-region priority strategy.

Our work distinguishes from 3D LUT HRP #OTHEREFR by emphasizing the significance of the user guidance and investigating interactive region-aware portrait retouching, whcih provides the flexibility to retouch different instances according to users' intents.

Region-Aware Portrait Retouching With Sparse Interactive Guidance

Abstract: Portrait retouching aims to improve the aesthetic quality of input portrait photos and especially requires humanregion priority. The deep learning-based methods largely elevate the retouching efficiency and provide promising retouched results. However, existing portrait retouching methods focus on automatic retouching, which treats all human-regions equally and ignores users' preferences for specific individuals, thus suffering from limited flexibility in interactive scenarios. In this work, we emphasize the importance of users' intents and explore the interactive portrait retouching task. Specifically, we propose a regionaware retouching framework with two branches: an automatic branch and an interactive branch. The automatic branch involves an encoding-decoding process, which searches region candidates and performs automatic region-aware retouching without user guidance. The interactive branch encodes sparse user guidance into a priority condition vector and modulates latent features with a region selection module to further emphasize the userspecified regions. Experimental results show that our interactive branch effectively captures users' intents and generalizes well to unseen scenes with sparse user guidance, while our automatic branch also outperforms the state-of-the-art retouching methods due to improved region-awareness.

Deep Bilateral Learning For Real-Time Image Enhancement

Abstract: Performance is a critical challenge in mobile image processing. Given a reference imaging pipeline, or even human-adjusted pairs of images, we seek to reproduce the enhancements and enable real-time evaluation. For this, we introduce a new neural network architecture inspired by bilateral grid processing and local affine color transforms. Using pairs of input/output images, we train a convolutional neural network to predict the coefficients of a locally-affine model in bilateral space. Our architecture learns to make local, global, and content-dependent decisions to approximate the desired image transformation. At runtime, the neural network consumes a low-resolution version of the input image, produces a set of affine transformations in bilateral space, upsamples those transformations in an edge-preserving fashion using a new slicing node, and then applies those upsampled transformations to the full-resolution image. Our algorithm processes high-resolution images on a smartphone in milliseconds, provides a real-time viewfinder at 1080p resolution, and matches the quality of state-of-the-art approximation techniques on a large class of image operators. Unlike previous work, our model is trained off-line from data and therefore does not require access to the original operator at runtime. This allows our model to learn complex, scene-dependent transformations for which no reference implementation is available, such as the photographic edits of a human retoucher.


For example, #REFR selects the most common samples with the lowest diversity for replay, believing that the most representative samples will elevate the effectiveness of replay.

By doing so, the model can be trained with samples from both current and previous classes, resulting in better generalization.

However, since the number of selected samples in the memory is much smaller than those within the new classes, the selected samples are easy to be ignored or cause overfitting when training due to the small number.

Careful selection of the samples is required, which naturally brings the question: How to select the best samples for replay?

Some attempts have been made to answer the question, aiming to seek the most effective samples for replay.

Researchers propose different criteria that are mostly manually designed based on some heuristic factors like diversity #OTHEREFR .

However, the most common samples may not always be the samples being forgotten in later stages.

#OTHEREFR proposes to save both the low-diversity samples near the distribution center and high-diversity samples near the classification boundaries.

However, new challenges arise since the memory length is limited, so it is challenging to find the optimal quotas for the two kinds of samples to promote replay effectiveness to the greatest extent.

Moreover, most of the existing methods are designed based on a single factor, the selection performance, however, can be influenced by many factors with complicated relationships.

For example, besides diversity, memory sample selection should also be class-dependent because the hard classes need more samples to replay in order to alleviate the more severe catastrophic forgetting issue.

Continual Semantic Segmentation With Automatic Memory Sample Selection

Abstract: Continual Semantic Segmentation (CSS) extends static semantic segmentation by incrementally introducing new classes for training. To alleviate the catastrophic forgetting issue in CSS, a memory buffer that stores a small number of samples from the previous classes is constructed for replay. However, existing methods select the memory samples either randomly or based on a single-factor-driven handcrafted strategy, which has no guarantee to be optimal. In this work, we propose a novel memory sample selection mechanism that selects informative samples for effective replay in a fully automatic way by considering comprehensive factors including sample diversity and class performance. Our mechanism regards the selection operation as a decision-making process and learns an optimal selection policy that directly maximizes the validation performance on a reward set. To facilitate the selection decision, we design a novel state representation and a dual-stage action space. Our extensive experiments on Pascal-VOC 2012 and ADE 20K datasets demonstrate the effectiveness of our approach with state-of-the-art (SOTA) performance achieved, outperforming the second-place one by 12.54% for the 6stage setting on Pascal-VOC 2012.

Incremental Learning Techniques For Semantic Segmentation

Abstract: Deep learning architectures exhibit a critical drop of performance due to catastrophic forgetting when they are required to incrementally learn new tasks. Contemporary incremental learning frameworks focus on image classification and object detection while in this work we formally introduce the incremental learning problem for semantic segmentation in which a pixel-wise labeling is considered. To tackle this task we propose to distill the knowledge of the previous model to retain the information about previously learned classes, whilst updating the current model to learn the new ones. We propose various approaches working both on the output logits and on intermediate features. In opposition to some recent frameworks, we do not store any image from previously learned classes and only the last model is needed to preserve high accuracy on these classes. The experimental evaluation on the Pascal VOC2012 dataset shows the effectiveness of the proposed approaches.


We optimize PanoTransformer by minimizing a pixelwise reverse Huber loss #REFR between the predicted panoramas and corresponding ground truth.

Since using a standard L1 loss function to learn a binary light mask heavily penalizes even small shifts of a light source position, the reverse Huber loss takes advantage of L1 loss and L2 loss as below:

L B =    |y −ŷ|, |y −ŷ| ≤ T (y −ŷ) 2 + T 2 2T , |y −ŷ| ≥ T (4)

where y is the ground truth value andŷ is the prediction.

The threshold T is set to 0.2 in our experiments.

To generate more realistic details, an extra adversarial loss is also involved in the training process.

Local-To-Global Panorama Inpainting For Locale-Aware Indoor Lighting Prediction

Abstract: Figure 1. We propose a locale-aware indoor illumination prediction method that can generate a full and texture-rich HDR panorama (the third column in each group) at any locale in the scene, enabling spatially-varying and consistent shading after virtual object insertion (the first and second columns).

Deeper Depth Prediction With Fully Convolutional Residual Networks

Abstract: This paper addresses the problem of estimating the depth map of a scene given a single RGB image. To model the ambiguous mapping between monocular images and depth maps, we leverage on deep learning capabilities and present a fully convolutional architecture encompassing residual learning. The proposed model is deeper than the current state of the art, but contains fewer parameters and requires less training data, while still outperforming all current CNN approaches aimed at the same task. We further present a novel way to efficiently learn feature map up-sampling within the network. For optimization we introduce the reverse Huber loss, particularly suited for the task at hand and driven by the value distributions commonly present in depth maps. The predictions are given by a single architecture, trained end-to-end, that does not rely on post-processing techniques, such as CRFs or other additional refinement steps.


In the email, we pledged to donate US$ 2 per completed response to the United Nations World Food Programme #REFR .

We also merged records where individuals used different names but the same email address, or different email addresses but the same or an extremely similar name for different commits.

When judgment was needed, we made decisions as conservatively as possible. We ultimately obtained a list of 1,555 unique individuals.

In case of multiple email addresses per person, we selected only one, preferring personal email addresses over professional ones because the person's commits to the projects were partially already quite old and the person might have moved organizations since.

We sent an email to all 1,555 developers to invite them to our survey #OTHEREFR (which was part of a larger data collection effort for multiple studies).

We sent the invitation with a personalized link to the survey.

We also sent two reminders #OTHEREFR , including one that linked to an official university web page confirming the authenticity of the survey as some developers were concerned that the invitation might be a scam.

Because 165 emails bounced, we reached 1,390 developers (89.4% deliverable emails), of which 194 developers started the survey, and 124 completed it.

We dropped all respondents that provided implausible values for their age (≤10 or ≥100), leaving us with a preliminary sample of 121 developers.

Although this number may seem low, it is of a similar magnitude as the total number even from studies that assessed personality not based on self-reports but on mining vast corpora of emails.

Calefato et al., for instance, used about 1.35 million emails from 46,304 developers but only obtained full personality profiles for 211 of them (<0.5%), of which only 118 had made source code commits #OTHEREFR .

The Type To Take Out A Loan? A Study Of Developer Personality And Technical Debt

Abstract: Background: Technical debt (TD) has been widely discussed in software engineering research, and there is an emerging literature linking it to developer characteristics. However, developer personality has not yet been studied in this context. Aims and Method: We explore the relationship between various personality traits (Five Factor Model, regulatory focus, and narcissism) of developers and the introduction and removal of TD. To this end, we complement an existing TD dataset with novel selfreport personality data gathered by surveying developers, and analyze 2,145 commits from 19 developers. Results: We find that conscientiousness, emotional stability, openness to experience, and prevention focus are negatively associated with TD. There were no significant results for extraversion, agreeableness, promotion focus, or narcissism. Conclusions: We take our results as first evidence that developer personality has a systematic influence on the introduction and removal of TD. This has implications not only for future research, which could, for example, study the effects of personality on downstream consequences of TD like defects, but also for software engineering practitioners who may, for example, consider developer personality in staffing decisions.

Sampling In Software Engineering Research: A Critical Review And Guidelines

Abstract: Representative sampling appears rare in software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This study therefore investigates the state of sampling in recent, high-quality software engineering research. The key findings are: (1) random sampling is rare; (2) sophisticated sampling strategies are very rare; (3) sampling, representativeness and randomness do not appear well-understood. To address these problems, the paper synthesizes existing knowledge of sampling into a succinct primer and proposes extensive guidelines for improving the conduct, presentation and evaluation of sampling in software engineering research. It is further recommended that while researchers should strive for more representative samples, disparaging non-probability sampling is generally capricious and particularly misguided for predominately qualitative research.


By incentivizing incorrect behavior, misspecified objectives can lead to useless or even dangerous outcomes #REFR .

Standard AI and machine learning algorithms require the designer to specify a cost or reward function.

This objective incentivizes desired behavior and penalizes mistakes, teaching the system how to perform the task.

While such objectives are easy to manually specify for problems with clear win conditions, such as games #OTHEREFR and tasks with clear goals, such as image classification #OTHEREFR , they can be challenging to formalize for more nuanced tasks #OTHEREFR . For example, Lee et al.

#OTHEREFR find that humans struggle to define an objective that incentivizes bipedal locomotion, despite being experts in both machine learning and walking.

Ensuring that AI systems optimize objectives that align with our own is a crucial part of building safe and beneficial AI.

Reward learning techniques enable AI systems to learn their objectives by observing and interacting with humans instead of requiring their designers to specify these objectives manually #OTHEREFR .

Humans can train reward learning systems using a variety of feedback modalities, including demonstrations #OTHEREFR , pairwise comparisons #OTHEREFR , natural language #OTHEREFR , numeric values #OTHEREFR , corrections #OTHEREFR , and proxy rewards #OTHEREFR .

Reward learning from pairwise comparisons in particular has proven remarkably effective across a variety of tasks, including complex physical maneuvers for continuous control systems #OTHEREFR and text summarization for language language models #OTHEREFR .

In the future, it may even be possible to use reward learning to train AI systems to assist humans in researching safe AI #OTHEREFR .

Active Reward Learning From Multiple Teachers

Abstract: Reward learning algorithms utilize human feedback to infer a reward function, which is then used to train an AI system. This human feedback is often a preference comparison, in which the human teacher compares several samples of AI behavior and chooses which they believe best accomplishes the objective. While reward learning typically assumes that all feedback comes from a single teacher, in practice these systems often query multiple teachers to gather sufficient training data. In this paper, we investigate this disparity, and find that algorithmic evaluation of these different sources of feedback facilitates more accurate and efficient reward learning. We formally analyze the value of information (VOI) when reward learning from teachers with varying levels of rationality, and define and evaluate an algorithm that utilizes this VOI to actively select teachers to query for feedback. Surprisingly, we find that it is often more informative to query comparatively irrational teachers. By formalizing this problem and deriving an analytical solution, we hope to facilitate improvement in reward learning approaches to aligning AI behavior with human values.

Scalable Agent Alignment Via Reward Modeling: A Research Direction

Abstract: One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create agents that behave in accordance with the user's intentions? We outline a high-level research direction to solve the agent alignment problem centered around reward modeling: learning a reward function from interaction with the user and optimizing the learned reward function with reinforcement learning. We discuss the key challenges we expect to face when scaling reward modeling to complex and general domains, concrete approaches to mitigate these challenges, and ways to establish trust in the resulting agents.


Instead of a two-stage encoder for feature extraction #REFR , our FoCoViL makes full use of a single auto-encoder in an end-to-end fashion that already achieves promising results.

To maintain the action representation, we employ an effective sequential autoencoder as the backbone network sharing among the multi-view actions (see Fig. 2(c) ).

From #OTHEREFR , the encoder usually plays a more important role than the decoder to integrate representative features.

We thus consider a portable decoder by feeding the empty frame (zero vector) to every step of the decoder, such that the model only focuses on the hidden representation delivered from the encoded output.

The structure consists of a three-layer bi-directional encoder f e to derive the latent representation, a linear projection net g that is specifically designed for contrastive learning, and a single-layer uni-directional decoder f d for reconstruction purposes.

Both f e and f d are under the Gated Recurrent Unit (GRU) architecture to process frame-wise information.

For each action X u i , the reconstruction loss L r is defined as:

where T is the total number of frames in the action video.

Focalized Contrastive View-Invariant Learning For Self-Supervised Skeleton-Based Action Recognition

Abstract: Learning view-invariant representation is a key to improving feature discrimination power for skeleton-based action recognition. Existing approaches cannot effectively remove the impact of viewpoint due to the implicit view-dependent representations. In this work, we propose a self-supervised framework called Focalized Contrastive View-invariant Learning (FoCoViL), which significantly suppresses the view-specific information on the representation space where the viewpoints are coarsely aligned. By maximizing mutual information with an effective contrastive loss between multi-view sample pairs, FoCoViL associates actions with common view-invariant properties and simultaneously separates the dissimilar ones. We further propose an adaptive focalization method based on pairwise similarity to enhance contrastive learning for a clearer cluster boundary in the learned space. Different from many existing self-supervised representation learning work that rely heavily on supervised classifiers, FoCoViL performs well on both unsupervised and supervised classifiers with superior recognition performance. Extensive experiments also show that the proposed contrastive-based focalization generates a more discriminative latent representation.

Predict&Cluster: Unsupervised Skeleton Based Action Recognition

Abstract: We propose a novel system for unsupervised skeleton-based action recognition. Given inputs of body keypoints sequences obtained during various movements, our system associates the sequences with actions. Our system is based on an encoder-decoder recurrent neural network, where the encoder learns a separable feature representation within its hidden states formed by training the model to perform prediction task. We show that according to such unsupervised training the decoder and the encoder self-organize their hidden states into a feature space which clusters similar movements into the same cluster and distinct movements into distant clusters. Current state-of-the-art methods for action recognition are strongly supervised, i.e., rely on providing labels for training. Unsupervised methods have been proposed, however, they require camera and depth inputs (RGB+D) at each time step. In contrast, our system is fully unsupervised, does not require labels of actions at any stage, and can operate with body keypoints input only. Furthermore, the method can perform on various dimensions of body keypoints (2D or 3D) and include additional cues describing movements. We evaluate our system on three extensive action recognition benchmarks with different number of actions and examples. Our results outperform prior unsupervised skeleton-based methods, unsupervised RGB+D based methods on cross-view tests and while being unsupervised have similar performance to supervised skeleton-based action recognition.


For the given user and TTS pronunciations, the phoneme embedding sequences are concatenated and used as input to train a GBDT model using XGBoost #REFR with logistic loss.

We train a Gradient Boosted Decision Tree (GBDT) #OTHEREFR classifier using phoneme embeddings as input.

The annotations are binary labels, where 0 represents both pronunciations are the same, and 1 otherwise.

Dtw-Siamesenet: Dynamic Time Warped Siamese Network For Mispronunciation Detection And Correction

Abstract: Personal Digital Assistants (PDAs)-such as Siri, Alexa and Google Assistant, to name a few-play an increasingly important role to access information and complete tasks spanning multiple domains, and by diverse groups of users. A textto-speech (TTS) module allows PDAs to interact in a natural, human-like manner, and play a vital role when the interaction involves people with visual impairments or other disabilities. To cater to the needs of a diverse set of users, inclusive TTS is important to recognize and pronounce correctly text in different languages and dialects. Despite great progress in speech synthesis, the pronunciation accuracy of named entities in a multilingual setting still has a large room for improvement. Existing approaches to correct named entity (NE) mispronunciations, like retraining Grapheme-to-Phoneme (G2P) models, or maintaining a TTS pronunciation dictionary, require expensive annotation of the ground truth pronunciation, which is also time consuming. In this work, we present a highly-precise, PDA-compatible pronunciation learning framework for the task of TTS mispronunciation detection and correction. In addition, we also propose a novel mispronunciation detection model called DTW-SiameseNet, which employs metric learning with a Siamese architecture for Dynamic Time Warping (DTW) with triplet loss. We demonstrate that a locale-agnostic, privacypreserving solution to the problem of TTS mispronunciation detection is feasible. We evaluate our approach on a real-world dataset, and a corpus of NE pronunciations of an anonymized audio dataset of person names recorded by participants from 10 different locales. Human evaluation shows our proposed approach improves pronunciation accuracy on average by ≈ 6% compared to strong phoneme-based and audio-based baselines.

Xgboost: A Scalable Tree Boosting System

Abstract: Tree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.


Furthermore, the results of #REFR showed that there is a trade-off between the minimum repair bandwidth and the storage capacity of the network.

The rapid development of distributed storage systems raised the question of how failed nodes in these systems can be efficiently repaired.

A large body of literature on erasure codes for distributed storage addressed this question over the past decade.

One approach to assess repair efficiency is to measure the so-called repair bandwidth, which is the amount of information downloaded from other nodes for the repair.

This approach, first introduced in #OTHEREFR , assumes the nodes form a homogeneous network and determines the repair bandwidth by considering the information flow in course of the repair.

In particular, #OTHEREFR derived a bound on the smallest number of symbols required for the repair of a single failed node, known as the cut-set bound on the repair bandwidth.

In this paper we shall focus on codes with minimum storage overhead and optimal repair bandwidth, namely, maximum distance separable (MDS) codes with optimal repair bandwidth.

Such codes are termed minimum-storage regenerating (MSR) codes in the literature.

The problem of MSR codes has received much attention and affords several variants.

In its basic form, the problem concerns repair of a single failed node, but it has been generalized to repair of multiple failed nodes.

There are generally two different models for repairing multiple failed nodes: the centralized repair model #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR and the cooperative repair model #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR .

Rack-Aware Minimum-Storage Regenerating Codes With Optimal Access

Abstract: We derive a lower bound on the amount of information accessed to repair failed nodes within a single rack from any number of helper racks in the rack-aware storage model that allows collective information processing in the nodes that share the same rack. Furthermore, we construct a family of rack-aware minimum-storage regenerating (MSR) codes with the property that the number of symbols accessed for repairing a single failed node attains the bound with equality for all admissible parameters. Constructions of rack-aware optimal-access MSR codes were only known for limited parameters. We also present a family of Reed-Solomon (RS) codes that only require accessing a relatively small number of symbols to repair multiple failed nodes in a single rack. In particular, for certain code parameters, the RS construction attains the bound on the access complexity with equality and thus has optimal access.

Network Coding For Distributed Storage Systems

Abstract: Peer-to-peer distributed storage systems provide reliable access to data through redundancy spread over nodes across the Internet. A key goal is to minimize the amount of bandwidth used to maintain that redundancy. Storing a file using an erasure code, in fragments spread across nodes, promises to require less redundancy and hence less maintenance bandwidth than simple replication to provide the same level of reliability. However, since fragments must be periodically replaced as nodes fail, a key question is how to generate a new fragment in a distributed way while transferring as little data as possible across the network. In this paper, we introduce a general technique to analyze storage architectures that combine any form of coding and replication, as well as presenting two new schemes for maintaining redundancy using erasure codes. First, we show how to optimally generate MDS fragments directly from existing fragments in the system. Second, we introduce a new scheme called regenerating codes which use slightly larger fragments than MDS but have lower overall bandwidth use. We also show through simulation that in realistic environments, regenerating codes can reduce maintenance bandwidth use by 25% or more compared with the best previous design - a hybrid of replication and erasure codes - while simplifying system architecture.


This definition is consistent with that given by the American Statistical Association [6] and with other work in statistics #REFR .

From this follows the desideratum that scientific predictions also agree well with experimental observations made across sufficiently similar circumstances. Defining this rigorously is not straightforward.

In 2019 the USA's National Academies established a Committee on Reproducibility and Replicability in Science.

Their report #OTHEREFR is essential reading for those interested in this topic.

Though their focus is primarily on scientific hypotheses, their definition is a good starting point for this discussion.

Conclusion 3-1 on page 36 states: "Replicability is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data".

A distinct concept is that of repeatability: a repeatable prediction approach produces predictions without variation across independent tests carried out by repeating the entire process, including data collection, on the same individual or sampling unit #OTHEREFR . This is important but is not examined here.

Replicability is also used in contrast to reproducibility, defined as "obtaining consistent results using the same input data, computational steps, methods, and code, and conditions of analysis" #OTHEREFR .

Usage of these terms is often inconsistent and is plainly reversed in computer science -a sarcastic twist in the parallel evolution of the same concepts in siloed fields.

A thread of literature debates and documents related terminologies and their usage #OTHEREFR .

For predictions, I propose to modify the NAS definition to say:

Defining Replicability Of Prediction Rules

Abstract: In this article I propose an approach for defining replicability for prediction rules. Motivated by a recent NAS report, I start from the perspective that replicability is obtaining consistent results across studies suitable to address the same prediction question, each of which has obtained its own data. I then discuss concept and issues in defining key elements of this statement. I focus specifically on the meaning of "consistent results" in typical utilization contexts, and propose a multi-agent framework for defining replicability, in which agents are neither partners nor adversaries. I recover some of the prevalent practical approaches as special cases. I hope to provide guidance for a more systematic assessment of replicability in machine learning.

Deciding Whether Follow-Up Studies Have Replicated Findings In A Preliminary Large-Scale"Omics' Study"

Abstract: We propose a formal method to declare that findings from a primary study have been replicated in a follow-up study. Our proposal is appropriate for primary studies that involve large-scale searches for rare true positives (i.e., needles in a haystack). Our proposal assigns an r value to each finding; this is the lowest false discovery rate at which the finding can be called replicated. Examples are given and software is available.


Specifically, EG3D starts with randomly sampled GAN latent codes, and then a feature generator based on Style-GAN #REFR converts latent codes into 2D features and maps them into 3D tri-planes.

We first briefly review the network architecture of a SOTA 3D network, EG3D #OTHEREFR .

Our generator G 3 , G 3 and G 3 are finetuned on EG3D.

An MLP decoder predicts features for 3D point projections on tri-planes to generate color and density.

Finally, volume rendering is used to generate an image on the orientation of the camera pose. This process can be formulated as follows:

where W 3 ∈ R 1×14×512 is the lantent code. P ∈ R 1×25 is the camera pose.

is the weight parameters of the EG3D generator G 3 . I is the generated images. CLIP-guided Loss.

Itportrait: Image-Text Coupled 3D Portrait Domain Adaptation

Abstract: Artistic portraits have many applications [10, 12, 22, 24, 34] in our daily lives, especially in industries related to animation, art, and the metaverse. As shown in Fig. 1, artistic portraits can be regarded as a portrait domain adaptation task, which refers to transforming the artistic style, cross-species identity, and expression shape change. The current domain adaption methods are mainly divided into two categories: guided by the vision-based method (artistic-image [6, 21, 34]), or guided by the language-based method (text-description [23, 36]). Combining image-guided and text-driven guidance can not only transfer the precise and detailed style of the reference image but also have the text-driven flexible editing ability. Therefore, Image-Text coupled guidance has better style control-ability and artistic merit [8, 27]. However, the potential of Vision-Language (Image-Text) multi-modal guidance is under-explored.

A Style-Based Generator Architecture For Generative Adversarial Networks

Abstract: We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.


For Blended #REFR , we blend the poisoned samples with a Hello-Ketty image and the blended ratio is 0.1.

We evaluate our method on PreAct-ResNet18 #OTHEREFR and VGG19-BN #OTHEREFR networks.

We compare our method with SOTA defense methods on three datasets and the two networks with a 10% poisining ratio and 5% clean samples for defense.

To study the effectiveness of our method under different poisoning ratios, we compare with SOTA defense methods on CIFAR-10 dataset and PreAct-ResNet18 network on 5% and 1% poisoning ratios.

Attack Details. We introduce some details about the backdoor attacks here.

For BadNets-A2O and BadNets-A2A [12], we patch a 3 × 3 white square in the lower right corner of the images for CIFAR-10 and GTSRB datasets, and 6 × 6 white square for Tiny ImageNet.

The seven SOTA defense methods can be divided into two types based on what the defender is given.

AC [4] and ABL #OTHEREFR assumes that the defender is given a poisoned dataset, while the remaining six defense methods assumes that the defender can acquire a subset of clean samples and a backdoored model.

The learning rate for all methods is set to 0.01, and the batch size is set to 256.

The threshold for ANP #OTHEREFR is set to 0.4 since we find that the recommended threshold 0.2 fails to remove backdoors.

Enhancing Fine-Tuning Based Backdoor Defense With Sharpness-Aware Minimization

Abstract: Backdoor defense, which aims to detect or mitigate the effect of malicious triggers introduced by attackers, is becoming increasingly critical for machine learning security and integrity. Fine-tuning based on benign data is a natural defense to erase the backdoor effect in a backdoored model. However, recent studies show that, given limited benign data, vanilla fine-tuning has poor defense performance. In this work, we provide a deep study of fine-tuning the backdoored model from the neuron perspective and find that backdoorrelated neurons fail to escape the local minimum in the fine-tuning process. Inspired by observing that the backdoorrelated neurons often have larger norms, we propose FT-SAM, a novel backdoor defense paradigm that aims to shrink the norms of backdoor-related neurons by incorporating sharpness-aware minimization with fine-tuning. We demonstrate the effectiveness of our method on several benchmark datasets and network architectures, where it achieves stateof-the-art defense performance. Overall, our work provides a promising avenue for improving the robustness of machine learning models against backdoor attacks.

Targeted Backdoor Attacks On Deep Learning Systems Using Data Poisoning

Abstract: Deep learning models have achieved high performance on many tasks, and thus have been applied to many security-critical scenarios. For example, deep learning-based face recognition systems have been used to authenticate users to access many security-sensitive applications like payment apps. Such usages of deep learning systems provide the adversaries with sufficient incentives to perform attacks against these systems for their adversarial purposes. In this work, we consider a new type of attacks, called backdoor attacks, where the attacker's goal is to create a backdoor into a learning-based authentication system, so that he can easily circumvent the system by leveraging the backdoor. Specifically, the adversary aims at creating backdoor instances, so that the victim learning system will be misled to classify the backdoor instances as a target label specified by the adversary. In particular, we study backdoor poisoning attacks, which achieve backdoor attacks using poisoning strategies. Different from all existing work, our studied poisoning strategies can apply under a very weak threat model: (1) the adversary has no knowledge of the model and the training set used by the victim system; (2) the attacker is allowed to inject only a small amount of poisoning samples; (3) the backdoor key is hard to notice even by human beings to achieve stealthiness. We conduct evaluation to demonstrate that a backdoor adversary can inject only around 50 poisoning samples, while achieving an attack success rate of above 90%. We are also the first work to show that a data poisoning attack can create physically implementable backdoors without touching the training process. Our work demonstrates that backdoor poisoning attacks pose real threats to a learning system, and thus highlights the importance of further investigation and proposing defense strategies against them.


Afterwards, we train the plots with Convolutional Neural Network, which is proved to be powerful in image-driven pattern recognition #REFR .

However, the problems of a high False Positive rate and unstable performance need to be addressed.

The workflow of the RPCNN model is sketched in Fig. 4 .

First, we take two classes of time windows either close to or far from the interlocks, where the window length is a tunable model parameter.

Explicitly, the interlock samples are taken as sliding windows from 1 s to 15 s before interlocks.

Second, we transform each 1-dimensional time series of the 376 channels to a 2-dimensional Recurrence Plot, which could be interpreted as a pairwise distance measure of the time series and is capable of extracting finer dynamical patterns.

The output is a probability score inside the range [0, 1] indicating the likelihood of a sample belonging to the positive (i.e. close to interlock) class.

More details such as model architecture are published in #OTHEREFR .

Typical binary classification metrics in a confusion matrix are defined and applied in our setting.

A True Positive (TP) means an interlock sample -a sample less than 15 s before an interlock -being classified as an interlock.

A False Positive (FP) means a stable sample -a sample at least 10 minutes away from an interlock -being mistaken as an interlock.

Forecasting Particle Accelerator Interruptions Using Logistic Lasso Regression

Abstract: Unforeseen particle accelerator interruptions, also known as interlocks, lead to abrupt operational changes despite being necessary safety measures. These may result in substantial loss of beam time and perhaps even equipment damage. We propose a simple yet powerful binary classification model aiming to forecast such interruptions, in the case of the High Intensity Proton Accelerator complex at the Paul Scherrer Institut. The model is formulated as logistic regression penalized by least absolute shrinkage and selection operator, based on a statistical two sample test to distinguish between unstable and stable states of the accelerator. The primary objective for receiving alarms prior to interlocks is to allow for countermeasures and reduce beam time loss. Hence, a continuous evaluation metric is developed to measure the saved beam time in any period, given the assumption that interlocks could be circumvented by reducing the beam current. The best-performing interlock-to-stable classifier can potentially increase the beam time by around 5 min in a day. Possible instrumentation for fast adjustment of the beam current is also listed and discussed.

An Introduction To Convolutional Neural Networks

Abstract: The field of machine learning has taken a dramatic twist in recent times, with the rise of the Artificial Neural Network (ANN). These biologically inspired computational models are able to far exceed the performance of previous forms of artificial intelligence in common machine learning tasks. One of the most impressive forms of ANN architecture is that of the Convolutional Neural Network (CNN). CNNs are primarily used to solve difficult image-driven pattern recognition tasks and with their precise yet simple architecture, offers a simplified method of getting started with ANNs. ::: This document provides a brief introduction to CNNs, discussing recently published papers and newly formed techniques in developing these brilliantly fantastic image recognition models. This introduction assumes you are familiar with the fundamentals of ANNs and machine learning.


In this section we study BERT #REFR finetuning, and use the tools developed in Section 4 to clearly differentiate the behavior of BERT-Large from BERT-Base.

For our experiment, we finetune pretrained checkpoints of both models 1,000 times each on the MRPC #OTHEREFR task.

MRPC contains 5,801 sentence pairs labeled yes/no for whether the second sentence paraphrases the first, and has a training set of 3,668 examples, validation set of 408 examples, and test set of 1,725 examples.

Previous works #OTHEREFR report and investigate training instability for BERT finetuning, noting particular instability for BERT-Large.

In our experiment, we find that both BERT-Large and BERT-Base have substantial variance in validation-set performance between runs.

BERT-Base has a standard deviation of 0.80% and BERT-Large has 2.24%, which seems to imply that both models are unstable, with BERT-Large being only somewhat moreso.

Calibrated Chaos: Variance Between Runs Of Neural Network Training Is Harmless And Inevitable

Abstract: Typical neural network trainings have substantial variance in test-set performance between repeated runs, impeding hyperparameter comparison and training reproducibility. We present the following results towards understanding this variation. (1) Despite having significant variance on their test-sets, we demonstrate that standard CIFAR-10 and ImageNet trainings have very little variance in their performance on the test-distributions from which their test-sets are sampled, suggesting that variance is less of a practical issue than previously thought. (2) We present a simplifying statistical assumption which closely approximates the structure of the test-set accuracy distribution. (3) We argue that test-set variance is inevitable in the following two senses. First, we show that variance is largely caused by high sensitivity of the training process to initial conditions, rather than by specific sources of randomness like the data order and augmentations. Second, we prove that variance is unavoidable given the observation that ensembles of trained networks are well-calibrated. (4) We conduct preliminary studies of distribution-shift, fine-tuning, data augmentation and learning rate through the lens of variance between runs.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


Compatibility Of Our Methods To Other Types Of Attacks

The results show that direction tuning attacks are perfectly Table 5 : The ASR (%) comparison on six models on ImageNet #REFR .

In this subsection, we combine our methods with a feature importanceaware attack (i.e., FIA #OTHEREFR ), three input transformation methods (i.e., DIM #OTHEREFR , TIM #OTHEREFR and SIM #OTHEREFR ), SGM #OTHEREFR and More Bayesian (MB) #OTHEREFR to verify the compatibility of our methods (including both direction tuning attack and network pruning method).

Table 4 shows the ASR on six victim models when the gradient-based attacks are combined with the feature importance-aware attack (i.e., FIA #OTHEREFR ).

The results show that our DTA and VDTA can assist FIA to achieve the greatest ASR in comparison with other gradient-based attacks (i.e., MI/NIFGSM and their variance tuning-based version).

Besides, our DTA also spends less time consumption in comparison with VMI/VNIFGSM, in which the gradient of input in our method is computed 100 times (i.e., K•T = 10×10), while that in variance tuning-based gradient attacks is 200 times (i.e., N • T = 20 × 10).

Table 5 shows the ASR on six victim models when our methods, including DTA, VDTA, and NP, are combined with three input transformation methods, respectively.

The gradient-based attacks are enhanced by DIM #OTHEREFR , SIM #OTHEREFR and TIM #OTHEREFR , respectively.

The adversarial examples are generated by the surrogate model, ResNet50 #OTHEREFR . * denotes the ASR under the white-box setting. Avg. means the average value except *.

compatible with each input transformation method, and the network pruning method can further improve the transferability of adversarial examples.

In particular, the combination of both DTA+NP and DIM #OTHEREFR not only has the highest transferability but also has less time consumption in comparison with variance tuning-based gradient attacks.

Note that, in this evaluation, VDTA computes the gradient of input 2000 times (i.e., K • N • T = 10 × 20 × 10).

Improving The Transferability Of Adversarial Examples Via Direction Tuning

Abstract: ing the oscillating component. By doing so, our direction tuning attack can achieve better convergence and enhance the transferability of the generated adversarial examples. In addition, a network pruning method is proposed to smooth the decision boundary, thereby further decreasing the update oscillation and enhancing the transferability of the generated adversarial examples. The experiment results on ImageNet demonstrate that the average attack success rate (ASR) of the adversarial examples generated by our method can be improved from 87.9% to 94.5% on five victim models without defenses, and from 69.1% to 76.2% on eight advanced defense methods, in comparison with that of latest gradient-based attacks.

Imagenet Large Scale Visual Recognition Challenge

Abstract: The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the five years of the challenge, and propose future directions and improvements.


With the emitted color c and volume density σ predicted by the rendering network, novel views can be synthesized via volume rendering, which is implemented with differential ray marching as in NeRF #REFR .

4constructed from the encoder f θ is fed into the NeRF decoder g φ for predicting NeRF color and density, as formulated in Eq. (1). Rendering network.

We follow prior works #OTHEREFR to construct a MLP-based rendering network.

Similarly, we also include the texture priors by concatenating the color information sampled on all input views with the given position.

But unlike the typical MLP decoder that processes all points on a ray independently, we further explore introducing cross-point interactions by fusing the rendered information along a ray via a Transformer.

We adopt IBRNet's #OTHEREFR ray Transformer in our implementation for convenience. Volume rendering.

Specifically, to estimate the color C of a pixel, radiance needs to be accumulated across all sampled shading points on the corresponding ray that passes through the pixel,

where c i , σ i refer to the color and density of the i-th sampled 3D point on the ray.

T i is the volume transmittance and δ i denotes the distances between adjacent points.

K is the total number of sampled 3D points on a ray.

Explicit Correspondence Matching For Generalizable Neural Radiance Fields

Abstract: We present a new generalizable NeRF method that is able to directly generalize to new unseen scenarios and perform novel view synthesis with as few as two source views. The key to our approach lies in the explicitly modeled correspondence matching information, so as to provide the geometry prior to the prediction of NeRF color and density for volume rendering. The explicit correspondence matching is quantified with the cosine similarity between image features sampled at the 2D projections of a 3D point on different views, which is able to provide reliable cues about the surface geometry. Unlike previous methods where image features are extracted independently for each view, we consider modeling the cross-view interactions via Transformer cross-attention, which greatly improves the feature matching quality. Our method achieves state-of-the-art results on different evaluation settings, with the experiments showing a strong correlation between our learned cosine feature similarity and volume density, demonstrating the effectiveness and superiority of our proposed method. Code is at https://github.com/donydchen/matchnerf.

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.


According to #REFR , a way to improve the model's performance is by increasing the number of steps in the training.

We fine-tune our models based on two criteria: a) Sequence Length: After analyzing in detail the outputs generated by previous works, we observed that some of the code sequences produced by the models were incomplete compared to the target ones.

Consequently, we tokenized the training and validation sets with SentencePiece model #OTHEREFR .

We then computed the largest sequence data, and used its length for both the inputs and the targets.

b) Number of steps: Since we increased the length of sequences in our model, we increased the number of fine-tuning steps.

We apply both criteria initializing the fine-tuning from CoTexT checkpoints 2CC and 1CC, respectively.

CoTexT-1CC is pretrained on unimodal data (only code), and CoTexT-2CC is pretrained on bimodal data (both code and natural language).

Results of these experiments are shown in Table III .

Jacotext: A Pretrained Model For Java Code-Text Generation

Abstract: Pretrained transformer-based models have shown high performance in natural language generation task. However, a new wave of interest has surged: automatic programming language generation. This task consists of translating natural language instructions to a programming code. Despite the fact that well-known pretrained models on language generation have achieved good performance in learning programming languages, effort is still needed in automatic code generation. In this paper, we introduce JaCoText, a model based on Transformers neural network. It aims to generate java source code from natural language text. JaCoText leverages advantages of both natural language and code generation models. More specifically, we study some findings from the state of the art and use them to (1) initialize our model from powerful pretrained models, (2) explore additional pretraining on our java dataset, (3) carry out experiments combining the unimodal and bimodal data in the training, and (4) scale the input and output length during the fine-tuning of the model. Conducted experiments on CONCODE dataset show that JaCoText achieves new state-of-the-art results.

Exploring The Limits Of Transfer Learning With A Unified Text-To-Text Transformer

Abstract: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.


This is similar to studies #REFR Team et al., ai) that have leveraged upsampling for under-represented languages during pre-training large language models in machine translation.

Relatively Equal Model Parameters: The XLS-R model consists of 315, 703, 690 model parameters while the Wav2Vec2.0-Large model has 315, 693, 962 -a requirement enforced in order to ensure that neither model has an edge over the other based on their size.

Handling Class Imbalance: The AfroDigits dataset currently has very small audio samples for our focus languages, with an unequal balance of digits for each language (see Figures 5 -16) .

In order to prevent the model from overfitting on the classes with many samples, we implemented weighted sampling #OTHEREFR .

With weighted sampling in the data loading process, different from the normal sampling which favors the majority classes #OTHEREFR , the labels are chosen with a probability inversely proportional to their size in the training set.

This means that at each training step and for each language, the labels with few samples are more likely to be chosen for training the model.

Training Setup: All audio samples were resampled to 16kHz for the finetuning experiments.

We froze the encoders of each model and finetuned for 100 epochs.

We used the Adam optimizer Kingma & Ba (2015), with a learning rate of 3e − 5 for both models.

We did not do any search for optimal hyperparameters but instead used the recommended settings from the authors.

We ran our finetuning experiments with five different seeds, then we took the average over the different runs as well as the standard deviation.

Afrodigits: A Community-Driven Spoken Digit Dataset For African Languages

Abstract: The advancement of speech technologies has been remarkable, yet its integration with African languages remains limited due to the scarcity of African speech corpora. To address this issue, we present AfroDigits, a minimalist, community-driven dataset of spoken digits for African languages, currently covering 38 African languages. As a demonstration of the practical applications of AfroDigits, we conduct audio digit classification experiments on six African languages [Igbo (ibo), Yoruba (yor), Rundi (run), Oshiwambo (kua), Shona (sna), and Oromo (gax)] using the Wav2Vec2.0-Large and XLS-R models. Our experiments reveal a useful insight on the effect of mixing African speech corpora during finetuning. AfroDigits is the first published audio digit dataset for African languages and we believe it will, among other things, pave the way for Afro-centric speech applications such as the recognition of telephone numbers, and street numbers. We release the dataset and platform publicly at https://huggingface.co/datasets/ chrisjay/crowd-speech-africa and https://huggingface.co/ spaces/chrisjay/afro-speech respectively.

Massively Multilingual Neural Machine Translation In The Wild: Findings And Challenges

Abstract: We introduce our efforts towards building a universal neural machine translation (NMT) system capable of translating between any language pair. We set a milestone towards this goal by building a single massively multilingual NMT model handling 103 languages trained on over 25 billion examples. Our system demonstrates effective transfer learning ability, significantly improving translation quality of low-resource languages, while keeping high-resource language translation quality on-par with competitive bilingual baselines. We provide in-depth analysis of various aspects of model building that are crucial to achieving quality and practicality in universal NMT. While we prototype a high-quality universal translation system, our extensive empirical analysis exposes issues that need to be further addressed, and we suggest directions for future research.

under-represented languages
machine translation


The reason for using a soft label is that a soft label usually provides more information #REFR .

Note that p i is a soft pseudo label rather than a hard pseudo label.

By using the proposed test time self-distillation, the network could map the uniformity for the current samples to improve the quality of representations.

Although we use an entropy filter to drop noisy labels when computing prototypes, there are still some mistake predictions inevitable.

We propose that, for a reliable sample, the outputs of the linear fully connected layer and prototype-based classifier should be similar.

Hence, we adopt the consistency filter to identify the mistake predictions.

Specially, if the linear classifier and prototype-based classifier produce the same predictions, i.e.

Feature Alignment And Uniformity For Test Time Adaptation

Abstract: Test time adaptation (TTA) aims to adapt deep neural networks when receiving out of distribution test domain samples. In this setting, the model can only access online unlabeled test samples and pre-trained models on the training domains. We first address TTA as a feature revision problem due to the domain gap between source domains and target domains. After that, we follow the two measurements alignment and uniformity to discuss the test time feature revision. For test time feature uniformity, we propose a test time self-distillation strategy to guarantee the consistency of uniformity between representations of the current batch and all the previous batches. For test time feature alignment, we propose a memorized spatial local clustering strategy to align the representations among the neighborhood samples for the upcoming batch. To deal with the common noisy label problem, we propound the entropy and consistency filters to select and drop the possible noisy labels. To prove the scalability and efficacy of our method, we conduct experiments on four domain generalization benchmarks and four medical image segmentation tasks with various backbones. Experiment results show that our method not only improves baseline stably but also outperforms existing state-of-the-art test time adaptation methods.

Distilling The Knowledge In A Neural Network

Abstract: A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.