As for the aggregation of the global model, we adopt the weighted averaging strategy in FedAvg #REFR . Site Contrastive based Channel Selection.

At round t, all sites receive the same parameters (φ t−1 g ,θ t−1 g ) from the server.

The global part is initialized with φ t−1 g , and the personalized part is initialized withθ t k which is obtained from the AA module based on θ t−1 g and the local parameters from the previous round (e.g., θ t−1 k for site k).

Each site updates its model by optimizing the local objective with its own data and its site encoding c k utilized in SCR

where GRD(•) denotes the local gradient-based update.

Personalized FL paradigms may suffer from confusion or over-personalization when data heterogeneity is low, performing even worse than traditional FL methods #OTHEREFR .

Hence, the SCR module is designed to enhance the distance/contrast of inter-site data representations through site-contrastive learning based channel attention, which in turn facilitates personalization.

Specifically, taking the k-th site as an example, a one-hot site encoding c k (i.e., the k-th position is 1 and others are 0) and the output feature f k from the encoder F e are given.

c k is expanded to a length of C through two fully connected layers to obtain c * k , which is then concatenated with the global average pooled feature of f k .

After passing through a fully connected layer with Sigmoid activation, the site channel attention valueĉ k is obtained.

Unifying And Personalizing Weakly-Supervised Federated Medical Image Segmentation Via Adaptive Representation And Aggregation

Abstract: Federated learning (FL) enables multiple sites to collaboratively train powerful deep models without compromising data privacy and security. The statistical heterogeneity (e.g., non-IID data and domain shifts) is a primary obstacle in FL, impairing the generalization performance of the global model. Weakly supervised segmentation, which uses sparsely-grained (i.e., point-, bounding box-, scribble-, block-wise) supervision, is increasingly being paid attention to due to its great potential of reducing annotation costs. However, there may exist label heterogeneity, i.e., different annotation forms across sites. In this paper, we propose a novel personalized FL framework for medical image segmentation, named FedICRA, which uniformly leverages heterogeneous weak supervision via adaptIve Contrastive Representation and Aggregation. Concretely, to facilitate personalized modeling and to avoid confusion, a channel selection based site contrastive representation module is employed to adaptively cluster intra-site embeddings and separate inter-site ones. To effectively integrate the common knowledge from the global model with the unique knowledge from each local model, an adaptive aggregation module is applied for updating and initializing local models at the element level. Additionally, a weakly supervised objective function that leverages a multiscale tree energy loss and a gated CRF loss is employed to generate more precise pseudo-labels and further boost the segmentation performance. Through extensive experiments on two distinct medical image segmentation tasks of different modalities, the proposed FedICRA demonstrates overwhelming performance over other state-ofthe-art personalized FL methods. Its performance even approaches that of fully supervised training on centralized data. Our code and data are available at https://github.com/llmir/FedICRA.

Communication-Efficient Learning Of Deep Networks From Decentralized Data

Abstract: Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning. We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.


In this regard, nonorthogonal multiple access (NOMA) is seen as a strong candidate for IoT networks #REFR since it allows multiple devices to share the same resource blocks by splitting them into the power domain.

For instance, in 5G new radio (NR) standards #OTHEREFR , with a 20 MHz bandwidth (i.e., maximum bandwidth in LTE legacy ), a maximum number of 92 RBs 1 can be allocated to users within a subframe.

However, in massive IoT applications (e.g., smart agriculture), thousands of sensor or control nodes may require wireless access.

In these cases, whether we need more bandwidth, that is costly or more complex resource allocation (RB scheduling algorithms in layer 2) solutions are required.

Therefore, it is not possible to allocate each IoT device to an orthogonal resource to avoid interference, since a massive number of devices need to be served in small, dense areas.

For this reason, more than one IoT devices should share a resource block to enable mMTC.

In this way, the spectral efficiency of the network increases and it becomes possible to serve multiple devices with the number of more than the available resource blocks #OTHEREFR .

Accordingly, NOMA is seen as an enabler for ultra-dense networks, and tremendous efforts have been devoted to integrating NOMA in IoT applications #OTHEREFR - #OTHEREFR .

Bit-Interleaved Multiple Access: Improved Fairness, Reliability, And Latency For Massive Iot Networks

Abstract: Internet-of-Things (IoT) networks require massive connections in dense areas. Therefore, a resource efficient multiple access scheme seems inevitable to enable immense connectivity where multiple devices have to share the same resource block. Non-orthogonal multiple access (NOMA) has been considered as the strongest candidate in recent years. However, in this paper, by considering the practical implementation, we first provide a true power allocation (PA) constraint with finite alphabet inputs for conventional downlink NOMA and demonstrate that it cannot support massive connections in practical systems. To this end, we propose bit-interleaved multiple access (BIMA) scheme in downlink IoT networks. The proposed BIMA scheme implements bitwise multiaccess interleaving and deinterleaving at the transceiver ends and there are no strict PA constraints, unlike conventional NOMA, thus allowing a high number of devices in the same resource block. We provide a comprehensive analytical framework for BIMA by investigating all key performance indicators (KPIs) to present both information-theoretic (i.e., ergodic capacity [EC] and outage probability [OP]) and finite alphabet inputs (i.e., bit error rate [BER]) performance metrics with both instantaneous and statistical channel ordering. In addition, we define Jain's fairness index and proportional fairness index in terms of all KPIs. Based on the extensive computer simulations, we reveal that BIMA outperforms conventional NOMA significantly, with a performance gain of up to 20-30 dB in terms of KPIs in some scenarios. In other words, compared to conventional NOMA schemes, the same KPIs are met in BIMA with 20-30 dB less transmit power, which is quite promising for energy-limited use cases. Moreover, this performance gain becomes greater when more IoT devices are supported. BIMA provides a full diversity order for all IoT devices and enables the implementation of an arbitrary number of devices and modulation orders, which is crucial for IoT networks where a huge number of devices should be supported in a single resource block in dense areas. In addition to the overall performance gain, BIMA guarantees a fairness system where none of the devices gets a severely degraded performance and the sum-rate is shared in a fair manner among devices. It guarantees QoS satisfaction for all devices. Finally, we provide an intense complexity and latency analysis for BIMA and demonstrate that it provides lower latency compared to conventional NOMA receivers, since it allows parallel computation at the receivers and no iterative operations are required. We show that compared to conventional NOMA receivers, BIMA reduces latency by up to 350% for specific IoT devices and 170% on average.

Massive Non-Orthogonal Multiple Access For Cellular Iot: Potentials And Limitations

Abstract: The Internet of Things promises ubiquitous connectivity of everything everywhere, which represents the biggest technology trend in the years to come. It is expected that by 2020 over 25 billion devices will be connected to cellular networks; far beyond the number of devices in current wireless networks. Machine-to-machine communications aims to provide the communication infrastructure for enabling IoT by facilitating the billions of multi-role devices to communicate with each other and with the underlying data transport infrastructure without, or with little, human intervention. Providing this infrastructure will require a dramatic shift from the current protocols mostly designed for human-to-human applications. This article reviews recent 3GPP solutions for enabling massive cellular IoT and investigates the random access strategies for M2M communications, which shows that cellular networks must evolve to handle the new ways in which devices will connect and communicate with the system. A massive non-orthogonal multiple access technique is then presented as a promising solution to support a massive number of IoT devices in cellular networks, where we also identify its practical challenges and future research directions.

IoT networks
nonorthogonal multiple access


Since a ranked list of top-K items is required in both stages, Bayesian Personalized Ranking (BPR) #REFR pairwise learning is adopted to optimize the parameters of PEGA.

We leverage a two-stage training strategy to alleviate the sparsity issue of group-item interaction.

BPR pairwise learning aims to maximize the score difference between positive and negative items.

Specifically, we obtain the user and the item embeddings by minimizing the user-level BPR pairwise loss L :

where O is the set of user training instances.

Each instance , , contains a positive item that the user has interacted with and a negative item that the user hasn't interacted with yet.

Pega: Personality-Guided Preference Aggregator For Ephemeral Group Recommendation

Abstract: Recently, making recommendations for ephemeral groups which contain dynamic users and few historic interactions have received an increasing number of attention. The main challenge of ephemeral group recommender is how to aggregate individual preferences to represent the group's overall preference. Score aggregation and preference aggregation are two commonly-used methods that adopt hand-craft predefined strategies and data-driven strategies, respectively. However, they neglect to take into account the importance of the individual inherent factors such as personality in the group. In addition, they fail to work well due to a small number of interactive records. To address these issues, we propose a Personality-Guided Preference Aggregator (PEGA) for ephemeral group recommendation. Concretely, we first adopt hyper-rectangle to define the concept of Group Personality. We then use the personality attention mechanism to aggregate group preferences. The role of personality in our approach is twofold: (1) To estimate individual users' importance in a group and provide explainability; (2) to alleviate the data sparsity issue that occurred in ephemeral groups. The experimental results demonstrate that our model significantly outperforms the state-of-the-art methods w.r.t. the score of both Recall and NDCG on Amazon and Yelp datasets. CCS CONCEPTS • Information systems → Recommender systems; • Computing methodologies → Neural networks.

Bpr: Bayesian Personalized Ranking From Implicit Feedback

Abstract: Item recommendation is the task of predicting a personalized ranking on a set of items (e.g. websites, movies, products). In this paper, we investigate the most common scenario with implicit feedback (e.g. clicks, purchases). There are many methods for item recommendation from implicit feedback like matrix factorization (MF) or adaptive k-nearest-neighbor (kNN). Even though these methods are designed for the item prediction task of personalized ranking, none of them is directly optimized for ranking. In this paper we present a generic optimization criterion BPR-Opt for personalized ranking that is the maximum posterior estimator derived from a Bayesian analysis of the problem. We also provide a generic learning algorithm for optimizing models with respect to BPR-Opt. The learning method is based on stochastic gradient descent with bootstrap sampling. We show how to apply our method to two state-of-the-art recommender models: matrix factorization and adaptive kNN. Our experiments indicate that for the task of personalized ranking our optimization method outperforms the standard learning techniques for MF and kNN. The results show the importance of optimizing models for the right criterion.


We train our prompt-based image captioner for 40 epochs from scratch, using the weight initialization strategy described in #REFR .

We explore two different prompt-based image captioning models, i.e., SAT #OTHEREFR and GRIT #OTHEREFR to act as classifiers.

To train our promptbased image captioner, we split the original validation set, around 10 4 images, into training, validation, and testing sets, i.e., 70%, 10%, and 20%, respectively. These splits are balanced based on the protected attribute.

All captioning models are trained using the same training configurations mentioned in #OTHEREFR .

The added template T is "Therefore, the gender is [Answer]", "Therefore, The race is [Answer]" and "Therefore, the emotion is [Answer]" for gender, race, and emotions, respectively.

Adam optimizer #OTHEREFR and mini-batch size of 128 are used for training all our models.

Our metric is implemented in Python using the PyTorch framework. All experiments are conducted using four NVIDIA V100 GPUs.

Imagecaptioner 2 : Image Captioner For Image Captioning Bias Amplification Assessment

Abstract: Most pre-trained learning systems are known to suffer from bias, which typically emerges from the data, the model, or both. Measuring and quantifying bias and its sources is a challenging task and has been extensively studied in image captioning. Despite the significant effort in this direction, we observed that existing metrics lack consistency in the inclusion of the visual signal. In this paper, we introduce a new bias assessment metric, dubbed ImageCaptioner 2 , for image captioning. Instead of measuring the absolute bias in the model or the data, ImageCaptioner 2 pay more attention to the bias introduced by the model w.r.t the data bias, termed bias amplification. Unlike the existing methods, which only evaluate the image captioning algorithms based on the generated captions only, ImageCaptioner 2 incorporates the image while measuring the bias. In addition, we design a formulation for measuring the bias of generated captions as promptbased image captioning instead of using language classifiers. Finally, we apply our ImageCaptioner 2 metric across 11 different image captioning architectures on three different datasets, i.e., MS-COCO caption dataset, Artemis V1, and Artemis V2, and on three different protected attributes, i.e., gender, race, and emotions. Consequently, we verify the effectiveness of our ImageCaptioner 2 metric by proposing AnonymousBench, which is a novel human evaluation paradigm for bias metrics. Our metric shows significant superiority over the recent bias metric; LIC, in terms of human alignment, where the correlation scores are 80% and 54% for our metric and LIC, respectively. The code is available at https://eslambakr.github.io/imagecaptioner2.github.io/. Recent efforts focus on estimating model bias, driven by the fact that more than balanced data is needed to create

Delving Deep Into Rectifiers: Surpassing Human-Level Performance On Imagenet Classification

Abstract: Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work, we study rectifier neural networks for image classification from two aspects. First, we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit. PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities. This method enables us to train extremely deep rectified models directly from scratch and to investigate deeper or wider network architectures. Based on the learnable activation and advanced initialization, we achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset. This is a 26% relative improvement over the ILSVRC 2014 winner (GoogLeNet, 6.66% [33]). To our knowledge, our result is the first to surpass the reported human-level performance (5.1%, [26]) on this dataset.


There is evidence that artificial neural networks are overparameterized at the start of training and functionally sparse, as supported by results from network pruning, most notably the Lottery Ticket Hypothesis #REFR .

This is because the ReLU MLP is not constrained to the data manifold, having a bias term and no L 2 normalization.

This allows it to learn weights and biases that better maximize NCL validation accuracy.

If we perform joint training of the ConvMixer and SDM module then the ConvMixer can learn to create a manifold for SDM to tile that does maximize validation accuracy, performing on par with the ReLU MLP.

This is what happens in a test where we train on the whole CIFAR10 dataset in the NCL setting, as long as k ≥ 250, but independent of if there are 1,000 or 10,000 neurons in the SDM layer. This result is shown in Fig.

14a and suggests that the ReLU network only needs at least 250 neurons in its final layer to backpropagate gradients and achieve high prediction accuracy.

However, this result is again manifold dependent whereby training instead on ImageNet32, even with k = 2, 500 and 10,000 neurons still harms performance compared to a ReLU network.

We believe this is because ImageNet32 has a dramatically more complex data manifold with ∼ 1.2M images from 1,000 different classes, compared to 50,000 images in 10 classes for CIFAR10.

This means that even a k of 2500 is too small and harms the model's representational capacity.

Figure 14 : Sufficiently small k values harm network training.

ConvMixers with an SDM layer at the end trained on CIFAR10.

Sparse Distributed Memory Is A Continual Learner

Abstract: Continual learning is a problem for artificial neural networks that their biological counterparts are adept at solving. Building on work using Sparse Distributed Memory (SDM) to connect a core neural circuit with the powerful Transformer model, we create a modified Multi-Layered Perceptron (MLP) that is a strong continual learner. We find that every component of our MLP variant translated from biology is necessary for continual learning. Our solution is also free from any memory replay or task information, and introduces novel methods to train sparse networks that may be broadly applicable.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Abstract: Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance. We find that a standard technique for pruning weights naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the"lottery ticket hypothesis": dense, randomly-initialized feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - arrive at comparable test accuracy in a comparable number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective. We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Furthermore, the winning tickets we find above that size learn faster than the original network and exhibit higher test accuracy.

artificial neural networks
network pruning


Furthermore, self-BLEU #REFR scores indicate EDA is the least effective in ensuring candidate diversity compared to other approaches.

Our ranking method generates increasingly lexically diverse schemata as shown by the increase in Jaccard distance across schema variants (Table 1) .

This aspect is much more difficult to achieve with EDA without significantly affecting semantics.

The BLEU difference between the SGD-X variants v1 and v5 is 15.2 but smaller (0.66) for our approach.

Hence, the PEGASUS + BART copies n-grams from the input and includes additional information.

This information is not always meaning-preserving: City where the event is happening is paraphrased as The bustling city where the event is taking place (v5) but End date for the reservation or to find the house is paraphrased as End date for hotel reservation to allow time for a replacement both at the struck and in the run up to the event (v5).

The self-BLEU of the SGD-X schemas decreases faster compared to the automatically generated paraphrases, suggesting that Jaccard distance increases partly due to hallucination.

Entailment scores show that backtranslation is effective in preserving semantics.

More Robust Schema-Guided Dialogue State Tracking Via Tree-Based Paraphrase Ranking

Abstract: The schema-guided paradigm overcomes scalability issues inherent in building task-oriented dialogue (TOD) agents with static ontologies. Instead of operating on dialogue context alone, agents have access to hierarchical schemas containing task-relevant natural language descriptions. Fine-tuned language models excel at schema-guided dialogue state tracking (DST) but are sensitive to the writing style of the schemas. We explore methods for improving the robustness of DST models. We propose a framework 1 for generating synthetic schemas which uses tree-based ranking to jointly optimise lexical diversity and semantic faithfulness. The generalisation of strong baselines is improved when augmenting their training data with prompts generated by our framework, as demonstrated by marked improvements in average joint goal accuracy (JGA) and schema sensitivity (SS) on the SGD-X benchmark.

Texygen: A Benchmarking Platform For Text Generation Models

Abstract: We introduce Texygen, a benchmarking platform to support research on open-domain text generation models. Texygen has not only implemented a majority of text generation models, but also covered a set of metrics that evaluate the diversity, the quality and the consistency of the generated texts. The Texygen platform could help standardize the research on text generation and facilitate the sharing of fine-tuned open-source implementations among researchers for their work. As a consequence, this would help in improving the reproductivity and reliability of future research work in text generation.


Before pre-training, the text encoder was initialized using DistilRoBERTa #REFR , with a maximum length of 128, and the music encoder was initialized using two settings: M3-S/512 and M3-S/1024.

Both models were trained for 40 epochs on WebMT with 6 encoder layers and 3 decoder layers, an embedding size of 768, and a mask ratio of 45%.

Based on these two M3 encoders, we developed CLaMP-S/512 and CLaMP-S/1024.

Both of them were trained for 20 epochs, using the AdamW optimizer #OTHEREFR with β 1 = 0.9, β 2 = 0.999, = 10 −8 , and a weight decay coefficient of 0.01.

The batch size is set to 640, and the temperature τ = 0.2.

The training process was accelerated and memory was saved by using mixed precision #OTHEREFR .

Clamp: Contrastive Language-Music Pre-Training For Cross-Modal Symbolic Music Information Retrieval

Abstract: We introduce CLaMP 1 : Contrastive Language-Music Pre-training, which learns cross-modal representations between natural language and symbolic music using a music encoder and a text encoder trained jointly with a contrastive loss. To pre-train CLaMP, we collected a large dataset of 1.4 million music-text pairs. It employed text dropout as a data augmentation technique and bar patching to efficiently represent music data which reduces sequence length to less than 10%. In addition, we developed a masked music model pre-training objective to enhance the music encoder's comprehension of musical context and structure. CLaMP integrates textual information to enable semantic search and zero-shot classification for symbolic music, surpassing the capabilities of previous models. To support the evaluation of semantic search and music classification, we publicly release WikiMusicText (WikiMT), a dataset of 1010 lead sheets in ABC notation, each accompanied by a title, artist, genre, and description. In comparison to state-of-the-art models that require fine-tuning, zero-shot CLaMP demonstrated comparable or superior performance on score-oriented datasets.

Distilbert, A Distilled Version Of Bert: Smaller, Faster, Cheaper And Lighter

Abstract: As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.


SOGATs correspond to the class of theories classified by representable map categories, which were introduced by Uemura #REFR .

This kind of approach does not seem suitable for the coherence theorems that we are interested in.

The main result of this paper is a coherence theorem for type theories that is analogous to the fact the statement (1) can be deduced from the statement (2).

It states that to establish the conservativity of the equational extension T → T E , it suffices to check that the ∞-congruence over T freely generated by the equations of E exists and is 0-truncated.

This 0-truncatedness condition encodes the same idea as the fact that every diagram made up of associators and unitors commutes in a freely generated monoidal category.

In general, we work at the level of second-order generalized algebraic theories (SOGATs) and their classifying (Σ, Π rep )-CwFs.

First-order generalized algebraic theories are also SOGATs, so our results also apply to theories that are not type theories, such as monoidal categories, etc.

A SOGAT T is specified by its classifying (Σ, Π rep )-CwF, also denoted by T , which can be seen as a syntactic model of some type theory.

A SOGAT has functorial semantics in (Σ, Π rep )-CwFs.

Even if one is only interested in a specific theory, working with SOGATs explicitly is advantageous.

Indeed, many semantic conditions can instead be stated more concisely and syntactically directly at the level of the (Σ, Π rep )-CwF T .

Towards Coherence Theorems For Equational Extensions Of Type Theories

Abstract: We study the conservativity of extensions by additional strict equalities of dependent type theories (and more general second-order generalized algebraic theories). The conservativity of Extensional Type Theory over Intensional Type Theory was proven by Hofmann. Our goal is to generalize such results to type theories without the Uniqueness of Identity Proofs principles, such as variants of Homotopy Type Theory. For this purpose, we construct what is essentially the ∞-congruence on the base theory that is freely generated by the considered equations. This induces a factorization of any equational extension, whose two factors can be studied independently. We conjecture that the first factor is always an equivalence when the base theory is well-behaved. We prove that the second factor is an equivalence when the ∞-congruence is 0-truncated.

A General Framework For The Semantics Of Type Theory

Abstract: We propose an abstract notion of a type theory to unify the semantics of various type theories including Martin-Lof type theory, two-level type theory and cubical type theory. We establish basic results in the semantics of type theory: every type theory has a bi-initial model; every model of a type theory has its internal language; the category of theories over a type theory is bi-equivalent to a full sub-2-category of the 2-category of models of the type theory.


While MIF can assist in diagnosis and treatment by fusing multiple medical imaging modalities for precise detection of abnormality locations #REFR .

Image fusion integrates essential information from multiple source images to create high-quality fused images #OTHEREFR , encompassing various source image types like digital #OTHEREFR , multi-modal #OTHEREFR , and remote sensing #OTHEREFR .

This technology provides a clearer representation of objects and scenes, and has diverse applications such as saliency detection #OTHEREFR , object detection #OTHEREFR , and semantic segmentation #OTHEREFR .

Among the different subcategories of image fusion, Infrared-Visible image Fusion (IVF) and Medical Image Fusion (MIF) are particularly challenging in Multi-Modality Image Fusion (MMIF) since they focus on modeling crossmodality features and preserving critical information from Figure 2 : DDFM (marked in yellow) outperforms all other methods on MSRS #OTHEREFR and RoadScene #OTHEREFR across six metrics. all sensors and modalities.

Specifically, in IVF, fused images aim to retain both thermal radiation from infrared images and detailed texture information from visible images, thereby avoiding the limitations of visible images being sensitive to illumination conditions and infrared images being noisy and low-resolution.

There have been numerous methods devised recently to address the challenges posed by MMIF #OTHEREFR , and generative models #OTHEREFR have been extensively utilized to model the distribution of fused images and achieve satisfactory fusion effects.

Among them, models based on Generative Adversarial Networks (GANs) #OTHEREFR are dominant. The workflow of GAN-based models, illustrated in Fig.

1a , involves a generator that creates images containing information from source images, and a discriminator that determines whether the generated images are in a similar manifold to the source images.

Although GAN-based methods have the ability to generate high-quality fused images, they suffer from unstable training, lack of interpretability and mode collapse, which seriously affect the quality of the generated samples.

Moreover, as a black-box model, it is difficult to comprehend the internal mechanisms and behaviors of GANs, making it challenging to achieve controllable generation.

Ddfm: Denoising Diffusion Model For Multi-Modality Image Fusion

Abstract: Multi-modality image fusion aims to combine different modalities to produce fused images that retain the complementary features of each modality, such as functional highlights and texture details. To leverage strong generative priors and address challenges such as unstable training and lack of interpretability for GAN-based generative methods, we propose a novel fusion algorithm based on the denoising diffusion probabilistic model (DDPM). The fusion task is formulated as a conditional generation problem under the DDPM sampling framework, which is further divided into an unconditional generation subproblem and a maximum likelihood subproblem. The latter is modeled in a hierarchical Bayesian manner with latent variables and inferred by the expectation-maximization algorithm. By integrating the inference solution into the diffusion sampling iteration, our method can generate high-quality fused images with natural image generative priors and cross-modality information from source images. Note that all we required is an unconditional pre-trained generative model, and no fine-tuning is needed. Our extensive experiments indicate that our approach yields promising fusion results in infrared-visible image fusion and medical image fusion. The code will be released.

Medical Image Fusion: A Survey Of The State Of The Art

Abstract: Medical image fusion is the process of registering and combining multiple images from single or multiple imaging modalities to improve the imaging quality and reduce randomness and redundancy in order to increase the clinical applicability of medical images for diagnosis and assessment of medical problems. Multi-modal medical image fusion algorithms and devices have shown notable achievements in improving clinical accuracy of decisions based on medical images. This review article provides a factual listing of methods and summarizes the broad scientific challenges faced in the field of medical image fusion. We characterize the medical image fusion research based on (1) the widely used image fusion methods, (2) imaging modalities, and (3) imaging of organs that are under study. This review concludes that even though there exists several open ended technological and scientific challenges, the fusion of medical images has proved to be useful for advancing the clinical reliability of using medical imaging for medical diagnostics and analysis, and is a scientific discipline that has the potential to significantly grow in the coming years.


In our approach, inspired by SuperGlue #REFR , we further process individual object embeddings in a graph neural network which relates objects within images and across images to encode image context.

of object bounding boxes into feature vectors which are used to measure similarity of detected objects between views using cosine similarity.

In Associative3D #OTHEREFR , also feature embeddings per objects are learned which are used to match objects between views.

Both #OTHEREFR and #OTHEREFR leverage semantic and spatial information for finding 3D correspondences.

Recently, #OTHEREFR propose to use human models to estimate correspondences among wide-baseline view changes.

Different to object associations, keypoint matching methods yield local image correspondences between pairs of images for moderate view point changes #OTHEREFR , #OTHEREFR , #OTHEREFR .

Our approach can be an alternative or complementary approach for matching objects across views to geometric or keypoint-based approaches.

Learning-Based Relational Object Matching Across Views

Abstract: Intelligent robots require object-level scene understanding to reason about possible tasks and interactions with the environment. Moreover, many perception tasks such as scene reconstruction, image retrieval, or place recognition can benefit from reasoning on the level of objects. While keypoint-based matching can yield strong results for finding correspondences for images with small to medium view point changes, for large view point changes, matching semantically on the object-level becomes advantageous. In this paper, we propose a learningbased approach which combines local keypoints with novel object-level features for matching object detections between RGB images. We train our object-level matching features based on appearance and inter-frame and cross-frame spatial relations between objects in an associative graph neural network. We demonstrate our approach in a large variety of views on realistically rendered synthetic images. Our approach compares favorably to previous state-of-the-art object-level matching approaches and achieves improved performance over a pure keypoint-based approach for large viewpoint changes.

Superglue: Learning Feature Matching With Graph Neural Networks

Abstract: This paper introduces SuperGlue, a neural network that matches two sets of local features by jointly finding correspondences and rejecting non-matchable points. Assignments are estimated by solving a differentiable optimal transport problem, whose costs are predicted by a graph neural network. We introduce a flexible context aggregation mechanism based on attention, enabling SuperGlue to reason about the underlying 3D scene and feature assignments jointly. Compared to traditional, hand-designed heuristics, our technique learns priors over geometric transformations and regularities of the 3D world through end-to-end training from image pairs. SuperGlue outperforms other learned approaches and achieves state-of-the-art results on the task of pose estimation in challenging real-world indoor and outdoor environments. The proposed method performs matching in real-time on a modern GPU and can be readily integrated into modern SfM or SLAM systems. The code and trained weights are publicly available at https://github.com/magicleap/SuperGluePretrainedNetwork.

individual object embeddings
graph neural network


Dollar Street And Coco. Applying This Insight From

As expected, finetuning GeoDE achieves a higher accuracy (where a prediction is correct if the object is one of the top-5 predicted labels #REFR ) of 13.1% on Dollar Street compared to finetuning ImageNet with 8.4%.

CelebA that relatively minor manipulations to the proportion of the dataset from underrepresented subcategories can significantly impact the performance of those subcategories, we now turn to the more complex and realistic object datasets of Dollar Street and COCO.

We consider the task of recognizing 15 objects in Dollar Street that have corresponding objects in COCO.

Although the object classes are the same between datasets, their visual distribution is different as COCO images largely come from only higher-income regions #OTHEREFR whereas Dollar Street was collected to be more geographically diverse.

Nevertheless COCO images are more plentiful; to simulate this, we consider a finetuning dataset of 128 images where 90% are from COCO and 10% from Dollar Street.

We use two pretrained models, named after the dataset each is trained on: ImageNet #OTHEREFR , where the training data is more similar to the COCO distribution, and GeoDE #OTHEREFR , which is trained on a newer and more geographically diverse dataset.

However, what we ultimately want to investigate is how much investing in a better finetuning dataset can help overcome the problem (Fig. 6) .

Thus, we manipulate the finetuning dataset (simulating the collection of more Dollar Street-like images, while keeping the overall finetuning number the same), and observe that with just 20% rather than 10% of images coming from Dollar Street, ImageNet is albe to outperform the performance of the GeoDE baseline with an accuracy of 21.7%.

Overcoming Bias In Pretrained Models By Manipulating The Finetuning Dataset

Abstract: Transfer learning is beneficial by allowing the expressive features of models pretrained on large-scale datasets to be finetuned for the target task of smaller, more domainspecific datasets. However, there is a concern that these pretrained models may come with their own biases which would propagate into the finetuned model. In this work, we investigate bias when conceptualized as both spurious correlations between the target task and a sensitive attribute as well as underrepresentation of a particular group in the dataset. Under both notions of bias, we find that (1) models finetuned on top of pretrained models can indeed inherit their biases, but (2) this bias can be corrected for through relatively minor interventions to the finetuning dataset, and often with a negligible impact to performance. Our findings imply that careful curation of the finetuning dataset is important for reducing biases on a downstream task, and doing so can even compensate for bias in the pretrained model.

Does Object Recognition Work For Everyone?

Abstract: The paper analyzes the accuracy of publicly available object-recognition systems on a geographically diverse dataset. This dataset contains household items and was designed to have a more representative geographical coverage than commonly used image datasets in object recognition. We find that the systems perform relatively poorly on household items that commonly occur in countries with a low household income. Qualitative analyses suggest the drop in performance is primarily due to appearance differences within an object class (e.g., dish soap) and due to items appearing in a different context (e.g., toothbrushes appearing outside of bathrooms). The results of our study suggest that further work is needed to make object-recognition systems work equally well for people across different countries and income levels.


The term deep hashing has been coined as an umbrella term for methods which aim at extracting compact and stable representations with deep learning techniques #REFR .

A binary vector b of length nm can be transformed back to an integer vector q of length n by mapping consecutive chunks of m bits to their decimal representation.

Further, b can be transformed to an integer set s of variable length.

For instance, this feature set can consist of all indexes of 1s in the binary vector, s = {i|b i = 1} with b i ∈ b.

This feature type transformation has been successfully applied to feature vectors obtained by deep learning-based feature extractors, e.g. in #OTHEREFR . Compact, e.g.

binary, representations (which additionally turn out to be beneficial for workload reduction in biometric identification systems #OTHEREFR ) can also be extracted by deep learning techniques.

If applied to biometric data, such methods would need to overcome intra-class variations.

Consequently, a reliable extraction of stable feature vectors (deep hashes) would enable a subsequent application of conventional (and provable secure) cryptographic algorithms for the purpose of template protection.

In the recent past, deep hashes have been extracted from different biometric characteristics in various ways, e.g. in #OTHEREFR - #OTHEREFR , including multi-biometrics #OTHEREFR .

Most of the proposed schemes, however, are focusing on facial images and have been recently surveyed in #OTHEREFR .

It is well-known that a single biometric characteristic, e.g.

Deep Learning In The Field Of Biometric Template Protection: An Overview

Abstract: Today, deep learning represents the most popular and successful form of machine learning. Deep learning has revolutionised the field of pattern recognition, including biometric recognition. Biometric systems utilising deep learning have been shown to achieve auspicious recognition accuracy, surpassing human performance. Apart from said breakthrough advances in terms of biometric performance, the use of deep learning was reported to impact different covariates of biometrics such as algorithmic fairness, vulnerability to attacks, or template protection. Technologies of biometric template protection are designed to enable a secure and privacy-preserving deployment of biometrics. In the recent past, deep learning techniques have been frequently applied in biometric template protection systems for various purposes. This work provides an overview of how advances in deep learning take influence on the field of biometric template protection. The interrelation between improved biometric performance rates and security in biometric template protection is elaborated. Further, the use of deep learning for obtaining feature representations that are suitable for biometric template protection is discussed. Novel methods that apply deep learning to achieve various goals of biometric template protection are surveyed along with deep learning-based attacks.

Hashnet: Deep Learning To Hash By Continuation

Abstract: Learning to hash has been widely applied to approximate nearest neighbor search for large-scale multimedia retrieval, due to its computation efficiency and retrieval quality. Deep learning to hash, which improves retrieval quality by end-to-end representation learning and hash encoding, has received increasing attention recently. Subject to the ill-posed gradient difficulty in the optimization with sign activations, existing deep learning to hash methods need to first learn continuous representations and then generate binary hash codes in a separated binarization step, which suffer from substantial loss of retrieval quality. This work presents HashNet, a novel deep architecture for deep learning to hash by continuation method with convergence guarantees, which learns exactly binary hash codes from imbalanced similarity data. The key idea is to attack the ill-posed gradient problem in optimizing deep networks with non-smooth binary activations by continuation method, in which we begin from learning an easier network with smoothed activation function and let it evolve during the training, until it eventually goes back to being the original, difficult to optimize, deep network with the sign activation function. Comprehensive empirical evidence shows that HashNet can generate exactly binary hash codes and yield state-of-the-art multimedia retrieval performance on standard benchmarks.


Based on this assumption, we design a new message passing mechanism by resorting to information bottleneck #REFR to only propagate class-agnostic knowledge between nodes of different classes.

If simply applying these methods to graph-structured data by individually treating each node, the topological structure and the interaction between nodes will be ignored. Recently, #OTHEREFR

(2020a) ; #OTHEREFR propose to overcome catastrophic forgetting for graph data.

However, They focus on static graph snapshots, and utilize static GNN for each snapshot, thus largely ignoring fine-grained temporal topological information.

In this paper, we put forward the first class-incremental learning approach towards open temporal dynamic graphs, called OTGNet.

To mitigate the issue of heterophily propagation, we assume the information of a node can be disentangled into class-relevant and class-agnostic one.

In this way, we can well avoid transferring conflictive information.

To prevent catastrophic knowledge forgetting over old classes, we propose to select representative sub-graph structures generated from old classes, and incorporate them into the learning process of new classes.

Previous works #OTHEREFR point out triad structure (triangle-shape structure) is a fundamental element of temporal graph and can capture evolution patterns.

Motivated by this, we devise a value function to select not only important but also diverse triad structures, and replay them for continual learning.

Due to the combinational property, optimizing the value function is NP-hard.

Towards Open Temporal Graph Neural Net-Works

Abstract: Graph neural networks (GNNs) for temporal graphs have recently attracted increasing attentions, where a common assumption is that the class set for nodes is closed. However, in real-world scenarios, it often faces the open set problem with the dynamically increased class set as the time passes by. This will bring two big challenges to the existing temporal GNN methods: (i) How to dynamically propagate appropriate information in an open temporal graph, where new class nodes are often linked to old class nodes. This case will lead to a sharp contradiction. This is because typical GNNs are prone to make the embeddings of connected nodes become similar, while we expect the embeddings of these two interactive nodes to be distinguishable since they belong to different classes. (ii) How to avoid catastrophic knowledge forgetting over old classes when learning new classes occurred in temporal graphs. In this paper, we propose a general and principled learning approach for open temporal graphs, called OTGNet, with the goal of addressing the above two challenges. We assume the knowledge of a node can be disentangled into class-relevant and class-agnostic one, and thus explore a new message passing mechanism by extending the information bottleneck principle to only propagate class-agnostic knowledge between nodes of different classes, avoiding aggregating conflictive information. Moreover, we devise a strategy to select both important and diverse triad sub-graph structures for effective class-incremental learning. Extensive experiments on three real-world datasets of different domains demonstrate the superiority of our method, compared to the baselines.

Deep Variational Information Bottleneck

Abstract: We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method"Deep Variational Information Bottleneck", or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.

class-agnostic knowledge
information bottleneck


The re-timing problem #REFR is convex, since the objective is convex, the first constraint is linear, and the second constraint can be expressed as 2(N − 1) linear inequalities.

Specifically, we limit the maximum change in adjacent scaling factors as |η j+1 − η j | ≤ κ, where κ > 0.

minimize J 1 + J 2 subject to N j=1 η j T j = T, |η j+1 − η j | ≤ κ, j = 1, . . . , N − 1. (19)

The variables are the factors η 1 , . . .

, η N , which are subject to the implicit constraint η j > 0.

More precisely, it can be verified that problem #OTHEREFR is representable as an SOCP [39, §2.3] .

Given that this problem has only N variables and sparse structure, it can be solved extremely quickly.

After solving problem #OTHEREFR previous times and path.

Independently of the success of the iteration, we then decrease the value of κ.

This process is repeated until κ becomes smaller than a fixed tolerance ε > 0.

Fast Path Planning Through Large Collections Of Safe Boxes

Abstract: We present a fast algorithm for the design of smooth paths (or trajectories) that are constrained to lie in a collection of axis-aligned boxes. We consider the case where the number of these safe boxes is large, and basic preprocessing of them (such as finding their intersections) can be done offline. At runtime we quickly generate a smooth path between given initial and terminal positions. Our algorithm designs trajectories that are guaranteed to be safe at all times, and it detects infeasibility whenever such a trajectory does not exist. Our algorithm is based on two subproblems that we can solve very efficiently: finding a shortest path in a weighted graph, and solving (multiple) convex optimal control problems. We demonstrate the proposed path planner on large-scale numerical examples, and we provide an efficient open-source software implementation, fastpathplanning.

Funnel Libraries For Real-Time Robust Feedback Motion Planning

Abstract: We consider the problem of generating motion plans for a robot that are guaranteed to succeed despite uncertainty in the environment, parametric model uncertainty, and disturbances. Furthermore, we consider scenarios where these plans must be generated in real-time, because constraints such as obstacles in the environment may not be known until they are perceived (with a noisy sensor) at runtime. Our approach is to pre-compute a library of "funnels" along different maneuvers of the system that the state is guaranteed to remain within (despite bounded disturbances) when the feedback controller corresponding to the maneuver is executed. We leverage powerful computational machinery from convex optimization (sums-of-squares programming in particular) to compute these funnels. The resulting funnel library is then used to sequentially compose motion plans at runtime while ensuring the safety of the robot. A major advantage of the work presented here is that by explicitly taking into account the effect of uncertainty, the robot can evaluate motion plans based on how vulnerable they are to disturbances. ::: We demonstrate and validate our method using extensive hardware experiments on a small fixed-wing airplane avoiding obstacles at high speed (~12 mph), along with thorough simulation experiments of ground vehicle and quadrotor models navigating through cluttered environments. To our knowledge, these demonstrations constitute one of the first examples of provably safe and robust control for robotic systems with complex nonlinear dynamics that need to plan in real-time in environments with complex geometric constraints.


We compare the results with the AAA method #REFR , which is based on randomly sampled points within the region of interest and can produce varying performances.

In this subsection, we present two numerical examples on a square region and an annulus, respectively.

To account for this variability, we conduct five tests and use the median result for comparison.

The first example is on the square region [−0.5, 0.5] × [−0.5, 0.5] with f (z) = exp(1/(5.1 2 + (10z) 2 )), which has two essential singularities at ±0.51i.

In Figure 8 , the AAA method converges rapidly and reaches the error tolerance of 10 −10 with 10 4 random sample points.

However, the method does not accurately represent the drastic local changes near the singularities with only 10 4 random sample points.

In contrast, the rational interpolation with F = D(0.510001i, 10 −6 ) ∪ D(−0.510001i, 10 −6 ) converges exponentially as n increases.

Barycentric Interpolation Based On Equilibrium Logarithmic Potential

Abstract: A novel barycentric interpolation algorithm with a specific exponential convergence rate is designed for analytic functions defined on the complex plane, with singularities located near the interpolation region, where the region is compact and can be disconnected or multiconnected. The core of the method is the efficient computation of the interpolation nodes and poles using discrete distributions that approximate the equilibrium logarithmic potential, achieved by solving a Symm's integral equation. It takes different strategies to distribute the poles for isolated singularities and branch points, respectively. In particular, if poles are not considered, it derives a polynomial interpolation with exponential convergence. Numerical experiments illustrate the superior performance of the proposed method.

The Aaa Algorithm For Rational Approximation

Abstract: We introduce a new algorithm for approximation by rational functions on a real or complex set of points, implementable in 40 lines of Matlab and requiring no user input parameters. Even on a disk or interval the algorithm may outperform existing methods, and on more complicated domains it is especially competitive. The core ideas are (1) representation of the rational approximant in barycentric form with interpolation at certain support points and (2) greedy selection of the support points to avoid exponential instabilities. The name AAA stands for"adaptive Antoulas--Anderson"in honor of the authors who introduced a scheme based on (1). We present the core algorithm with a Matlab code and nine applications and describe variants targeted at problems of different kinds. Comparisons are made with vector fitting, RKFIT, and other existing methods for rational approximation.


However, unlike our work and the work of Patro et al. #REFR , Sühr et al.

It is possible for a user with unique tastes to receive low utility recommendations and still not prefer another user's recommendation lists.

Also, our fairness formulation extends beyond the users receiving recommendations to providers of recommended items and envy-freeness provides no way to compare users who are getting different types of benefits from a system.

In addition our fairness definitions are dynamic, a case not considered by #OTHEREFR .

Like Patro et al. #OTHEREFR , the work of Sühr et al.

#OTHEREFR investigates fairness in two-sided platforms, specifically those like Uber or Lyft where income opportunities are allocated to drivers.

#OTHEREFR take proportionality as their definition of fairness, specifically proportionality with respect to time in a dynamic setting, and ensure that there is a fair distribution of income to the provider side of the platform.

#OTHEREFR investigate what they call dynamic social choice functions in settings where a fixed set of agents select a single item to share over a series of time steps.

The work focuses on overall utility to the agents instead of considering the multiple sides of the recommendation interaction.

Their problem is fundamentally a voting problem since all agents share the result, whereas we are focused on personalized recommendation.

Dynamic Fairness-Aware Recommendation Through Multi-Agent Social Choice

Abstract: Algorithmic fairness in the context of personalized recommendation presents significantly different challenges to those commonly encountered in classification tasks. Researchers studying classification have generally considered fairness to be a matter of achieving equality of outcomes between a protected and unprotected group, and built algorithmic interventions on this basis. We argue that fairness in real-world application settings in general, and especially in the context of personalized recommendation, is much more complex and multi-faceted, requiring a more general approach. We propose a model to formalize multistakeholder fairness in recommender systems as a two stage social choice problem. In particular, we express recommendation fairness as a novel combination of an allocation and an aggregation problem, which integrate both fairness concerns and personalized recommendation provisions, and derive new recommendation techniques based on this formulation. Simulations demonstrate the ability of the framework to integrate multiple fairness concerns in a dynamic way. CCS Concepts: • Information systems → Recommender systems; • Computing methodologies → Multi-agent systems; • Social and professional topics → User characteristics.

Fairrec: Two-Sided Fairness For Personalized Recommendations In Two-Sided Platforms

Abstract: We investigate the problem of fair recommendation in the context of two-sided online platforms, comprising customers on one side and producers on the other. Traditionally, recommendation services in these platforms have focused on maximizing customer satisfaction by tailoring the results according to the personalized preferences of individual customers. However, our investigation reveals that such customer-centric design may lead to unfair distribution of exposure among the producers, which may adversely impact their well-being. On the other hand, a producer-centric design might become unfair to the customers. Thus, we consider fairness issues that span both customers and producers. Our approach involves a novel mapping of the fair recommendation problem to a constrained version of the problem of fairly allocating indivisible goods. Our proposed FairRec algorithm guarantees at least Maximin Share (MMS) of exposure for most of the producers and Envy-Free up to One item (EF1) fairness for every customer. Extensive evaluations over multiple real-world datasets show the effectiveness of FairRec in ensuring two-sided fairness while incurring a marginal loss in the overall recommendation quality.


Each RN in the R in layer processes input spikes asynchronously from a corresponding pixel in the event camera following equation #REFR in real time without preprocessing.

The network is formed sequentially by an RN layer (R in ) as a feature descriptor for local temporal encoding, Convolution layers (C 1 -C 4 ), a spike conversion (SC) layer, another RN layer (R f ) for global temporal encoding and fully-connected layers (FC, F 1 -F 2 ) for classification.

Figure 1 illustrates the overall architecture of the proposed network.

The R in layer has the same size as the inputs.

Examples of the output generated by R in for DVS Lip inputs are shown in the left bottom insets of Figure 1 .

Depending on the application, different temporal resolutions can be chosen by adjusting the time constant τ of equation 2 and the frequency of R in output acquisitions.

A shorter acquisition interval creates more frequent activations for the following layers.

A shorter time constant and more frequent acquisitions produce temporally finer outputs, while a longer time constant and less frequent acquisitions produce spatially more detailed outputs owing to the slower decay.

#OTHEREFR To keep the discussions simple, here we use the same time constant and acquisition frequency for both datasets.

Retinanet: Reservoir-Enabled Time Integrated Attention Network For Event-Based Video Processing

Abstract: Event-based cameras are inspired by the sparse and asynchronous spike representation of the biological visual system. However, processing the even data requires either using expensive feature descriptors to transform spikes into frames, or using spiking neural networks that are difficult to train. In this work, we propose a neural network architecture based on simple convolution layers integrated with dynamic temporal encoding reservoirs with low hardware and training costs. The Reservoir-enabled Time Integrated Attention Network (RetinaNet) allows the network to efficiently process asynchronous temporal features, and achieves the highest accuracy of 99.2% for DVS128 Gesture reported to date, and one of the highest accuracy of 67.5% for DVS Lip dataset at a much smaller network size. By leveraging the internal dynamics of memristors, asynchronous temporal feature encoding can be implemented at very low hardware cost without preprocessing or dedicated memory and arithmetic units. The use of simple DNN blocks and backpropagation based training rules further reduces its implementation cost. Code will be publicly available.

Converting Static Image Datasets To Spiking Neuromorphic Datasets Using Saccades

Abstract: Creating datasets for Neuromorphic Vision is a challenging task. A lack of available recordings from Neuromorphic Vision sensors means that data must typically be recorded specifically for dataset creation rather than collecting and labelling existing data. The task is further complicated by a desire to simultaneously provide traditional frame-based recordings to allow for direct comparison with traditional Computer Vision algorithms. Here we propose a method for converting existing Computer Vision static image datasets into Neuromorphic Vision datasets using an actuated pan-tilt camera platform. Moving the sensor rather than the scene or image is a more biologically realistic approach to sensing and eliminates timing artifacts introduced by monitor updates when simulating motion on a computer monitor. We present conversion of two popular image datasets (MNIST and Caltech101) which have played important roles in the development of Computer Vision, and we provide performance metrics on these datasets using spike-based recognition algorithms. This work contributes datasets for future use in the field, as well as results from spike-based algorithms against which future works can compare. Furthermore, by converting datasets already popular in Computer Vision, we enable more direct comparison with frame-based approaches.


Since warmup can not be clearly distinguished from warmed up state by statistical methods #REFR , we always use the same count of warmup and measurement iterations.

In the following, we exemplarily describe two parts of the configuration: How to choose VM and execution count (part of the measurement parametrization), whether to parallelize the measurements (part of the technical measurement environment) and whether to remove outliers (part of the analysis configuration). Finally, we discuss the generalizability of our configuration. Our dataset is available.

#OTHEREFR a) Parametrization: To identify performance changes reliably, we need to choose the warmup iteration, measurement iteration, repetition and VM count.

The same product repetitions * iterations implies the same overall workload executions. For every VM, iteration measurement values are taken.

Figure 5 shows the average F 1 -score of all workload types for different iteration, repetition and VM counts using t-test like recommended by #OTHEREFR .

Since we only want 1 % false positives, we set the significance level of the t-test to 99 %.

The F 1 -score is not only influenced by the measurement configuration, but also by the statistical test.

Additionally to t-test, we used confidence interval comparison like recommended by literature #OTHEREFR , #OTHEREFR , and Mann-Whitney-test like recommended by literature #OTHEREFR , #OTHEREFR .

Automated Identification Of Performance Changes At Code Level

Abstract: To develop software with optimal performance, even small performance changes need to be identified. Identifying performance changes is challenging since the performance of software is influenced by non-deterministic factors. Therefore, not every performance change is measurable with reasonable effort. In this work, we discuss which performance changes are measurable at code level with reasonable measurement effort and how to identify them. We present (1) an analysis of the boundaries of measuring performance changes, (2) an approach for determining a configuration for reproducible performance change identification, and (3) an evaluation comparing of how well our approach is able to identify performance changes in the application server Jetty compared with the usage of Jetty's own performance regression benchmarks. Thereby, we find (1) that small performance differences are only measurable by fine-grained measurement workloads, (2) that performance changes caused by the change of one operation can be identified using a unit-test-sized workload definition and a suitable configuration, and (3) that using our approach identifies small performance regressions more efficiently than using Jetty's performance regression benchmarks.

Virtual Machine Warmup Blows Hot And Cold

Abstract: Virtual Machines (VMs) with Just-In-Time (JIT) compilers are traditionally thought to execute programs in two phases: the initial warmup phase determines which parts of a program would most benefit from dynamic compilation, before JIT compiling those parts into machine code; subsequently the program is said to be at a steady state of peak performance. Measurement methodologies almost always discard data collected during the warmup phase such that reported measurements focus entirely on peak performance. We introduce a fully automated statistical approach, based on changepoint analysis, which allows us to determine if a program has reached a steady state and, if so, whether that represents peak performance or not. Using this, we show that even when run in the most controlled of circumstances, small, deterministic, widely studied microbenchmarks often fail to reach a steady state of peak performance on a variety of common VMs. Repeating our experiment on 3 different machines, we found that at most 43.5% of pairs consistently reach a steady state of peak performance.


Therefore, taking inspiration from sketch/handwriting generation literature #REFR , we define our sketch-coordinate in terms of offset values to make it scale-agnostic.

Absolute coordinate based sketch-vector representation is scale dependent as the user can draw (see Fig. 2 ) the same concept in varying scales.

In particular, instead of three elements with absolute coordinate (x i , y i ) and stroke token (b i ), now we represent every point as a five-element vector

(∆x i , ∆y i , p 1 i , p 2 i , p 3 i ), where ∆x i = (x i+1 − x i ) and ∆y i = (y i+1 − y i ).

represents three pen-state situations: pen touching the paper, pen being lifted and end of drawing.

Convolutional Encoder Instead of any complicated backbone architectures #OTHEREFR , we use a straight-forward VGG-16 as the backbone convolutional encoder, which takes a photo (image) P ∈ R H×W ×3 as input and outputs multiscale convolutional feature maps as F ∈

Sketch2Saliency: Learning To Detect Salient Objects From Human Drawings

Abstract: Human sketch has already proved its worth in various visual understanding tasks (e.g., retrieval, segmentation, image-captioning, etc). In this paper, we reveal a new trait of sketches-that they are also salient. This is intuitive as sketching is a natural attentive process at its core. More specifically, we aim to study how sketches can be used as a weak label to detect salient objects present in an image. To this end, we propose a novel method that emphasises on how "salient object" could be explained by handdrawn sketches. To accomplish this, we introduce a phototo-sketch generation model that aims to generate sequential sketch coordinates corresponding to a given visual photo through a 2D attention mechanism. Attention maps accumulated across the time steps give rise to salient regions in the process. Extensive quantitative and qualitative experiments prove our hypothesis and delineate how our sketchbased saliency detection model gives a competitive performance compared to the state-of-the-art.

A Neural Representation Of Sketch Drawings

Abstract: We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format.


Spherical Multi-Resolution Hash Encoding

This representation can be interpreted as an omnidirectional extension of a Normalized Device Coordinates (NDC) used for novel view synthesis in forward facing scenes in #REFR .

We convert the input location represented in Cartesian coordinates (x, y, z) ∈ R 3 to spherical coordinates (φ, θ, 1 1+r ) ∈ [0, 1] 3 as follows:

This enables at the same time an efficient feature allocation strategy related to the pixel's measurement accuracy and a representation that is not limited by the frustum boundary.

Feature Allocation An increasing distance to the camera center leads to a larger measurement uncertainty as disparity is inversely-proportional to its depth #OTHEREFR .

To allocate more capacity in areas of high certainty, it is therefore efficient to allocate the features evenly in inverse depth space.

While the NDC parameterization contracts the scene along inverse depth, the region is strictly bounded by the model view frustum, which discards the training view information outside of the frustum and also causes artifacts around the frustum boundary as shown in Fig. 4 .

In contrast, our spherical representation can make full use of the training view information without being limited to the field of view.

Unbounded Scenes Our representation can naturally capture unbounded scenes.

While other parameterizations such as the recently proposed space contraction from #OTHEREFR can also represent 360 • unbounded scenes, it explicitly separates foreground and background.

This is effective only when the camera trajectory is object-centric and inward-looking, and the region of interest needs to be known a priori in world coordinates.

Newton: Neural View-Centric Mapping For On-The-Fly Large-Scale Slam

Abstract: Neural field-based 3D representations have recently been adopted in many areas including SLAM systems. Current neural SLAM or online mapping systems lead to impressive results in the presence of simple captures, but they rely on a world-centric map representation as only a single neural field model is used. To define such a world-centric representation, accurate and static prior information about the scene, such as its boundaries and initial camera poses, are required. However, in real-time and on-the-fly scene capture applications, this prior knowledge cannot be assumed as fixed or static, since it dynamically changes and it is subject to significant updates based on run-time observations. Particularly in the context of large-scale mapping, significant camera pose drift is inevitable, necessitating the correction via loop closure. To overcome this limitation, we propose NEWTON, a view-centric mapping method that dynamically constructs neural fields based on run-time observation. In contrast to prior works, our method enables camera pose updates using loop closures and scene boundary updates by representing the scene with multiple neural fields, where each is defined in a local coordinate system of a selected keyframe. The experimental results demonstrate the superior performance of our method over existing world-centric neural field-based SLAM systems, in particular for large-scale scenes subject to camera pose updates.

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.

forward facing scenes
novel view synthesis


We experiment with FOVs of 180°, 120°, 60°, and uniformly random in the #REFR 120] • interval.

The mean relative error and RMSE are 4.25 % and 173.0 on the entire test set.

We observe that the network struggles with larger temperature variations across the image.

However, temperature is accurately predicted despite color changes in the input.

Planar illuminance Here, we experiment with three types of inputs: a calibrated HDR image ("HDR"), a linear LDR image (reexposed HDR clipped to the [0, 1] interval) with ("LDR+scale") and without ("LDR") knowledge of the exposure.

In addition, we also evaluate the impact of the FOV of the input.

The image is stored in an equirectangular representation for 180°, and perspective projection for the other, lower FOVs.

Tab. 2 shows the results of these series of experiments.

We report the RMSE and R 2 for each combination of input type and FOV.

First, observe that the experiment with a FOV of 180°with the HDR image (top-left in tab. 2) amounts to learning the illuminance integration (eq. (1)).

Unsurprisingly, narrowing the FOV results in decreased performance, due to the hidden lights beyond the FOV which may directly affect the planar illuminance.

Beyond The Pixel: A Photometrically Calibrated Hdr Dataset For Luminance And Color Temperature Prediction

Abstract: Light plays an important role in human well-being. However, most computer vision tasks treat pixels without considering their relationship to physical luminance. To address this shortcoming, we present the first large-scale photometrically calibrated dataset of high dynamic range 360°panoramas. Our key contribution is the calibration of an existing, uncalibrated HDR Dataset. We do so by accurately capturing RAW bracketed exposures simultaneously with a professional photometric measurement device (chroma meter) for multiple scenes across a variety of lighting conditions. Using the resulting measurements, we establish the calibration coefficients to be applied to the HDR images. The resulting dataset is a rich representation of indoor scenes which displays a wide range of illuminance and color temperature, and varied types of light sources. We exploit the dataset to introduce three novel tasks: where per-pixel luminance, per-pixel temperature and planar illuminance can be predicted from a single input image. Finally, we also capture another smaller calibrated dataset with a commercial 360°camera, to experiment on generalization across cameras. We are optimistic that the release of our datasets and associated code will spark interest in physically accurate light estimation within the community.

U-Net: Convolutional Networks For Biomedical Image Segmentation

Abstract: There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .


Outside-Knowledge Visual Question Answering (OK-VQA) #REFR : This dataset consists of triplets, including an image, a question about the image, and an answer to the mentioned question.

Answering most of the questions in this dataset needs a piece of information that is not provided in the image.

Therefore, accessing an external source of information is required for this task.

A retrieval dataset based on a Wikipedia dump 7 with 11 million passages was later constructed by Qu et al.

#OTHEREFR , which we use to train and evaluate our retrievers.

This dataset contains 9009 questions for training, 2523 questions for validation, and 2523 for testing #OTHEREFR .

A Symmetric Dual Encoding Dense Retrieval Framework For Knowledge-Intensive Visual Question Answering

Abstract: Knowledge-Intensive Visual Question Answering (KI-VQA) refers to answering a question about an image whose answer does not lie in the image. This paper presents a new pipeline for KI-VQA tasks, consisting of a retriever and a reader. First, we introduce DEDR, a symmetric dual encoding dense retrieval framework in which documents and queries are encoded into a shared embedding space using uni-modal (textual) and multi-modal encoders. We introduce an iterative knowledge distillation approach that bridges the gap between the representation spaces in these two encoders. Extensive evaluation on two well-established KI-VQA datasets, i.e., OK-VQA and FVQA, suggests that DEDR outperforms state-of-the-art baselines by 11.6% and 30.9% on OK-VQA and FVQA, respectively. Utilizing the passages retrieved by DEDR, we further introduce MM-FiD, an encoder-decoder multi-modal fusion-in-decoder model, for generating a textual answer for KI-VQA tasks. MM-FiD encodes the question, the image, and each retrieved passage separately and uses all passages jointly in its decoder. Compared to competitive baselines in the literature, this approach leads to 5.5% and 8.5% improvements in terms of question answering accuracy on OK-VQA and FVQA, respectively.

Ok-Vqa: A Visual Question Answering Benchmark Requiring External Knowledge

Abstract: Visual Question Answering (VQA) in its ideal form lets us study reasoning in the joint space of vision and language and serves as a proxy for the AI task of scene understanding. However, most VQA benchmarks to date are focused on questions such as simple counting, visual attributes, and object detection that do not require reasoning or knowledge beyond what is in the image. In this paper, we address the task of knowledge-based visual question answering and provide a benchmark, called OK-VQA, where the image content is not sufficient to answer the questions, encouraging methods that rely on external knowledge resources. Our new dataset includes more than 14,000 questions that require external knowledge to answer. We show that the performance of the state-of-the-art VQA models degrades drastically in this new setting. Our analysis shows that our knowledge-based VQA task is diverse, difficult, and large compared to previous knowledge-based VQA datasets. We hope that this dataset enables researchers to open up new avenues for research in this domain.


CenterTrack #REFR learns to generate a center heatmap for object detection and an offset vector that represents the displacement from one frame to the next using the two consecutive frames and the center heatmap of the previous frame.

TubeTK proposes the concept of bounding-tubes to represent the spatiotemporal location of objects detected in a video.

A bounding-tube is defined as a series of three bounding boxes of the same object from different frames.

Tracks are broken down into a combination of tubes where the bounding box at each frame is defined as the middle bounding box of a tube in order to link tubes together.

The three bounding boxes forming a tube do not have to be from consecutive frames, which allows the interpolation of the object location within a tube. Tubes are linked using IoU as a distance metric.

Chained-Tracker takes two adjacent frames as input and generates detection pairs of each object in both frames and then link each consecutive pair using IoU as a distance metric and then the Hungarian algorithm #OTHEREFR for matching.

It then regresses the bounding boxes from the center points of the objects.

Data association is done using greedy matching between the position of the objects in the previous frame and the position of the predicted offset.

Tracktor #OTHEREFR exploits the bounding box regression module of Faster R-CNN to perform tracking directly from the information provided by the detector.

Moreover, the model can be extended with a motion model and a reidentification algorithm to achieve better results.

FairMOT #OTHEREFR aims to reconcile the inherent bias in favor of the detection task when training a MOT model.

Toptrack: Tracking Objects By Their Top

Abstract: In recent years, the joint detection-and-tracking paradigm has been a very popular way of tackling the multi-object tracking (MOT) task. Many of the methods following this paradigm use the object center keypoint for detection. However, we argue that the center point is not optimal since it is often not visible in crowded scenarios, which results in many missed detections when the objects are partially occluded. We propose TopTrack, a joint detection-and-tracking method that uses the top of the object as a keypoint for detection instead of the center because it is more often visible. Furthermore, Top-Track processes consecutive frames in separate streams in order to facilitate training. We performed experiments to show that using the object top as a keypoint for detection can reduce the amount of missed detections, which in turn leads to more complete trajectories and less lost trajectories. TopTrack manages to achieve competitive results with other state-of-the-art trackers on two MOT benchmarks.

Tracking Objects As Points

Abstract: Tracking has traditionally been the art of following interest points through space and time. This changed with the rise of powerful deep networks. Nowadays, tracking is dominated by pipelines that perform object detection followed by temporal association, also known as tracking-by-detection. In this paper, we present a simultaneous detection and tracking algorithm that is simpler, faster, and more accurate than the state of the art. Our tracker, CenterTrack, applies a detection model to a pair of images and detections from the prior frame. Given this minimal input, CenterTrack localizes objects and predicts their associations with the previous frame. That's it. CenterTrack is simple, online (no peeking into the future), and real-time. It achieves 67.3% MOTA on the MOT17 challenge at 22 FPS and 89.4% MOTA on the KITTI tracking benchmark at 15 FPS, setting a new state of the art on both datasets. CenterTrack is easily extended to monocular 3D tracking by regressing additional 3D attributes. Using monocular video input, it achieves 28.3% AMOTA@0.2 on the newly released nuScenes 3D tracking benchmark, substantially outperforming the monocular baseline on this benchmark while running at 28 FPS.


The implication of using quantised or low-precision variables is that it not only reduces memory footprint but can also reduce the compute cost of matrix multiplications #REFR at the expense of loss in precision.

Thus, for a neural network with |W| parameters, at any point during the training there are an additional ≈ 3 ⋅ |W| variables stored in memory.

Further, the intermediate activations at layer l, h l are also stored in memory to efficiently perform backpropagation.

Note that all the scalar entries of W, g t , m t , v t , h l ∈ R.

On most computers, these real numbers are discretised into floating point-32 (FP32) format; wherein each variable requires 32 bits.

Resource efficiency in this work is primarily addressed by reducing the precision of these variables by quantisation #OTHEREFR .

Note, however, that the overhead of performing quantisation in some cases might outweigh the gains in computation.

In this work, we investigate a combination of the following three quantisation strategies:

1. Gradients and intermediate activations:

have been studied extensively in literature #OTHEREFR .

Operating Critical Machine Learning Models In Resource Constrained Regimes

Abstract: The accelerated development of machine learning methods, primarily deep learning, are causal to the recent breakthroughs in medical image analysis and computer aided intervention. The resource consumption of deep learning models in terms of amount of training data, compute and energy costs are known to be massive. These large resource costs can be barriers in deploying these models in clinics, globally. To address this, there are cogent efforts within the machine learning community to introduce notions of resource efficiency. For instance, using quantisation to alleviate memory consumption. While most of these methods are shown to reduce the resource utilisation, they could come at a cost in performance. In this work, we probe into the trade-off between resource consumption and performance, specifically, when dealing with models that are used in critical settings such as in clinics.

Quantized Neural Networks: Training Neural Networks With Low Precision Weights And Activations

Abstract: We introduce a method to train Quantized Neural Networks (QNNs) -- neural networks with extremely low precision (e.g., 1-bit) weights and activations, at run-time. At traintime the quantized weights and activations are used for computing the parameter gradients. During the forward pass, QNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations. As a result, power consumption is expected to be drastically reduced. We trained QNNs over the MNIST, CIFAR-10, SVHN and ImageNet datasets. The resulting QNNs achieve prediction accuracy comparable to their 32-bit counterparts. For example, our quantized version of AlexNet with 1-bit weights and 2-bit activations achieves 51% top-1 accuracy. Moreover, we quantize the parameter gradients to 6-bits as well which enables gradients computation using only bit-wise operation. Quantized recurrent neural networks were tested over the Penn Treebank dataset, and achieved comparable accuracy as their 32-bit counterparts using only 4-bits. Last but not least, we programmed a binary matrix multiplication GPU kernel with which it is possible to run our MNIST QNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The QNN code is available online.


For instance, an e-commerce item carries its category as well as many other attributes, and the user might have a preference for a certain category #REFR .

The Borda winner is intuitively appealing and always well-defined for any set of preferential probabilities.

The Borda score also does not require the problem instance to obey any consistency or transitivity, and it is considered one of the most general criteria.

To identify the Borda winner, estimations of the Borda scores are needed.

Since estimating the Borda score for one item requires comparing it with every other items, the sample complexity is prohibitively high when there are numerous items.

On the other hand, in many real-world applications, the agent has access to side information that can assist the evaluation of p i,j .

For a movie, the genre and the plot as well as the directors and actors can also be taken into consideration when making choices #OTHEREFR .

Based on the above motivation, we consider Generalized Linear Dueling Bandits.

At each round, the agent selects two items from a finite set of items and receives a comparison result of the preferred item.

The comparisons depend on known intrinsic contexts/features associated with each pair of items.

The contexts can be obtained from upstream tasks, such as topic modeling #OTHEREFR or embedding #OTHEREFR .

Borda Regret Minimization For Generalized Linear Dueling Bandits

Abstract: Dueling bandits are widely used to model preferential feedback that is prevalent in machine learning applications such as recommendation systems and ranking. In this paper, we study the Borda regret minimization problem for dueling bandits, which aims to identify the item with the highest Borda score while minimizing the cumulative regret. We propose a new and highly expressive generalized linear dueling bandits model, which covers many existing models. Surprisingly, the Borda regret minimization problem turns out to be difficult, as we prove a regret lower bound of order Ω(d 2/3 T 2/3), where d is the dimension of contextual vectors and T is the time horizon. To attain the lower bound, we propose an explore-then-commit type algorithm, which has a nearly matching regret upper bound O(d 2/3 T 2/3). When the number of items/arms K is small, our algorithm can achieve a smaller regret O((d log K) 1/3 T 2/3) with proper choices of hyperparameters. We also conduct empirical experiments on both synthetic data and a simulated real-world environment, which corroborate our theoretical analysis.

Billion-Scale Commodity Embedding For E-Commerce Recommendation In Alibaba

Abstract: Recommender systems (RSs) have been the most important technology for increasing the business in Taobao, the largest online consumer-to-consumer (C2C) platform in China. There are three major challenges facing RS in Taobao: scalability, sparsity and cold start. In this paper, we present our technical solutions to address these three challenges. The methods are based on a well-known graph embedding framework. We first construct an item graph from users' behavior history, and learn the embeddings of all items in the graph. The item embeddings are employed to compute pairwise similarities between all items, which are then used in the recommendation process. To alleviate the sparsity and cold start problems, side information is incorporated into the graph embedding framework. We propose two aggregation methods to integrate the embeddings of items and the corresponding side information. Experimental results from offline experiments show that methods incorporating side information are superior to those that do not. Further, we describe the platform upon which the embedding methods are deployed and the workflow to process the billion-scale data in Taobao. Using A/B test, we show that the online Click-Through-Rates (CTRs) are improved comparing to the previous collaborative filtering based methods widely used in Taobao, further demonstrating the effectiveness and feasibility of our proposed methods in Taobao's live production environment.


Remark 3 (Escape And Converge). When The Noise Ξ

Similar results have been obtained in #REFR when assuming the deep learning algorithms are driven by an Ornstein-Uhlenbeck process.

2 , as the scale of noise ξ ULN k (θ) is large.

Reciprocally, we follow #OTHEREFR and suggest that when the SGD with unbiased random label noises converges, the algorithm would converge to a point

θ * with small 1 N N i=1 ∇ θ f (x i , θ * ) 2

Considering ησ 2 /B as the coefficient balancing the implicit regularizer and vanilla SGD, one can regularize/penalize the SGD learning procedure with the fixed η and B more fiercely using a larger σ 2 .

More specifically, we could expect to obtain a regularized solution with #OTHEREFR 2 or higher inference stability of neural networks, as regularization effects become stronger when σ 2 increases.

Stochastic Gradient Descent With Random Label Noises: Doubly Stochastic Models And Inference Stabilizer

Abstract: Random label noises (or observational noises) widely exist in practical machine learning settings. While previous studies primarily focus on the affects of label noises to the performance of learning, our work intends to investigate the implicit regularization effects of the label noises, under mini-batch sampling settings of stochastic gradient descent (SGD), with assumptions that label noises are unbiased. Specifically, we analyze the learning dynamics of SGD over the quadratic loss with unbiased label noises, where we model the dynamics of SGD as a stochastic differentiable equation (SDE) with two diffusion terms (namely a Doubly Stochastic Model). While the first diffusion term is caused by mini-batch sampling over the (label-noiseless) loss gradients as many other works on SGD [1, 2], our model investigates the second noise term of SGD dynamics, which is caused by mini-batch sampling over the label noises, as an implicit regularizer. Our theoretical analysis finds such implicit regularizer would favor some convergence points that could stabilize model outputs against perturbation of parameters (namely inference stability). Though similar phenomenon have been investigated in [3], our work

Implicit Regularization For Deep Neural Networks Driven By An Ornstein-Uhlenbeck Like Process

Abstract: We consider deep networks, trained via stochastic gradient descent to minimize L2 loss, with the training labels perturbed by independent noise at each iteration. We characterize the behavior of the training dynamics near any parameter vector that achieves zero training error, in terms of an implicit regularization term corresponding to the sum over the data points, of the squared L2 norm of the gradient of the model with respect to the parameter vector, evaluated at each data point. We then leverage this general characterization, which holds for networks of any connectivity, width, depth, and choice of activation function, to show that for 2-layer ReLU networks of arbitrary width and L2 loss, when trained on one-dimensional labeled data $(x_1,y_1),\ldots,(x_n,y_n),$ the only stable solutions with zero training error correspond to functions that: 1) are linear over any set of three or more co-linear training points (i.e. the function has no extra "kinks"); and 2) change convexity the minimum number of times that is necessary to fit the training data. Additionally, for 2-layer networks of arbitrary width, with tanh or logistic activations, we show that when trained on a single $d$-dimensional point $(x,y)$ the only stable solutions correspond to networks where the activations of all hidden units at the datapoint, and all weights from the hidden units to the output, take at most two distinct values, or are zero. In this sense, we show that when trained on "simple" data, models corresponding to stable parameters are also "simple"; in short, despite fitting in an over-parameterized regime where the vast majority of expressible functions are complicated and badly behaved, stable parameters reached by training with noise express nearly the "simplest possible" hypothesis consistent with the data. These results shed light on the mystery of why deep networks generalize so well in practice.