Upper Bound On The Competitive Ratio Of Move To Front

This significantly improves the result of Kamali and López-Ortiz #REFR , who showed that Move To Front has an asymptotic competitive ratio of 6µ + 7, i.e., for any item list R, they showed cost(MF, R) ≤ (6µ + 7) • OPT(R) + 3(µ + 1).

In this section we prove the first main result of our paper.

For d = 1, our result implies that Move To Front has a competitive ratio of at most 2µ + 2.

Our result also nearly settles their conjecture of the CR being 2µ + 1.

Their analysis decomposes the active span into segments of length (µ + 1) and compares the cost of OPT with the cost of Move To Front in each such interval. It turns out that this decomposition is sub-optimal.

Instead, we directly use the nature of the Move To Front algorithm and develop a novel decomposition of the usage periods of each bin B into intervals based on whether or not in the interval B is the most recently used bin. We now prove Theorem 2.

Suppose Move To Front uses m bins B 1 , B 2 , . . . , B m on an input sequence R.

As mentioned earlier, we can assume that ∪ r∈R I(r) = [0, span(R)) and that the usage period of each bin is We say a bin is a leader at time t if it is in the front of the list L at time t.

Dynamic Vector Bin Packing For Online Resource Allocation In The Cloud

Abstract: Several cloud-based applications, such as cloud gaming, rent servers to execute jobs which arrive in an online fashion. Each job has a resource demand, such as GPU requirement, and must be dispatched to a cloud server which has enough resources to execute the job, which departs after its completion. Under the "pay-as-you-go" billing model, the server rental cost is proportional to the total time that servers are actively running jobs. The problem of efficiently allocating a sequence of online jobs to servers without exceeding the resource capacity of any server while minimizing total server usage time can be modelled as a variant of the dynamic bin packing problem (DBP), called MinUsageTime DBP [21]. In this work, we initiate the study of the problem with multi-dimensional resource demands (e.g. CPU/GPU usage, memory requirement, bandwidth usage, etc.), called MinUsageTime Dynamic Vector Bin Packing (DVBP). We study the competitive ratio (CR) of Any Fit packing algorithms for this problem. We show almost-tight bounds on the CR of three specific Any Fit packing algorithms, namely First Fit, Next Fit, and Move To Front. We prove that the CR of Move To Front is at most (2µ + 1)d + 1, where µ is the ratio of the max/min item durations. For d = 1, this implies a significant improvement over the previously known upper bound of 6µ + 7 [18]. We then prove the CR of First Fit and Next Fit are bounded by (µ + 2)d + 1 and 2µd + 1, respectively. Next, we prove a lower bound of (µ + 1)d on the CR of any Any Fit packing algorithm, an improved lower bound of 2µd for Next Fit, and a lower bound of 2µ for Move To Front in the 1-D case. All our bounds improve or match the best-known bounds for the 1-D case. Finally, we experimentally study the average-case performance of these algorithms on randomly generated synthetic data, and observe that Move To Front outperforms other Any Fit packing algorithms.

Efficient Online Strategies For Renting Servers In The Cloud

Abstract: When scheduling jobs for systems in the cloud, we often deal with jobs that arrive and depart in an online manner. Each job should be assigned to a server upon arrival. Jobs are annotated with sizes which define the amount of resources that they need. Servers have uniform capacity and, at all times, the total size of jobs assigned to a server should not exceed this capacity. This setting is closely related to the classic bin packing problem. The difference is that, in bin packing, the objective is to minimize the total number of used servers. In the cloud systems, however, the cost for each server is proportional to the length of the time interval it is rented for, and the goal is to minimize the cost involved in renting all used servers. Recently, certain bin packing strategies were considered for renting servers in the cloud [Li et al. SPAA’14]. There, it is proved that all Any-Fit strategies have a competitive ratio of at least μ, where μ is the max/min interval length ratio of jobs. It is also proved that First Fit has a competitive ratio of 2μ + 13, while Best Fit is not competitive at all. We observe that the lower bound of μ extends to all online algorithms. We also prove that, surprisingly, Next Fit algorithm has a competitive ratio of at most 2 μ + 1. We also show that a variant of Next Fit achieves a competitive ratio of K × max {1,μ/(K − 1)} + 1, where K is a parameter of the algorithm. In particular, if the value of μ is known, the algorithm has a competitive ratio of μ + 2; this improves upon the existing upper bound of μ + 8. Finally, we introduce a simple algorithm called Move To Front (Mtf) which has a competitive ratio of at most 6μ + 8. We experimentally study the average-case performance of different algorithms and observe that the typical behaviour of Mtf is better than other algorithms.


To alleviate the limitation, inspired by #REFR , we apply the geometry transformation to extract aligned local visual features.

To improve the hand pose estimation, we propose to first predict sparse 3D joint locations j h from the image and then use inverse kinematics to derive pose transformations θ h from the predicted joints.

In this way, we are able to obtain kinematic features e h and e o for the hand and the object respectively.

The visual feature encodes the visual appearance for the point x to provide more shape details.

Prior works #OTHEREFR use the same global visual feature for all the points, e.g., averaging the feature map of a SDF feature encoder on the spatial dimension.

Such global visual features suffers from imprecise geometry alignment between a point and its visual appearance.

Moreover, to address hard cases with occlusions and motion blur in a single image I t , we propose to enhance the local visual feature with its temporal contexts from videos using a spatio-temporal transformer.

We denote the local visual feature of a point as e v .

Finally, we concatenate the kinematic feature and local visual feature to predict the signed distance for x:

where f h and f o are the hand SDF decoder and the object SDF decoder respectively.

Gsdf: Geometry-Driven Signed Distance Functions For 3D Hand-Object Reconstruction

Abstract: Signed distance functions (SDFs) is an attractive framework that has recently shown promising results for 3D shape reconstruction from images. SDFs seamlessly generalize to different shape resolutions and topologies but lack explicit modelling of the underlying 3D geometry. In this work, we exploit the hand structure and use it as guidance for SDF-based shape reconstruction. In particular, we address reconstruction of hands and manipulated objects from monocular RGB images. To this end, we estimate poses of hands and objects and use them to guide 3D reconstruction. More specifically, we predict kinematic chains of pose transformations and align SDFs with highly-articulated hand poses. We improve the visual features of 3D points with geometry alignment and further leverage temporal information to enhance the robustness to occlusion and motion blurs. We conduct extensive experiments on the challenging ObMan and DexYCB benchmarks and demonstrate significant improvements of the proposed method over the state of the art.

Pifu: Pixel-Aligned Implicit Function For High-Resolution Clothed Human Digitization

Abstract: We introduce Pixel-aligned Implicit Function (PIFu), a highly effective implicit representation that locally aligns pixels of 2D images with the global context of their corresponding 3D object. Using PIFu, we propose an end-to-end deep learning method for digitizing highly detailed clothed humans that can infer both 3D surface and texture from a single image, and optionally, multiple input images. Highly intricate shapes, such as hairstyles, clothing, as well as their variations and deformations can be digitized in a unified way. Compared to existing representations used for 3D deep learning, PIFu can produce high-resolution surfaces including largely unseen regions such as the back of a person. In particular, it is memory efficient unlike the voxel representation, can handle arbitrary topology, and the resulting surface is spatially aligned with the input image. Furthermore, while previous techniques are designed to process either a single image or multiple views, PIFu extends naturally to arbitrary number of views. We demonstrate high-resolution and robust reconstructions on real world images from the DeepFashion dataset, which contains a variety of challenging clothing types. Our method achieves state-of-the-art performance on a public benchmark and outperforms the prior work for clothed human digitization from a single image.


Consequently, z is assumed to be independent of c and follow a tractable probability distribution, where the uniqueness of the mapping between λ and [c, z] #REFR is constructed.

Similar to DeepONets #OTHEREFR , the separate representation of the solutions is employed by

where c i represents the expansion coefficient, and φ i (x) is the basis function defined on the spatial domain.

The INN is designed to learn the bijective mapping between the parametric input λ and the unique pairs [c, z] of expansion coefficients and random latent variables.

The random latent variable z is utilized to capture the inherent information loss in the output layer, which is not contained in c.

In this work, the standard Gaussian distribution for z is adopted.

Furthermore, the INN architecture consists of alternating layers and affine coupling layers.

The partitioned input of the coupling layer remains unchanged for one subset, while the other subset undergoes a coupling operation.

Then, the two parts of output are exchanged in alternating layers, which ensure that every dimension of the input is updated #OTHEREFR .

Then, a fully-connected neural network named NB-Net is employed to obtain the spatial basis functions, and the outputs of the NB-Net can be orthogonalized.

Efficient Bayesian Inference Using Physics-Informed Invertible Neural Networks For Inverse Problems

Abstract: In the paper, we propose a novel approach for solving Bayesian inverse problems with physics-informed invertible neural networks (PI-INN). The architecture of PI-INN consists of two sub-networks: an invertible neural network (INN) and a neural basis network (NB-Net). The invertible map between the parametric input and the INN output with the aid of NB-Net is constructed to provide a tractable estimation of the posterior distribution, which enables efficient sampling and accurate density evaluation. Furthermore, the loss function of PI-INN includes two components: a residual-based physics-informed loss term and a new independence loss term. The presented independence loss term can Gaussianize the random latent variables and ensure statistical independence between two parts of INN output by effectively utilizing the estimated density function. Several numerical experiments are presented to demonstrate the efficiency and accuracy of the proposed PI-INN, including inverse kinematics, inverse problems of the 1-d and 2-d diffusion equations, and seismic traveltime tomography.

Analyzing Inverse Problems With Invertible Neural Networks

Abstract: In many tasks, in particular in natural science, the goal is to determine hidden system parameters from a set of measurements. Often, the forward process from parameter- to measurement-space is a well-defined function, whereas the inverse problem is ambiguous: one measurement may map to multiple different sets of parameters. In this setting, the posterior parameter distribution, conditioned on an input measurement, has to be determined. We argue that a particular class of neural networks is well suited for this task -- so-called Invertible Neural Networks (INNs). Although INNs are not new, they have, so far, received little attention in literature. While classical neural networks attempt to solve the ambiguous inverse problem directly, INNs are able to learn it jointly with the well-defined forward process, using additional latent output variables to capture the information otherwise lost. Given a specific measurement and sampled latent variables, the inverse pass of the INN provides a full distribution over parameter space. We verify experimentally, on artificial data and real-world problems from astrophysics and medicine, that INNs are a powerful analysis tool to find multi-modalities in parameter space, to uncover parameter correlations, and to identify unrecoverable parameters.

mapping
tractable probability distribution


The setup of #REFR is not directly usable for experiments with the RS@k loss, as large batch sizes are crucial to estimate recall@k accurately. Furthermore, their setup does not allow mixup.

The methods in the literature use different embedding sizes, d, therefore, the models for the RS@k loss are trained with two embedding sizes of d = 128 and d = 512 for image retrieval benchmarks #OTHEREFR - #OTHEREFR , and d = 2048 for ROxford/RParis #OTHEREFR , to allow a fair comparison.

In the standard split, the image retrieval benchmarks #OTHEREFR - #OTHEREFR do not contain an explicit validation set; as a result, image retrieval methods often tune the hyper-parameters on the test set, leading to the issue of training with test set feedback.

This issue has been studied in #OTHEREFR , which proposes to train different methods with identical hyper-parameters.

Therefore, instead of following #OTHEREFR , the issue is eliminated by using a part of the training set for validation as described above.

Abstract: I hereby declare I have written this doctoral dissertation independently and quoted all the sources of information used in accordance with methodological instructions on ethical principles for writing an academic dissertation. Moreover, I state that this dissertation has neither been submitted nor accepted for any other degree. The results presented in this dissertation were were achieved during my own research in cooperation with my dissertation supervisor Jiri Matas published in [1]-[8]. During the Ph.D., I also collaborated with several researchers on multiple projects, with R. Manmatha, Srikar

A Metric Learning Reality Check

Abstract: Deep metric learning papers from the past four years have consistently claimed great advances in accuracy, often more than doubling the performance of decade-old methods. In this paper, we take a closer look at the field to see if this is actually true. We find flaws in the experimental setup of these papers, and propose a new way to evaluate metric learning algorithms. Finally, we present experimental results that show that the improvements over time have been marginal at best.


We use the Sunspot time series 5 data and Abalone #REFR datasets for regression problems.

The Abalone dataset provides the ring age for Abalone based on eight features that represent physical properties such as length, width, and weight and associated target feature, i.e., the ring age.

Determining the age of Abalone is difficult as requires cutting the shell and counting the number of rings using a microscope.

However, other physical measurements can be used to predict the age and a model can be developed to use the physical features to determine the ring age.

Sunspots are regions of reduced surface temperature in the Sun's photosphere caused by concentrations of magnetic field flux, and appear as spots darker than the surrounding areas.

The Sunspot cycles are about every eleven years and over the solar cycle, the number of occurrence of Sunspot change more rapidly.

Bayesian Neural Networks Via Mcmc: A Python-Based Tutorial

Abstract: Bayesian inference provides a methodology for parameter estimation and uncertainty quantification in machine learning and deep learning methods. Variational inference and Markov Chain Monte-Carlo (MCMC) sampling techniques are used to implement Bayesian inference. In the past three decades, MCMC methods have faced a number of challenges in being adapted to larger models (such as in deep learning) and big data problems. Advanced proposals that incorporate gradients, such as a Langevin proposal distribution, provide a means to address some of the limitations of MCMC sampling for Bayesian neural networks. Furthermore, MCMC methods have typically been constrained to use by statisticians and are still not prominent among deep learning researchers. We present a tutorial for MCMC methods that covers simple Bayesian linear and logistic models, and Bayesian neural networks. The aim of this tutorial is to bridge the gap between theory and implementation via coding, given a general sparsity of libraries and tutorials to this end. This tutorial provides code in Python with data and instructions that enable their use and extension. We provide results for some benchmark problems showing the strengths and weaknesses of implementing the respective Bayesian models via MCMC. We highlight the challenges in sampling multi-modal posterior distributions in particular for the case of Bayesian neural networks, and the need for further improvement of convergence diagnosis.

Adam: A Method For Stochastic Optimization

Abstract: We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.


Studies on continual reinforcement learning seek to determine how learning one or more tasks can accelerate learning of other tasks, as well as avoid ruinous forgetting #REFR .

In policy transfer, behaviors are transferred to a new agent (student) by a pre-trained agent (teacher).

The student is trained to minimize the KullbackLeibler divergence to the teacher using policy distillation, for example [36] .

Other methods, re-use policies from source tasks directly to create a student policy #OTHEREFR .

On the other hand, a pre-learned state representation used in feature transfer encourages the agent to explore when it is presented with tasks #OTHEREFR .

In exploration transfer, in a task-agnostic and environment-centric scenario, the agent is first attempted by defining an intrinsic reward to visit the states as more as possible especially interesting states in multiple environments instead of learning the main task, and then this prior knowledge is transferred as a bias to the main environment in which the main task is to be trained #OTHEREFR .

Meta reinforcement learning and curriculum learning, on the other hand, focus on exploiting underlying similarities between tasks to speed up the learning process of new tasks #OTHEREFR .

Many studies have been done under the Lifelong RL framework, where the tasks change following a specific distribution #OTHEREFR .

In #OTHEREFR , by spending a lot of computational costs, knowledge transfer is done through transferring initializers.

Most are only compatible with ( , )-PAC algorithms #OTHEREFR In the mentioned incrementally learning task methods, it is assumed that there are several tasks/environments that, with the help of the experience gained on one or multiple tasks/environments, we seek to accelerate the learning of the same task in the new environment or the new task in the same environment.

As a result, the most important hypothesis is that tasks/environments are very similar to each other, which is not always consistent with reality and deprives us of the possibility of using these methods on any task and any environment.

Human-Inspired Framework To Accelerate Reinforcement Learning

Abstract: While deep reinforcement learning (RL) is becoming an integral part of good decision-making in data science, it is still plagued with sample inefficiency. This can be challenging when applying deep-RL in real-world environments where physical interactions are expensive and can risk system safety. To improve the sample efficiency of RL algorithms, this paper proposes a novel humaninspired framework that facilitates fast exploration and learning for difficult RL tasks. The main idea is to first provide the learning agent with simpler but similar tasks that gradually grow in difficulty and progress toward the main task. The proposed method requires no pre-training phase. Specifically, the learning of simpler tasks is only done for one iteration. The generated knowledge could be used by any transfer learning, including value transfer and policy transfer, to reduce the sample complexity while not adding to the computational complexity. So, it can be applied to any goal, environment, and reinforcement learning algorithm-both value-based methods and policy-based methods and both tabular methods and deep-RL methods. We have evaluated our proposed framework on both a simple Random Walk for illustration purposes and on more challenging optimal control problems with constraint. The experiments show the good performance of our proposed framework in improving the sample efficiency of RL-learning algorithms, especially when the main task is difficult.

Overcoming Catastrophic Forgetting In Neural Networks

Abstract: The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Until now neural networks have not been capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks that they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on a hand-written digit dataset and by learning several Atari 2600 games sequentially.


The Landau-Lifshitz-Gilbert-Equation. The Strong Form Of The Llg Equations Is To

Note that in case one of the weak solutions is also a strong solution, we have uniqueness #REFR .

For ease of notation we will omit the argument t in the subsequel and arrive formally at the following weak equation for m

α(∂ t m, ϕ) Ω + (m × ∂ t m, ϕ) Ω + C e (∇m, ∇ϕ) Ω = (H ext , ϕ) Ω for all ϕ ∈ T(m) .

Existence of weak solutions m ∈ H 1 (Ω) 3 is well-known (see, e.g., [27] ) and follows with a standard Galerkin-Fadeo approximation and compactness arguments.

Uniqueness of weak solutions is open in general and there are non-uniqueness results for the related harmonic map heat flow #OTHEREFR .

In terms of regularity, it is suspected that even smooth initial conditions can lead to blow up of ∇m in L ∞ (Ω) #OTHEREFR in finite time, however, no proof is known.

For LLG on smooth bounded domains with Neumann boundary conditions, smooth initial data close to constants lead to arbitrarily smooth solutions #OTHEREFR .

Adaptive Mesh Refinement For The Landau-Lifshitz-Gilbert Equation

Abstract: We propose a new adaptive algorithm for the approximation of the Landau-Lifshitz-Gilbert equation via a higher-order tangent plane scheme. We show that the adaptive approximation satisfies an energy inequality and demonstrate numerically, that the adaptive algorithm outperforms uniform approaches.

Weak-Strong Uniqueness For The Landau-Lifshitz-Gilbert Equation In Micromagnetics

Abstract: We consider the time-dependent Landau-Lifshitz-Gilbert equation. We prove that each weak solution coincides with the (unique) strong solution, as long as the latter exists in time. Unlike available results in the literature, our analysis also includes the physically relevant lower-order terms like Zeeman contribution, anisotropy, stray field, and the Dzyaloshinskii-Moriya interaction (which accounts for the emergence of magnetic Skyrmions). Moreover, our proof gives a template on how to approach weak-strong uniqueness for even more complicated problems, where LLG is (nonlinearly) coupled to other (nonlinear) PDE systems.


The pixel-shuffle downsampling (PD) #REFR also loosens the spatial correlation of the noise, but it generates severe checkerboard artifacts.

We propose a novel loss function that can be directly optimized on the input image without loss of information.

It consists of self-supervised loss and downsampled invariance loss that controls the extent of the blindness.

Our main idea of the downsampled invariance loss is to make a blind-spot network serve as regularization of the same network while preserving network parameters.

To this end, we propose a conditional blind-spot network (denoted C-BSN in the figure) to selectively mask the center pixel in the receptive field.

In addition, we introduce Random Subsampler (RS) to decorrelate noise spatially.

On the contrary, since our RS draws a pixel randomly from each grid, it does not produce such artifacts.

We denote the noisy input image as x and the corresponding clean image as y.

For brevity, the channel dimension is omitted, and spatial dimensions are vectorized, i.e., x, y ∈ R m .

Self-Supervised Image Denoising With Downsampled Invariance Loss And Conditional Blind-Spot Network

Abstract: There have been many image denoisers using deep neural networks, which outperform conventional model-based methods by large margins. Recently, self-supervised methods have attracted attention because constructing a large real noise dataset for supervised training is an enormous burden. The most representative self-supervised denoisers are based on blind-spot networks, which exclude the receptive field's center pixel. However, excluding any input pixel is abandoning some information, especially when the input pixel at the corresponding output position is excluded. In addition, a standard blind-spot network fails to reduce real camera noise due to the pixel-wise correlation of noise, though it successfully removes independently distributed synthetic noise. Hence, to realize a more practical denoiser, we propose a novel self-supervised training framework that can remove real noise. For this, we derive the theoretic upper bound of a supervised loss where the network is guided by the downsampled blinded output. Also, we design a conditional blind-spot network (C-BSN), which selectively controls the blindness of the network to use the center pixel information. Furthermore, we exploit a random subsampler to decorrelate noise spatially, making the C-BSN free of visual artifacts that were often seen in downsample-based methods. Extensive experiments show that the proposed C-BSN achieves state-of-the-art performance on real-world datasets as a self-supervised denoiser and shows qualitatively pleasing results without any postprocessing or refinement.

When Awgn-Based Denoiser Meets Real Noises

Abstract: Discriminative learning based image denoisers have achieved promising performance on synthetic noise such as the additive Gaussian noise. However, their performance on images with real noise is often not satisfactory. The main reason is that real noises are mostly spatially/channel-correlated and spatial/channel-variant. In contrast, the synthetic Additive White Gaussian Noise (AWGN) adopted in most previous work is pixel-independent. In this paper, we propose a novel approach to boost the performance of a real image denoiser which is trained only with synthetic pixel-independent noise data. First, we train a deep model that consists of a noise estimator and a denoiser with mixed AWGN and Random Value Impulse Noise (RVIN). We then investigate Pixel-shuffle Down-sampling (PD) strategy to adapt the trained model to real noises. Extensive experiments demonstrate the effectiveness and generalization ability of the proposed approach. Notably, our method achieves state-of-the-art performance on real sRGB images in the DND benchmark. Codes are available at this https URL.


Our initial analysis of AppTracker #REFR used a different inferred admixture model, a restricted set of generic temporal properties, and segmented data sets. There was no inductive coding nor menu redesign.

We developed our analytics over a number of years: refining the Markovian models, the temporal properties, and segmentation of data sets.

Our first study #OTHEREFR involved the iOS multi-player game Hungry Yoshi #OTHEREFR , one data set, a simple inferred model, simple temporal logic properties, and K = 2.

The app was different to AppTracker in that it had clear (user) goal states and (in addition to user interactions) external events when the device picked up scanned Wi-Fi access points.

We uncovered two interaction styles representing different strategies for playing the game, but could not implement a redesign because Apple's iOS changes meant we could no longer scan for Wi-Fi access points. The app is no longer under development.

The GPAM model was defined in #OTHEREFR , where we also introduced the possibility of logic formulae over the latent variables, though we did not consider the probability to transition between activity patterns.

Significantly, none of our earlier work involved specific design recommendations, the implementation and deployment of a new design, nor analysis of interaction styles in the new design.

Probabilistic Formal Modelling To Uncover And Interpret Interaction Styles

Abstract: We present a study using new computational methods, based on a novel combination of machine learning for inferring admixture hidden Markov models and probabilistic model checking, to uncover interaction styles in a mobile app. These styles are then used to inform a redesign, which is implemented, deployed, and then analysed using the same methods. The data sets are logged user traces, collected over two six-month deployments of each version, involving thousands of users and segmented into different time intervals. The methods do not assume tasks or absolute metrics such as measures of engagement, but uncover the styles through unsupervised inference of clusters and analysis with probabilistic temporal logic. For both versions there was a clear distinction between the styles adopted by users during the first day/week/month of usage, and during the second and third months, a result we had not anticipated.

Probabilistic Formal Analysis Of App Usage To Inform Redesign

Abstract: Evaluation and redesign of user-intensive mobile applications is challenging because users are often heterogeneous, adopting different patterns of activity, at different times. We set out a process of integrating statistical, longitudinal analysis of actual logged behaviours, formal, probabilistic discrete state models of activity patterns, and hypotheses over those models expressed as probabilistic temporal logic properties to inform redesign. We employ formal methods not to the design of the mobile application, but to characterise the different probabilistic patterns of actual use over various time cuts within a population of users. We define the whole process from identifying questions that give us insight into application usage, to event logging, data abstraction from logs, model inference, temporal logic property formulation, visualisation of results, and interpretation in the context of redesign. We illustrate the process through a real-life case study, which results in a new and principled way for selecting content for an extension to the mobile application.


Then, we use the corresponding ECGs to train a multi-label classifier with ResNet-34 #REFR architecture and 1-D convolutions.

We conduct various evaluations to compare our model against other baseline models: Diffusion model, Tacotron 2 + Griffin-Lim, and Tacotron 2 + Hifi-GAN.

The first evaluation is conducted to test how well the generated ECGs can capture representative cardiac diagnoses when they are given as textual input.

We first collect most frequent textual prompts with each diagnosis in our dataset, which is determined with a Word N-gram of N between 4 and 10.

Finally, we generate 128 ECG samples for each diagnosis with the collected textual prompts.

The classification results of different models are shown in Table 1 .

The second evaluation is conducted to assess text-ECG alignment performance, and is based on Contrastive Language-Image Pre-training (CLIP) #OTHEREFR method.

Instead of training an image encoder, we train an ECG encoder jointly with a text encoder to predict correct pairings of ECG and text.

We then generate ECGs based on full text reports in the test set with each model.

Text-To-Ecg: 12-Lead Electrocardiogram Synthesis Conditioned On Clinical Text Reports

Abstract: Electrocardiogram (ECG) synthesis is the area of research focused on generating realistic synthetic ECG signals for medical use without concerns over annotation costs or clinical data privacy restrictions. Traditional ECG generation models consider a single ECG lead and utilize GAN-based generative models. These models can only generate single lead samples and require separate training for each diagnosis class. The diagnosis classes of ECGs are insufficient to capture the intricate differences between ECGs depending on various features (e.g. patient demographic details, co-existing diagnosis classes, etc.). To alleviate these challenges, we present a text-to-ECG task, in which textual inputs are used to produce ECG outputs. Then we propose Auto-TTE, an autoregressive generative model conditioned on clinical text reports to synthesize 12-lead ECGs, for the first time to our knowledge. We compare the performance of our model with other representative models in text-to-speech and text-to-image. Experimental results show the superiority of our model in various quantitative evaluations and qualitative analysis. Finally, we conduct a user study with three board-certified cardiologists to confirm the fidelity and semantic alignment of generated samples. our code will be available at https:

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


This benchmark is proposed by #REFR which removes hypernyms and redundant synonyms in the 50-way benchmark, giving a cleaner relationship schema.

This benchmark contains the 50 most frequent relationships and 150 object classes in Visual Genome, and is widely used in many supervised VRD works #OTHEREFR .

This benchmark is proposed in #OTHEREFR by removing the 25 most frequent relationships from the 50-way benchmark.

Many existing few-shot VRD works #OTHEREFR evaluated their approach on this benchmark and use samples of removed relationships to pre-train their models.

We compare our method extensively with other baselines on all three benchmarks and conduct ablation studies on the 25-way benchmark with 5 training examples.

We refer readers to the appendix for the full list of relationships used in each benchmark and other statistics.

We evaluate the overall VRD performance of our method using Recall@k (R@k) and mean-Recall@k (mR@k), which measure how many labeled visual relations are recalled.

To support multi-dimensional and more fine-grained evaluation on the methods' generalization capability, we devise novel Seen/Unseen Recall metrics.

Knowledge-Augmented Few-Shot Visual Relation Detection

Abstract: Visual Relation Detection (VRD) aims to detect relationships between objects for image understanding. Most existing VRD methods rely on thousands of training samples of each relationship to achieve satisfactory performance. Some recent papers tackle this problem by few-shot learning with elaborately designed pipelines and pre-trained word vectors. However, the performance of existing few-shot VRD models is severely hampered by the poor generalization capability, as they struggle to handle the vast semantic diversity of visual relationships. Nonetheless, humans have the ability to learn new relationships with just few examples based on their knowledge. Inspired by this, we devise a knowledge-augmented, few-shot VRD framework leveraging both textual knowledge and visual relation knowledge to improve the generalization ability of few-shot VRD. The textual knowledge and visual relation knowledge are acquired from a pre-trained language model and an automatically constructed visual relation knowledge graph, respectively. We extensively validate the effectiveness of our framework. Experiments conducted on three benchmarks from the commonly used Visual Genome dataset show that our performance surpasses existing state-of-the-art models with a large improvement.

Scene Graph Prediction With Limited Labels

Abstract: Visual knowledge bases such as Visual Genome power numerous applications in computer vision, including visual question answering and captioning, but suffer from sparse, incomplete relationships. All scene graph models to date are limited to training on a small set of visual relationships that have thousands of training labels each. Hiring human annotators is expensive, and using textual knowledge base completion methods are incompatible with visual data. In this paper, we introduce a semi-supervised method that assigns probabilistic relationship labels to a large number of unlabeled images using few labeled examples. We analyze visual relationships to suggest two types of image-agnostic features that are used to generate noisy heuristics, whose outputs are aggregated using a factor graph-based generative model. With as few as 10 labeled examples per relationship, the generative model creates enough training data to train any existing state-of-the-art scene graph model. We demonstrate that our method outperforms all baseline approaches on scene graph prediction by5.16 recall@100 for PREDCLS. In our limited label setting, we define a complexity metric for relationships that serves as an indicator (R^2 = 0.778) for conditions under which our method succeeds over transfer learning, the de-facto approach for training with limited labels.


FinBert is a language model based on BERT, built to tackle NLP problems in the financial domain #REFR .

The sentiment prediction in this paper is done using a fine-tuned FinBERT-based model.

On various NLP tasks, pre-trained generic language models #OTHEREFR - #OTHEREFR have produced excellent results.

Such models can be used for practically any task after being unsupervised and trained on massive quantities of text.

One of the most effective language models now in use is BERT #OTHEREFR . The transformer encoder is the foundation of this concept.

The Transformer is a sequence-tosequence architecture that only relies on decoder and encoder attention methods.

Since the BERT design is not a sequence-to-sequence model, it ignores the decoder network and uses simply a transformer encoder (although it can be used in such tasks).

Cryptocurrency Price Prediction Using Twitter Sentiment Analysis

Abstract: The cryptocurrency ecosystem has been the centre of discussion on many social media platforms, following its noted volatility and varied opinions. Twitter is rapidly being utilised as a news source and a medium for bitcoin discussion. Our algorithm seeks to use historical prices and sentiment of tweets to forecast the price of Bitcoin. In this study, we develop an end-to-end model that can forecast the sentiment of a set of tweets (using a Bidirectional Encoder Representations from Transformers-based Neural Network Model) and forecast the price of Bitcoin (using Gated Recurrent Unit) using the predicted sentiment and other metrics like historical cryptocurrency price data, tweet volume, a user's following, and whether or not a user is verified. The sentiment prediction gave a Mean Absolute Percentage Error of 9.45%, an average of real-time data, and test data. The mean absolute percent error for the price prediction was 3.6%.

Finbert: Financial Sentiment Analysis With Pre-Trained Language Models

Abstract: Financial sentiment analysis is a challenging task due to the specialized language and lack of labeled data in that domain. General-purpose models are not effective enough because of the specialized language used in a financial context. We hypothesize that pre-trained language models can help with this problem because they require fewer labeled examples and they can be further trained on domain-specific corpora. We introduce FinBERT, a language model based on BERT, to tackle NLP tasks in the financial domain. Our results show improvement in every measured metric on current state-of-the-art results for two financial sentiment analysis datasets. We find that even with a smaller training set and fine-tuning only a part of the model, FinBERT outperforms state-of-the-art machine learning methods.


Each batch consists of 32 samples and the weights of the network are optimized using the Adam #REFR optimization algorithm with a weight decay factor of 5 × 10 −5 .

Hence, 20% of the data are reserved for testing, while the remaining 80% are used for training the model.

Again, an 80/20 split is performed on the remaining four folds to generate the training and the validation set.

Each split is performed in a such a manner that the number of examples for each class is identical in each partition.

When the model is trained for either one of the operating conditions listed in Tab.

II, 50,000 samples are available and thus the number of samples in the test, the training and the validation set correspond to (N test , N train , N val ) = (10000, 32000, 8000), which corresponds to 2000, 6400 and 1600 data samples per class label.

The weight decay factor is a regularization parameter that is multiplied with the L2 norm of the weight matrix.

Weight decay encourages the optimizer to keep the weights small, which can prevent overfitting and improve generalization ability #OTHEREFR .

The initial learning rate is set to λ = 0.001 but is decreased by a factor of 0.8, whenever the accuracy on the validation set has not increased by at least 0.01% over the last 5 epochs.

To even further mitigate the risk of overfitting, an early stopping scheme is implemented.

The model is trained for a minimum number of epochs, i.e., 65 epochs, but then training is stopped when the validation accuracy has not improved over the most recent 25 epochs. The maximum number of epochs is 120.

-D Residual Convolutional Neural Network Coupled With Data Augmentation And Regularization Techniques For The Icphm 2023 Data Challenge

Abstract: In this article, we present our contribution to the International Conference on Prognostics and Health Management (ICPHM) 2023 Data Challenge on Industrial Systems' Health Monitoring using Vibration Analysis. For the task of classifying sun gear faults in a gearbox, we propose a residual Convolutive Neural Network (CNN) that operates on raw three-channel time-domain vibration signals. In conjunction with data augmentation and regularization techniques, the proposed model yields very good results in a multi-class classification scenario with realworld data despite its relatively small size, i.e., with less than 30,000 trainable parameters. Even when presented with data obtained from multiple operating conditions, the network is still capable to accurately predict the condition of the gearbox under inspection.


Embedded-Compute-Ready Obstacle Detection Network Ewasr

The channel refinement module (CRM) applies the cARM #REFR module to enable a global cross-channel information flow.

However, the transformer applies token cross-attention #OTHEREFR , which still adds a computationally prohibitive overhead.

We propose a more efficient feature mixer that draws on findings of Yu et al.

#OTHEREFR that computationally-costly token cross-attentions in transformers can be replaced by alternative operations, as long as they implement cross-token information flow.

We thus propose a Light-weight Scale-Aware Semantic Extractor (LSSE) for the eWaSR feature mixer, which is composed of two metaformer refinement modules CRM and SRM (Figure 3) .

Both modules follow the metaformer #OTHEREFR structure and differ in the type of information flow implemented by the token mixer.

This is followed by a spatial refinement module (SRM), which applies sARM #OTHEREFR to enable cross-channel spatial information flow.

To make the LSSE suitable for our target hardware, we replace the commonlyused GeLU and layer normalization blocks of metaformer by ReLU and batch normalization.

The proposed LSSE is much more computationally efficent than SSE #OTHEREFR .

For example, with a ResNet-18 encoder, the SSE would contain 66.4M parameters (requires 3.2GFlops), while the LSSE contains 47.9M parameters (requires 2.1GFlops).

We also simplify the WaSR decoder following the TopFormer #OTHEREFR semantic-enrichment routines to avoid the computationally-costly FFM and ASPP modules.

Ewasr -An Embedded-Compute-Ready Maritime Obstacle Detection Network

Abstract: This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY

Bisenet: Bilateral Segmentation Network For Real-Time Semantic Segmentation

Abstract: Semantic segmentation requires both rich spatial information and sizeable receptive field. However, modern approaches usually compromise spatial resolution to achieve real-time inference speed, which leads to poor performance. In this paper, we address this dilemma with a novel Bilateral Segmentation Network (BiSeNet). We first design a Spatial Path with a small stride to preserve the spatial information and generate high-resolution features. Meanwhile, a Context Path with a fast downsampling strategy is employed to obtain sufficient receptive field. On top of the two paths, we introduce a new Feature Fusion Module to combine features efficiently. The proposed architecture makes a right balance between the speed and segmentation performance on Cityscapes, CamVid, and COCO-Stuff datasets. Specifically, for a 2048 \(\times \) 1024 input, we achieve 68.4% Mean IOU on the Cityscapes test dataset with speed of 105 FPS on one NVIDIA Titan XP card, which is significantly faster than the existing methods with comparable performance.


For more details on these terms, please refer to the original paper of SMPLify-X #REFR .

For the hands, SMPL-X uses a low-dimensional PCA pose space such that

where M are principal components capturing the finger pose variations and m hn are the corresponding PCA coefficients.

Thus, E m h (m h ) is an L2 prior on the coefficients m h .

E J represents the joint re-projection loss, and E α (θ b ) is a prior penalizing extreme bending only for elbows and knees.

E O is a bone-orientation term, which factors out the residual of the parent joint from the residual of the child joint.

For more details about this term, please refer to the original paper of RICH #OTHEREFR .

E P and E A are used to prevent self-interpenetration.

When self-contact occurs, the E P term pushes vertices that are inside the mesh to the surface, and E A aligns the surface normals of the vertices in contact.

For more details, please refer to the original paper of TUCH #OTHEREFR .

Reconstructing Signing Avatars From Video Using Linguistic Priors

Abstract: Figure 1. Given a monocular, in-the-wild video of a sign-language sign, SGNify automatically reconstructs a 3D body with accurate hand pose, facial motion, and body pose. Note that motion blur obscures the finger articulations in several video frames; this is a common problem. Our novel linguistic priors enable accurate 3D reconstruction despite such image degradation.

Expressive Body Capture: 3D Hands, Face, And Body From A Single Image

Abstract: To facilitate the analysis of human actions, interactions and emotions, we compute a 3D model of human body pose, hand pose, and facial expression from a single monocular image. To achieve this, we use thousands of 3D scans to train a new, unified, 3D model of the human body, SMPL-X, that extends SMPL with fully articulated hands and an expressive face. Learning to regress the parameters of SMPL-X directly from images is challenging without paired images and 3D ground truth. Consequently, we follow the approach of SMPLify, which estimates 2D features and then optimizes model parameters to fit the features. We improve on SMPLify in several significant ways: (1) we detect 2D features corresponding to the face, hands, and feet and fit the full SMPL-X model to these; (2) we train a new neural network pose prior using a large MoCap dataset; (3) we define a new interpenetration penalty that is both fast and accurate; (4) we automatically detect gender and the appropriate body models (male, female, or neutral); (5) our PyTorch implementation achieves a speedup of more than 8x over Chumpy. We use the new method, SMPLify-X, to fit SMPL-X to both controlled images and images in the wild. We evaluate 3D accuracy on a new curated dataset comprising 100 images with pseudo ground-truth. This is a step towards automatic expressive human capture from monocular RGB data. The models, code, and data are available for research purposes at this https URL.


We note that through optimized FuxiCTR implementation and sufficient hyper-parameter tuning, we obtain much better model performance than what was reported in #REFR 3 .

Implementation We reuse the baseline models and implement our models based on FuxiCTR #OTHEREFR , an open-source CTR prediction library 2 .

Our evaluation follows the same experimental settings with AFN #OTHEREFR , by setting the embedding dimension to 10, batch size to 4096, and the default MLP size to [400, 400, 400].

For DualMLP and FinalMLP, we tune the two MLPs in 1∼3 layers to enhance stream diversity.

We set the learning rate to 1e − 3 or 5e − 4.

We tune all the other hyper-parameters (e.g., embedding regularization and dropout rate) of all the studied models via extensive grid search (about 30 runs per model on average).

Thus, we report our own experimental results instead of reusing theirs to make a fair comparison.

To promote reproducible research, we open sourced the code and running logs of FinalMLP and all the baselines used.

Finalmlp: An Enhanced Two-Stream Mlp Model For Ctr Prediction

Abstract: Click-through rate (CTR) prediction is one of the fundamental tasks for online advertising and recommendation. While multi-layer perceptron (MLP) serves as a core component in many deep CTR prediction models, it has been widely recognized that applying a vanilla MLP network alone is inefficient in learning multiplicative feature interactions. As such, many two-stream interaction models (e.g., DeepFM and DCN) have been proposed by integrating an MLP network with another dedicated network for enhanced CTR prediction. As the MLP stream learns feature interactions implicitly, existing research focuses mainly on enhancing explicit feature interactions in the complementary stream. In contrast, our empirical study shows that a well-tuned two-stream MLP model that simply combines two MLPs can even achieve surprisingly good performance, which has never been reported before by existing work. Based on this observation, we further propose feature selection and interaction aggregation layers that can be easily plugged to make an enhanced two-stream MLP model, FinalMLP. In this way, it not only enables differentiated feature inputs but also effectively fuses stream-level interactions across two streams. Our evaluation results on four open benchmark datasets as well as an online A/B test in our industrial system show that FinalMLP achieves better performance than many sophisticated two-stream CTR models. Our source code will be available at MindSpore/models and FuxiCTR/model zoo.

Adaptive Factorization Network: Learning Adaptive-Order Feature Interactions

Abstract: Various factorization-based methods have been proposed to leverage second-order, or higher-order cross features for boosting the performance of predictive models. They generally enumerate all the cross features under a predefined maximum order, and then identify useful feature interactions through model training, which suffer from two drawbacks. First, they have to make a trade-off between the expressiveness of higher-order cross features and the computational cost, resulting in suboptimal predictions. Second, enumerating all the cross features, including irrelevant ones, may introduce noisy feature combinations that degrade model performance. In this work, we propose the Adaptive Factorization Network (AFN), a new model that learns arbitrary-order cross features adaptively from data. The core of AFN is a logarithmic transformation layer to convert the power of each feature in a feature combination into the coefficient to be learned. The experimental results on four real datasets demonstrate the superior predictive performance of AFN against the start-of-the-arts.


Following the same training strategy and hyperparameters in IRN baseline, our models were trained on the DIV2K #REFR dataset, which includes 800 HR training images.

IRN-M and IRN-A were trained with 500,000 and 250,000 iterations respectively.

Both models were evaluated across five benchmark datasets: Set5 #OTHEREFR , Set14 #OTHEREFR , BSD100 #OTHEREFR , Ur-ban100 #OTHEREFR and the validation set of DIV2K.

The upscaled images quality across different models were assessed via the peak noise-signal ratio (PSNR) and SSIM on the Y channel of the YCbCr color space.

Following previous works #OTHEREFR , as it is not beneficial to add randomness in restoring HR images, we setẑ as 0 during the inverse up-scaling process for both training and validation steps in all experiments.

Raising The Limit Of Image Rescaling Using Auxiliary Encoding

Abstract: Normalizing flow models using invertible neural networks (INN) have been widely investigated for successful generative image super-resolution (SR) by learning the transformation between the normal distribution of latent variable z and the conditional distribution of high-resolution (HR) images gave a low-resolution (LR) input. Recently, image rescaling models like IRN utilize the bidirectional nature of INN to push the performance limit of image upscaling by optimizing the downscaling and upscaling steps jointly. While the random sampling of latent variable z is useful in generating diverse photo-realistic images, it is not desirable for image rescaling when accurate restoration of the HR image is more important. Hence, in places of random sampling of z, we propose auxiliary encoding modules to further push the limit of image rescaling performance. Two options to store the encoded latent variables in downscaled LR images, both readily supported in existing image file format, are proposed. One is saved as the alpha-channel, the other is saved as meta-data in the image header, and the corresponding modules are denoted as suffixes-A and-M respectively. Optimal network architectural changes are investigated for both options to demonstrate their effectiveness in raising the rescaling performance limit on different baseline models including IRN and DLV-IRN.

Blind Super-Resolution Kernel Estimation Using An Internal-Gan

Abstract: Super resolution (SR) methods typically assume that the low-resolution (LR) image was downscaled from the unknown high-resolution (HR) image by a fixed `ideal’ downscaling kernel (e.g. Bicubic downscaling). However, this is rarely the case in real LR images, in contrast to synthetically generated SR datasets. When the assumed downscaling kernel deviates from the true one, the performance of SR methods significantly deteriorates. This gave rise to Blind-SR - namely, SR when the downscaling kernel (``SR-kernel’’) is unknown. It was further shown that the true SR-kernel is the one that maximizes the recurrence of patches across scales of the LR image. In this paper we show how this powerful cross-scale recurrence property can be realized using Deep Internal Learning. We introduce ``KernelGAN’’, an image-specific Internal-GAN, which trains solely on the LR test image at test time, and learns its internal distribution of patches. Its Generator is trained to produce a downscaled version of the LR test image, such that its Discriminator cannot distinguish between the patch distribution of the downscaled image, and the patch distribution of the original LR image. The Generator, once trained, constitutes the downscaling operation with the correct image-specific SR-kernel. KernelGAN is fully unsupervised, requires no training data other than the input image itself, and leads to state-of-the-art results in Blind-SR when plugged into existing SR algorithms.


We seek the real foggy dataset collected by DenseFogFlow #REFR , namely DenseFog, of which 2346 images for training and 100 images for testing.

We select 8400 images of Fog-KITTI2015 dataset for training and 400 images for testing.

GOF #OTHEREFR is a dataset containing four different scenes with synchronized gyro readings, such as regular scenes and adverse weather scenes.

We choose foggy images of GOF to compose a new foggy dataset, namely Fog-GOF, of which 1000 images for training and 105 images for testing.

We collect degraded videos under real foggy scenes from Youtube, with 1200 and 240 images for training and testing, respectively. Comparison Methods.

We choose three competing methods GyroFlow #OTHEREFR , DenseFogFlow #OTHEREFR and RobustFlow #OTHEREFR which are designed for adverse weather optical flow.

Moreover, we select several state-of-the-art supervised (GMA #OTHEREFR ) and unsupervised (SMURF #OTHEREFR , UFlow #OTHEREFR and Selflow #OTHEREFR ) optical flow approaches designed for clean scenes.

The unsupervised methods are first trained on KITTI2015 for initialization and re-trained on the target degraded dataset.

Unsupervised Cumulative Domain Adaptation For Foggy Scene Optical Flow

Abstract: Optical flow has achieved great success under clean scenes, but suffers from restricted performance under foggy scenes. To bridge the clean-to-foggy domain gap, the existing methods typically adopt the domain adaptation to transfer the motion knowledge from clean to synthetic foggy domain. However, these methods unexpectedly neglect the synthetic-to-real domain gap, and thus are erroneous when applied to real-world scenes. To handle the practical optical flow under real foggy scenes, in this work, we propose a novel unsupervised cumulative domain adaptation optical flow (UCDA-Flow) framework: depth-association motion adaptation and correlation-alignment motion adaptation. Specifically, we discover that depth is a key ingredient to influence the optical flow: the deeper depth, the inferior optical flow, which motivates us to design a depth-association motion adaptation module to bridge the clean-to-foggy domain gap. Moreover, we figure out that the cost volume correlation shares similar distribution of the synthetic and real foggy images, which enlightens us to devise a correlation-alignment motion adaptation module to distill motion knowledge of the synthetic foggy domain to the real foggy domain. Note that synthetic fog is designed as the intermediate domain. Under this unified framework, the proposed cumulative adaptation progressively transfers knowledge from clean scenes to real foggy scenes. Extensive experiments have been performed to verify the superiority of the proposed method.

Optical Flow In Dense Foggy Scenes Using Semi-Supervised Learning

Abstract: In dense foggy scenes, existing optical flow methods are erroneous. This is due to the degradation caused by dense fog particles that break the optical flow basic assumptions such as brightness and gradient constancy. To address the problem, we introduce a semi-supervised deep learning technique that employs real fog images without optical flow ground-truths in the training process. Our network integrates the domain transformation and optical flow networks in one framework. Initially, given a pair of synthetic fog images, its corresponding clean images and optical flow ground-truths, in one training batch we train our network in a supervised manner. Subsequently, given a pair of real fog images and a pair of clean images that are not corresponding to each other (unpaired), in the next training batch, we train our network in an unsupervised manner. We then alternate the training of synthetic and real data iteratively. We use real data without ground-truths, since to have ground-truths in such conditions is intractable, and also to avoid the overfitting problem of synthetic data training, where the knowledge learned on synthetic data cannot be generalized to real data testing. Together with the network architecture design, we propose a new training strategy that combines supervised synthetic-data training and unsupervised real-data training. Experimental results show that our method is effective and outperforms the state-of-the-art methods in estimating optical flow in dense foggy scenes.


While MOL-LR 3D performs recovery in 3D, MoDL #REFR could do it only in 2D due to lack of memory for multiple unrolls. MOL-LR 3D outperforms MoDL 2D as shown in Fig. 2 .

By contrast, the experiments in the second row of Fig.

1(a) show that MOL-SN and MOL-LR are relatively more robust to adversarial noise, compared to MoDL, UNET, and ADMM-Net.

Both MOL-SN and MOL-LR are associated with guarantees on robustness. The plots in Fig.

1(b) shows drop in PSNR with respect to percentage of Adversarial/Gaussian noise (0% to 20% for = [0.0, 0.2]) and shows similar trend.

We demonstrate the benefit of reduced memory demand in MOL over unrolled networks through experiments on 2D+time cardiac data, run on a 16GB GPU.

MoDL 2D's poor performance can be attributed to lack of temporal information which is well utilized by MOL-LR 3D.

Accelerated Parallel Mri Using Memory Efficient And Robust Monotone Operator Learning (Mol)

Abstract: Model-based deep learning methods that combine imaging physics with learned regularization priors have been emerging as powerful tools for parallel MRI acceleration. The main focus of this paper is to determine the utility of the monotone operator learning (MOL) framework in the parallel MRI setting. The MOL algorithm alternates between a gradient descent step using a monotone convolutional neural network (CNN) and a conjugate gradient algorithm to encourage data consistency. The benefits of this approach include similar guarantees as compressive sensing algorithms including uniqueness, convergence, and stability, while being significantly more memory efficient than unrolled methods. We validate the proposed scheme by comparing it with different unrolled algorithms in the context of accelerated parallel MRI for static and dynamic settings.

Modl: Model Based Deep Learning Architecture For Inverse Problems

Abstract: We introduce a model-based image reconstruction framework with a convolution neural network (CNN)-based regularization prior. The proposed formulation provides a systematic approach for deriving deep architectures for inverse problems with the arbitrary structure. Since the forward model is explicitly accounted for, a smaller network with fewer parameters is sufficient to capture the image information compared to direct inversion approaches. Thus, reducing the demand for training data and training time. Since we rely on end-to-end training with weight sharing across iterations, the CNN weights are customized to the forward model, thus offering improved performance over approaches that rely on pre-trained denoisers. Our experiments show that the decoupling of the number of iterations from the network complexity offered by this approach provides benefits, including lower demand for training data, reduced risk of overfitting, and implementations with significantly reduced memory footprint. We propose to enforce data-consistency by using numerical optimization blocks, such as conjugate gradients algorithm within the network. This approach offers faster convergence per iteration, compared to methods that rely on proximal gradients steps to enforce data consistency. Our experiments show that the faster convergence translates to improved performance, primarily when the available GPU memory restricts the number of iterations.


We substitute PTI's locality regularization with our codebook-perseverance regularization: we push randomly sampled codes to keep their syntheses unchanged via an LPIPS #REFR loss.

We extend the Pivotal Tuning Inversion #OTHEREFR to project real images into our codebook in Fig. 3 .

We start by concurrently optimizing a set of vectors in the underlying continuous space to produce the images we aim to invert.

Then, we find the indices of the nearest-neighbor variant for each gene in the codebook.

We train both the generator and the codebook to recreate the images, based only on these indices.

We find this step important to retain the perceptual quality of the genome.

Stylegenes: Discrete And Efficient Latent Distributions For Gans

Abstract: We propose a discrete latent distribution for Generative Adversarial Networks (GANs). Instead of drawing latent vectors from a continuous prior, we sample from a finite set of learnable latents. However, a direct parametrization of such a distribution leads to an intractable linear increase in memory in order to ensure sufficient sample diversity. We address this key issue by taking inspiration from the encoding of information in biological organisms. Instead of learning a separate latent vector for each sample, we split the latent space into a set of genes. For each gene, we train a small bank of gene variants. Thus, by independently sampling a variant for each gene and combining them into the final latent vector, our approach can represent a vast number of unique latent samples from a compact set of learnable parameters. Interestingly, our gene-inspired latent encoding allows for new and intuitive approaches to latent-space exploration, enabling conditional sampling from our unconditionally trained model. Moreover, our approach preserves state-of-the-art photo-realism while achieving better disentanglement than the widely-used StyleMapping network.

The Unreasonable Effectiveness Of Deep Features As A Perceptual Metric

Abstract: While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on the ImageNet classification task has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called"perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new Full Reference Image Quality Assessment (FR-IQA) dataset of perceptual human judgments, orders of magnitude larger than previous datasets. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by huge margins. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.

randomly sampled codes
PTI's locality regularization


As for fine-grained continual learning benchmarks in Table 2 , SLCA achieves the strongest performance for supervised pre-training, while for self-supervised pre-training the performance is comparable or slightly inferior to the SL version of BiC #REFR , which additionally employs some old training samples.

Overall Performance: All baseline approaches in Table 1, 2 are trained with our Slow Learner for a fair comparison, except the prompt-based approaches that keep the representation layer fixed.

For continual learning of relatively coarse-grained classification tasks, such as Split CIFAR-100 and Split ImageNet-R in Table 1 (also shown in Fig.

2 ), the SL can substantially enhance the performance of continual learning.

With the help of Classifier Alignment (CA) and its Logit Normalization (LN), our approach clearly outperforms L2P #OTHEREFR and DualPrompt #OTHEREFR , and almost reach the joint training upper bound (the performance gap is less than 2% for supervised pre-training and 4% for self-supervised pre-training).

Please note that the presented implementation of SLCA can be seen as adding CA+LN together with SL to the simplest sequential fine-tuning (Seq FT) baseline.

In fact, the proposed SLCA can be naturally plug-and-play with other continual learning approaches. We leave it as a further work.

It is worth noting that different replay-based approaches (w/ SL) behave differently in CLPM.

In general, BiC #OTHEREFR , which updated the entire model with the old training samples in a similar way to learning the new ones and added a bias correction layer to mitigate imbalance between old and new classes, receives the most substantial improvements.

While GDumb #OTHEREFR , which simply used the old training samples to train a new model from scratch at test time, has difficulty to adapt the pre-trained model with limited training samples and thus performs the worst.

Slca: Slow Learner With Classifier Alignment For Continual Learning On A Pre-Trained Model

Abstract: The goal of continual learning is to improve the performance of recognition models in learning sequentially arrived data. Although most existing works are established on the premise of learning from scratch, growing efforts have been devoted to incorporating the benefits of pre-training. However, how to adaptively exploit the pre-trained knowledge for each incremental task while maintaining its generalizability remains an open question. In this work, we present an extensive analysis for continual learning on a pre-trained model (CLPM), and attribute the key challenge to a progressive overfitting problem. Observing that selectively reducing the learning rate can almost resolve this issue in the representation layer, we propose a simple but extremely effective approach named Slow Learner with Classifier Alignment (SLCA), which further improves the classification layer by modeling the class-wise distributions and aligning the classification layers in a post-hoc fashion. Across a variety of scenarios, our proposal provides substantial improvements for CLPM (e.g., up to 49.76%, 50.05%, 44.69% and 40.16% on Split CIFAR-100, Split ImageNet-R, Split CUB-200 and Split Cars-196, respectively), and thus outperforms state-of-the-art approaches by a large margin. Based on such a strong baseline, critical factors and promising directions are analyzed in-depth to facilitate subsequent research.

Large Scale Incremental Learning

Abstract: Modern machine learning suffers from catastrophic forgetting when learning new classes incrementally. The performance dramatically degrades due to the missing data of old classes. Incremental learning methods have been proposed to retain the knowledge acquired from the old classes, by using knowledge distilling and keeping a few exemplars from the old classes. However, these methods struggle to scale up to a large number of classes. We believe this is because of the combination of two factors: (a) the data imbalance between the old and new classes, and (b) the increasing number of visually similar classes. Distinguishing between an increasing number of visually similar classes is particularly challenging, when the training data is unbalanced. We propose a simple and effective method to address this data imbalance issue. We found that the last fully connected layer has a strong bias towards the new classes, and this bias can be corrected by a linear model. With two bias parameters, our method performs remarkably well on two large datasets: ImageNet (1000 classes) and MS-Celeb-1M (10000 classes), outperforming the state-of-the-art algorithms by 11.1% and 13.2% respectively.


Researchers are especially interested in the problem of synthesizing a policy that maximizes the satisfaction probability of given high-level specifications expressed in LTL. This satisfaction problem is first introduced by #REFR .

Formal policy synthesis for stochastic systems has received substantial research attention.

Approaches to solving this satisfaction problem often fall into two categories: (a) automaton-based approach and (b) non-automaton-based approach.

Non-automaton-based approaches #OTHEREFR formulate a constrained optimization, where LTL formulas are encoded in constraints.

Such constrained optimization approaches leverage the empirical performance of state-of-art solvers and enjoy great successes in theoretical results #OTHEREFR as well as the application #OTHEREFR within the control community.

Specifically, authors #OTHEREFR propose a mixed-integer linear programming (MILP) for optimal control of nonlinear systems.

They encode LTL specifications as mixed-integer linear constraints to avoid the construction of the task automaton.

Topological Guided Actor-Critic Modular Learning Of Continuous Systems With Temporal Objectives

Abstract: This work investigates the formal policy synthesis of continuous-state stochastic dynamic systems given high-level specifications in linear temporal logic. To learn an optimal policy that maximizes the satisfaction probability, we take a product between a dynamic system and the translated automaton to construct a product system on which we solve an optimal planning problem. Since this product system has a hybrid product state space that results in reward sparsity, we introduce a generalized optimal backup order, in reverse to the topological order, to guide the value backups and accelerate the learning process. We provide the optimality proof for using the generalized optimal backup order in this optimal planning problem. Further, this paper presents an actor-critic reinforcement learning algorithm when topological order applies. This algorithm leverages advanced mathematical techniques and enjoys the property of hyperparameter self-tuning. We provide proof of the optimality and convergence of our proposed reinforcement learning algorithm. We use neural networks to approximate the value function and policy function for hybrid product state space. Furthermore, we observe that assigning integer numbers to automaton states can rank the value or policy function approximated by neural networks. To break the ordinal relationship, we use an individual neural network for each automaton state's value (policy) function, termed modular learning. We conduct two experiments. First, to show the efficacy of our reinforcement learning algorithm, we compare it with baselines on a classic control task, CartPole. Second, we demonstrate the empirical performance of our formal policy synthesis framework on motion planning of a Dubins car with a temporal specification.

Ltl Control In Uncertain Environments With Probabilistic Satisfaction Guarantees

Abstract: We present a method to generate a robot control strategy that maximizes the probability to accomplish a task. The task is given as a Linear Temporal Logic (LTL) formula over a set of properties that can be satisfied at the regions of a partitioned environment. We assume that the probabilities with which the properties are satisfied at the regions are known, and the robot can determine the truth value of a proposition only at the current region. Motivated by several results on partitioned-based abstractions, we assume that the motion is performed on a graph. To account for noisy sensors and actuators, we assume that a control action enables several transitions with known probabilities. We show that this problem can be reduced to the problem of generating a control policy for a Markov Decision Process (MDP) such that the probability of satisfying an LTL formula over its states is maximized. We provide a complete solution for the latter problem that builds on existing results from probabilistic model checking. We include an illustrative case study.


Alternative practical evaluation measures are desired, especially with the increasing prevalence of ML models #REFR .

The performance of an ML model can vary under domain shifts.

Different users may need to deploy the model on very different data distributions, with possibly widely different accuracy.

Yet, traditional evaluation methods report a single aggregate number on benchmarks or through leaderboards.

We propose to evaluate black-box classification model, not as one or few aggregate numbers, but as a surface defined on a space of input instance attributes that capture the variability of user expectations.

Indoor/outdoor, day/night, urban/rural may be attributes of input images for visual object recognition tasks.

Speaker age, gender, ethnicity/accent may be attributes of input audio for speech recognition tasks.

We call a combination of attributes in their Cartesian space an arm .

We present a Gaussian Process (GP)-based probabilistic estimator of the accuracy surface.

Robustness, Evaluation And Adaptation Of Machine Learning Models In The Wild

Abstract: This is to certify that Vihari Piratla (Roll No. 173050039) was admitted to the candidacy of Ph.D. degree on July 2017, after successfully completing all the courses required for the Ph.D. programme. The details of the course work done are given below.

Model Cards For Model Reporting

Abstract: Trained machine learning models are increasingly used to perform high-impact tasks in areas such as law enforcement, medicine, education, and employment. In order to clarify the intended use cases of machine learning models and minimize their usage in contexts for which they are not well suited, we recommend that released models be accompanied by documentation detailing their performance characteristics. In this paper, we propose a framework that we call model cards, to encourage such transparent model reporting. Model cards are short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups (e.g., race, geographic location, sex, Fitzpatrick skin type [15]) and intersectional groups (e.g., age and race, or sex and Fitzpatrick skin type) that are relevant to the intended application domains. Model cards also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information. While we focus primarily on human-centered machine learning models in the application fields of computer vision and natural language processing, this framework can be used to document any trained machine learning model. To solidify the concept, we provide cards for two supervised models: One trained to detect smiling faces in images, and one trained to detect toxic comments in text. We propose model cards as a step towards the responsible democratization of machine learning and related artificial intelligence technology, increasing transparency into how well artificial intelligence technology works. We hope this work encourages those releasing trained machine learning models to accompany model releases with similar detailed evaluation numbers and other relevant documentation.


Inspired by their findings, we propose a distance-based attention score for each i-th horizontal (Das h i ∈ R 1×W ×W ) and vertical (Das v i ∈ R 1×H×H ) local window, defined by Eqs. (10) and #REFR respectively. Here, N represents L 1 normalization.

However, softmax is computationally inefficient #OTHEREFR and lacks relevance to the natural language processing (NLP) or vision tasks.

For these reasons, some studies have attempted to replace softmax with alternative functions #OTHEREFR .

These studies empirically determined that the essential roles performed by softmax are as follows.

• Ensuring attention score to get non-negative value

Then, the local attention of each i-th local window is obtained using Eq.(12)

Egformer: Equirectangular Geometry-Biased Transformer For 360 Depth Estimation

Abstract: Estimating the depths of equirectangular (i.e. 360 •) images (EIs) is challenging given the distorted 180 • × 360 • field-of-view, which is hard to be addressed via convolutional neural network (CNN). Although a transformer with global attention achieves significant improvements over CNN for EI depth estimation task, it is computationally inefficient, which raises the need for transformer with local attention. However, to apply local attention successfully for EIs, a specific strategy, which addresses distorted equirectangular geometry and limited receptive field simultaneously, is required. Prior works have only cared either of them, resulting in unsatisfactory depths occasionally. In this paper, we propose an equirectangular geometry-biased transformer termed EGformer, which enables local attention extraction in a global manner considering the equirectangular geometry. To achieve this, we actively utilize the equirectangular geometry as the bias for the local attention instead of struggling to reduce the distortion of EIs. As compared to the most recent transformer based EI depth estimation studies, the proposed approach yields the best depth outcomes overall with the lowest computational cost and the fewest parameters, demonstrating the effectiveness of the proposed methods.

Deformable Convolutional Networks

Abstract: Convolutional neural networks (CNNs) are inherently limited to model geometric transformations due to the fixed geometric structures in their building modules. In this work, we introduce two new modules to enhance the transformation modeling capability of CNNs, namely, deformable convolution and deformable RoI pooling. Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from the target tasks, without additional supervision. The new modules can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back-propagation, giving rise to deformable convolutional networks. Extensive experiments validate the performance of our approach. For the first time, we show that learning dense spatial transformation in deep CNNs is effective for sophisticated vision tasks such as object detection and semantic segmentation. The code is released at https://github.com/msracver/Deformable-ConvNets.


Ocsdf For Implicit Shape Parametrization

Efficient data structures (e.g., using K-dtrees #REFR or Octrees (Meagher, 1980)) can mitigate this effect but do not scale well to high dimensions.

Our approach to learning the SDF contrasts with the computer graphics literature, where SDF is used to obtain the distance of a point to a surface (here defined as ∂X ).

In-deed, SDFs are usually learned in a supervised fashion, requiring the ground truth of l2 distance.

This is classically achieved using Nearest-Neighbor algorithms, which can be cumbersome, especially when the number of points is high.

Instead, OCSDF learns the SDF solely based on points contained in the support.

To illustrate this, we use models from Princeton's Model-Net10 dataset #OTHEREFR .

We sample n = 2048 within each mesh to obtain a 3d point cloud.

We fit the SDF on the point cloud with the same hyperparameters as the tabular experiment.

We use Lewiner marching algorithm #OTHEREFR as the level set of the iso-surface we plot.

Robust One-Class Classification With Signed Distance Function Using 1-Lipschitz Neural Networks

Abstract: We propose a new method, dubbed One Class Signed Distance Function (OCSDF), to perform One Class Classification (OCC) by provably learning the Signed Distance Function (SDF) to the boundary of the support of any distribution. The distance to the support can be interpreted as a normality score, and its approximation using 1-Lipschitz neural networks provides robustness bounds against l2 adversarial attacks, an underexplored weakness of deep learning-based OCC algorithms. As a result, OCSDF comes with a new metric, certified AUROC, that can be computed at the same cost as any classical AUROC. We show that OCSDF is competitive against concurrent methods on tabular and image data while being way more robust to adversarial attacks, illustrating its theoretical properties. Finally, as exploratory research perspectives, we theoretically and empirically show how OCSDF connects OCC with image generation and implicit neural surface parametrization. Our code is available at https://anonymous.4open. science/r/CSDFL

Analysis Of Approximate Nearest Neighbor Searching With Clustered Point Sets

Abstract: We present an empirical analysis of data structures for approximate nearest neighbor searching. We compare the well-known optimized kd-tree splitting method against two alternative splitting methods. The first, called the sliding-midpoint method, which attempts to balance the goals of producing subdivision cells of bounded aspect ratio, while not producing any empty cells. The second, called the minimum-ambiguity method is a query-based approach. In addition to the data points, it is also given a training set of query points for preprocessing. It employs a simple greedy algorithm to select the splitting plane that minimizes the average amount of ambiguity in the choice of the nearest neighbor for the training points. We provide an empirical analysis comparing these two methods against the optimized kd-tree construction for a number of synthetically generated data and query sets. We demonstrate that for clustered data and query sets, these algorithms can provide significant improvements over the standard kd-tree construction for approximate nearest neighbor searching.


On the privacy front, prior work has shown that I(U ; Y ) can be reduced while allowing the attacker to guess S correctly from Y with higher probability (see Example 1 in #REFR ).

To find a good privacy-preserving data processing strategy P Y |X , the privacy funnel solves the optimization

where R is a desired threshold on the utility of Y .

To apply it in data sharing problems, we could regard X as the original data, Y as the released data, and U as the secret data holder wants to protect (e.g., the fraction of a specific server type).

However, mutual information is not a good metric for either privacy or utility.

On the utility front, higher mutual information I(X; Y ) does not mean that the released data Y is a useful representation of X.

For example, Y could be an arbitrary one-to-one transformation of X.

In that case, I(X; Y ) is maximized, but the data structure could be completely destroyed.

In addition, privacy funnel #OTHEREFR only considers X and Y in discrete supports, which is too restrictive for our setting.

Summary Statistic Privacy In Data Sharing

Abstract: Data sharing between different parties has become increasingly common across industry and academia. An important class of privacy concerns that arises in data sharing scenarios regards the underlying distribution of data. For example, the total traffic volume of data from a networking company can reveal the scale of its business, which may be considered a trade secret. Unfortunately, existing privacy frameworks (e.g., differential privacy, anonymization) do not adequately address such concerns. In this paper, we propose summary statistic privacy, a framework for analyzing and protecting these summary statistic privacy concerns. We propose a class of quantization mechanisms that can be tailored to various data distributions and statistical secrets, and analyze their privacy-distortion tradeoffs under our framework. We prove corresponding lower bounds on the privacy-utility tradeoff, which match the tradeoffs of the quantization mechanism under certain regimes, up to small constant factors. Finally, we demonstrate that the proposed quantization mechanisms achieve better privacy-distortion tradeoffs than alternative privacy mechanisms on real-world datasets.

An Operational Approach To Information Leakage

Abstract: Given two random variables $X$ and $Y$ , an operational approach is undertaken to quantify the “leakage” of information from $X$ to $Y$ . The resulting measure $\mathcal {L}\left ({X \!\! \to \!\! Y}\right)$ is called maximal leakage , and is defined as the multiplicative increase, upon observing $Y$ , of the probability of correctly guessing a randomized function of $X$ , maximized over all such randomized functions. A closed-form expression for $\mathcal {L}\left ({X \!\! \to \!\! Y}\right)$ is given for discrete $X$ and $Y$ , and it is subsequently generalized to handle a large class of random variables. The resulting properties are shown to be consistent with an axiomatic view of a leakage measure, and the definition is shown to be robust to variations in the setup. Moreover, a variant of the Shannon cipher system is studied, in which performance of an encryption scheme is measured using maximal leakage. A single-letter characterization of the optimal limit of (normalized) maximal leakage is derived and asymptotically-optimal encryption schemes are demonstrated. Furthermore, the sample complexity of estimating maximal leakage from data is characterized up to subpolynomial factors. Finally, the guessing framework used to define maximal leakage is used to give operational interpretations of commonly used leakage measures, such as Shannon capacity, maximal correlation, and local differential privacy.


The reason may be that the steps in the image processing pipeline can make the noise of sRGB more complex than RAW #REFR , which leads to accuracy degradation.

The image processing pipeline includes digital gain, white balance, demosaicing, color correction, and gamma correction.

It is worth noting that quantizing the captured 14-bit RAW images to RAW images of various color encodings can be different from directly capturing them.

However, since a specific commercial camera model only support RAW capture with constant bit depth (typically 14 bits in high-end DSLR), here, we simulate their results with quantization for fast verification.

As shown in Table 4 , we can see the performance of both sRGB-dark and RAW-dark gradually increase from 8bit to 14-bit, and the results of sRGB-dark are very similar to RAW-dark.

Besides, we also notice that the results of RAWdark are slightly better than the sRGB-dark.

These quantitative results show that the high-bit property of RAW images plays a crucial part in low-light instance segmentation.

The unprocessing operation inverts sRGB images to synthetical RAW images, and noise injection simulates the corruption caused by limited photon count and imperfection of photodetectors.

As shown in Table 2 , they bring in 2.2 AP and 3.5 AP performance improvements, respectively.

Moreover, when we combine these two steps together, the accuracy increases from 31.6 AP to 38.0 AP, which is very close to the result of training with real RAW-dark images 39.0 AP.

Instance Segmentation In The Dark

Abstract: Existing instance segmentation techniques are primarily tailored for high-visibility inputs, but their performance significantly deteriorates in extremely low-light environments. In this work, we take a deep look at instance segmentation in the dark and introduce several techniques that substantially boost the low-light inference accuracy. The proposed method is motivated by the observation that noise in low-light images introduces high-frequency disturbances to the feature maps of neural networks, thereby significantly degrading performance. To suppress this "feature noise", we propose a novel learning method that relies on an adaptive weighted downsampling layer, a smooth-oriented convolutional block, and disturbance suppression learning. These components effectively reduce feature noise during downsampling and convolution operations, enabling the model to learn disturbance-invariant features. Furthermore, we discover that high-bit-depth RAW images can better preserve richer scene information in low-light conditions compared to typical camera sRGB outputs, thus supporting the use of RAW-input algorithms. Our analysis indicates that high bit

Unprocessing Images For Learned Raw Denoising

Abstract: Machine learning techniques work best when the data used for training resembles the data used for evaluation. This holds true for learned single-image denoising algorithms, which are applied to real raw camera sensor readings but, due to practical constraints, are often trained on synthetic image data. Though it is understood that generalizing from synthetic to real images requires careful consideration of the noise properties of camera sensors, the other aspects of an image processing pipeline (such as gain, color correction, and tone mapping) are often overlooked, despite their significant effect on how raw measurements are transformed into finished images. To address this, we present a technique to “unprocess” images by inverting each step of an image processing pipeline, thereby allowing us to synthesize realistic raw sensor measurements from commonly available Internet photos. We additionally model the relevant components of an image processing pipeline when evaluating our loss function, which allows training to be aware of all relevant photometric processing that will occur after denoising. By unprocessing and processing training data and model outputs in this way, we are able to train a simple convolutional neural network that has 14%-38% lower error rates and is 9×-18× faster than the previous state of the art on the Darmstadt Noise Dataset, and generalizes to sensors outside of that dataset as well.