Estimating Conditional Mutual Information With The Logging Policy (Cmip)

Given this divergence-based formulation of CMI, we employ a two-step approach suggested in #REFR .

In this section, we cover how to estimate the CMIP metric to quantify debiasedness.

First, we note that CMI can be expressed as the Kullback-Leibler divergence between two distributions:

which is a pseudo-distance between the joint distribution of all three variables occurring together and the distribution in which the predicted relevance scores˜and the relevance of the logging policies are independent, conditional on .

If the divergence between both distributions is zero, the joint distribution (which we actually observe) is equivalent to the distribution on which conditional independence holds.

First, we obtain samples from the marginal distribution on which conditional independence holds.

Second, we estimate the KL-divergence between the observed dataset and the generated samples, which is the estimate of our CMIP metric. 4.3.1 Sampling from the marginal distribution .

How can we obtain samples from the conditional independence distribution given our observational dataset ? For a proof that this methodology actually approximates , we refer to #OTHEREFR Theorem 1] .

We use a knn-based approach suggested in #OTHEREFR ; its simplicity and computational speed make it suitable for an evaluation metric.

Given a dataset of observed relevance labels for each document, R = {(˜( ), ( ), ( ))} ∈ D , we split the data into two equal parts R and R .

An Offline Metric For The Debiasedness Of Click Models

Abstract: A well-known problem when learning from user clicks are inherent biases prevalent in the data, such as position or trust bias. Click models are a common method for extracting information from user clicks, such as document relevance in web search, or to estimate click biases for downstream applications such as counterfactual learning-to-rank, ad placement, or fair ranking. Recent work shows that the current evaluation practices in the community fail to guarantee that a well-performing click model generalizes well to downstream tasks in which the ranking distribution differs from the training distribution, i.e., under covariate shift. In this work, we propose an evaluation metric based on conditional independence testing to detect a lack of robustness to covariate shift in click models. We introduce the concept of debiasedness and a metric for measuring it. We prove that debiasedness is a necessary condition for recovering unbiased and consistent relevance scores and for the invariance of click prediction under covariate shift. In extensive semi-synthetic experiments, we show that our proposed metric helps to predict the downstream performance of click models under covariate shift and is useful in an off-policy model selection setting. CCS CONCEPTS • Information systems → Query log analysis.

Ccmi : Classifier Based Conditional Mutual Information Estimation

Abstract: Conditional Mutual Information (CMI) is a measure of conditional dependence between random variables X and Y, given another random variable Z. It can be used to quantify conditional dependence among variables in many data-driven inference problems such as graphical models, causal learning, feature selection and time-series analysis. While k-nearest neighbor (kNN) based estimators as well as kernel-based methods have been widely used for CMI estimation, they suffer severely from the curse of dimensionality. In this paper, we leverage advances in classifiers and generative models to design methods for CMI estimation. Specifically, we introduce an estimator for KL-Divergence based on the likelihood ratio by training a classifier to distinguish the observed joint distribution from the product distribution. We then show how to construct several CMI estimators using this basic divergence estimator by drawing ideas from conditional generative models. We demonstrate that the estimates from our proposed approaches do not degrade in performance with increasing dimension and obtain significant improvement over the widely used KSG estimator. Finally, as an application of accurate CMI estimation, we use our best estimator for conditional independence testing and achieve superior performance than the state-of-the-art tester on both simulated and real data-sets.


The second problem we apply our scale separation algorithm to is a seismic problem where waves, or primary waves, are fast and S waves, or secondary waves are slow. In #REFR , Vogl et al.

present a method for a high-resolution seismic model of a fault slippage under the ocean.

In the Figure 4 : The DMD recreations of the fast and slow components of the toy problem right-hand side.

These recreations are fed into ARKODE as the fast and slow functions.

work they present a linear hyperbolic system of equations that model the 2D plane-strain case of isotropic linear elasticity:

Data-Driven Time-Scale Separation Of Ode Right-Hand Sides Using Dynamic Mode Decomposition And Time Delay Embedding

Abstract: Multi-physics simulation often involve multiple different scales. The ARKODE ODE solver package in the SUNDIALS library addresses multi-scale problems with a multi-rate time-integrator that can work with a right-hand side that has fast scale and slow scale components. In this report, we use dynamic mode decomposition and time delay embedding to extract the fast and and slow components of the right-hand sides of a simple ODE from data. We then use the extracted components to solve the ODE with ARKODE. Finally, to move towards a real-world use case, we attempt to extract fast and slow scale dynamics from synthetic seismic modeling data.

A High-Resolution Finite Volume Seismic Model To Generate Seafloor Deformation For Tsunami Modeling

Abstract: A high-resolution finite volume method approach to incorporating time-dependent slip across rectangular subfaults when modeling general fault geometry is presented. The fault slip is induced by a modification of the Riemann problem to the linear elasticity equations across cell interfaces aligned with the subfaults. This is illustrated in the context of the high-resolution wave-propagation algorithms that are implemented in the open source Clawpack software (www.clawpack.org), but this approach could be easily incorporated into other Riemann solver based numerical methods. Surface deformation results are obtained in both two and three dimensions and compared to those given by the steady-state, homogeneous half-space Okada solution.


And the improvement is more evident in the comparison between the third and forth rows, it proves that our midpoint interpolation result provides a better initial position, which benefits the update process under the same number of iterations. P3DConv vs. EdgeConv #REFR .

We employ the midpoint interpolation result P I to serve as both input of P2PNet and initial point cloud to be updated.

For validating the effectiveness of midpoint interpolation, we replace the input of P2PNet with the low-res point cloud P L .

And we also sample points from the Gaussian distribution N (p, σ 2 ) to get another initial point cloud, denoted as P S .

Finally, we combine different network inputs and initial point clouds to conduct the experiments, holding all other parameters the same.

From the second and forth rows of Tab 7, we conclude that using the denser interpolated point cloud P I as network input can achieve better performance, since it contains more geometrical information and benefits the feature extraction process.

The EdgeConv based feature extractor #OTHEREFR is widely used by previous work #OTHEREFR .

However, EdgeConv #OTHEREFR utilizes most of parameters to refine each individual feature.

While our P3DConv focuses on the feature aggregation achieved by generated convolution kernels, thus benefits the extraction of local and global features.

For verifying the superior performance of our P3DConv, we replace the dense block in our P2PNet with EdgeConv in #OTHEREFR , and we also fine-tune the number of EdgeConv layers to achieve the comparable network parameters, as Tab 8 shows. Table 8 .

4× comparative results on the PU1K dataset with different convolution layers.

Grad-Pu: Arbitrary-Scale Point Cloud Upsampling Via Gradient Descent With Learned Distance Functions

Abstract: Most existing point cloud upsampling methods have roughly three steps: feature extraction, feature expansion and 3D coordinate prediction. However, they usually suffer from two critical issues: (1) fixed upsampling rate after one-time training, since the feature expansion unit is customized for each upsampling rate; (2) outliers or shrinkage artifact caused by the difficulty of precisely predicting 3D coordinates or residuals of upsampled points. To adress them, we propose a new framework for accurate point cloud upsampling that supports arbitrary upsampling rates. Our method first interpolates the low-res point cloud according to a given upsampling rate. And then refine the positions of the interpolated points with an iterative optimization process, guided by a trained model estimating the difference between the current point cloud and the high-res target. Extensive quantitative and qualitative results on benchmarks and downstream tasks demonstrate that our method achieves the state-of-the-art accuracy and efficiency.

Patch-Based Progressive 3D Point Set Upsampling

Abstract: We present a detail-driven deep neural network for point set upsampling. A high-resolution point set is essential for point-based rendering and surface reconstruction. Inspired by the recent success of neural image super-resolution techniques, we progressively train a cascade of patch-based upsampling networks on different levels of detail end-to-end. We propose a series of architectural design contributions that lead to a substantial performance boost. The effect of each technical contribution is demonstrated in an ablation study. Qualitative and quantitative experiments show that our method significantly outperforms the state-of-the-art learning-based and optimazation-based approaches, both in terms of handling low-resolution inputs and revealing high-fidelity details.


From this observation, we can see that too small τ will degenerate the Gumbel softmax into binary sampling and make the mask become a binary mask, which could result in no gradients or improper training #REFR .

This is because they do not fully recapture the discriminative cues underlying in non-robust activations.

Our method with Separation leads to the highest robustness, showing that FSR well identifies the non-robust activations and recaptures discriminative cues from them. Hyperparameter Study.

We also compare the robustness as we vary the temperature τ (Eq. 2) that controls how "discrete" the mask is.

For low temperature values, the output mask becomes more discrete (i.e., most values are close to either 0 or 1), and for high temperature values, it becomes more uniform (i.e., most values are far away from 0 or 1) #OTHEREFR . As shown in Fig.

A1a , we achieve the highest robustness when τ = 0.1.

In contrast, too large τ will make the mask become more uniformly distributed and reduce the gap between the mask values applied on robust or non-robust activations, thus making our goal of disentanglement less feasible. In Fig. A1b and Fig.

A1c , we visualize the trends of model robustness as we vary the weights on our proposed loss functions L sep (Eq. 3) and L rec (Eq. 4).

Higher value of λ sep generally improves robustness under all attacks with the best performance achieved when λ sep = 1, showing that our proposed objectives help the model learn more robust feature representations.

Similar trends can also be observed for λ rec ; higher value of λ rec generally improves robustness with the best performance achieved when λ rec = 1.

Setting λ sep and λ rec to be too high, however, tends to degrade robustness.

Feature Separation And Recalibration For Adversarial Robustness

Abstract: Deep neural networks are susceptible to adversarial attacks due to the accumulation of perturbations in the feature level, and numerous works have boosted model robustness by deactivating the non-robust feature activations that cause model mispredictions. However, we claim that these malicious activations still contain discriminative cues and that with recalibration, they can capture additional useful information for correct model predictions. To this end, we propose a novel, easy-to-plugin approach named Feature Separation and Recalibration (FSR) that recalibrates the malicious, non-robust activations for more robust feature maps through Separation and Recalibration. The Separation part disentangles the input feature map into the robust feature with activations that help the model make correct predictions and the non-robust feature with activations that are responsible for model mispredictions upon adversarial attack. The Recalibration part then adjusts the non-robust activations to restore the potentially useful cues for model predictions. Extensive experiments verify the superiority of FSR compared to traditional deactivation techniques and demonstrate that it improves the robustness of existing adversarial training methods by up to 8.57% with small computational overhead. Codes are available at https://github.com/wkim97/FSR.

Spatially Adaptive Inference With Stochastic Feature Sampling And Interpolation

Abstract: In the feature maps of CNNs, there commonly exists considerable spatial redundancy that leads to much repetitive processing. Towards reducing this superfluous computation, we propose to compute features only at sparsely sampled locations, which are probabilistically chosen according to activation responses, and then densely reconstruct the feature map with an efficient interpolation procedure. With this sampling-interpolation scheme, our network avoids expending computation on spatial locations that can be effectively interpolated, while being robust to activation prediction errors through broadly distributed sampling. A technical challenge of this sampling-based approach is that the binary decision variables for representing discrete sampling locations are non-differentiable, making them incompatible with backpropagation. To circumvent this issue, we make use of a reparameterization trick based on the Gumbel-Softmax distribution, with which backpropagation can iterate these variables towards binary values. The presented network is experimentally shown to save substantial computation while maintaining accuracy over a variety of computer vision tasks.


We train the class prediction models for 50 epochs and the domain classifier for 80 epochs with learning rate 0.001, weight decay 0.0001 and batchsize 128 using the Adam optimizer #REFR .

Our base architectures are based on the AdaTime benchmark suite, which is a large-scale evaluation of domain adaptation algorithms on time-series data.

We extended the benchmark suite to support 11 state-of-the-art model architectures on multiple dataset types ranging from language, image to time-series data, addressed by Transformed Moons, Amazon Reviews, MiniDomainNet and the four time-series datasets (UCI-HAR, WISDM, HHAR, and Sleep-EDF) spanning in total 38 cross-domain real-world scenarios.

Transformed Moons For the Transformed Moons dataset we use two sequential blocks with fullyconnected layers, 1D-BatchNorm, ReLU activation functions and Dropout.

The full architecture specification can be found in Table 3 .

The domain classifier (density ratio estimator) uses the same architecture.

We share the same base architecture and training setup across every domain adaption method (e.g., DANN, HoMM, CMD). Additional hyper-parameters are reported in Table 9 . Table 7 and Table 8 for details.

Amazon Reviews For the Amazon Reviews dataset we use two sequential blocks with fullyconnected layers, 1D-BatchNorm, ReLU activation function and Dropout, analogous to the setup for Transformed Moons.

We also use the same architecture for the domain classifier.

We train the class prediction models for 50 epochs and the domain classifier for 80 epochs with learning rate 0.001, weight decay 0.0001 and batchsize 128 using the Adam optimizer #OTHEREFR .

Addressing Parameter Choice Issues In Unsu-Pervised Domain Adaptation By Aggregation

Abstract: We study the problem of choosing algorithm hyper-parameters in unsupervised domain adaptation, i.e., with labeled data in a source domain and unlabeled data in a target domain, drawn from a different input distribution. We follow the strategy to compute several models using different hyper-parameters, and, to subsequently compute a linear aggregation of the models. While several heuristics exist that follow this strategy, methods are still missing that rely on thorough theories for bounding the target error. In this turn, we propose a method that extends weighted least squares to vector-valued functions, e.g., deep neural networks. We show that the target error of the proposed algorithm is asymptotically not worse than twice the error of the unknown optimal aggregation. We also perform a large scale empirical comparative study on several datasets, including text, images, electroencephalogram, body sensor signals and signals from mobile phones. Our method 1 outperforms deep embedded validation (DEV) and importance weighted validation (IWV) on all datasets, setting a new state-of-the-art performance for solving parameter choice issues in unsupervised domain adaptation with theoretical error guarantees. We further study several competitive heuristics, all outperforming IWV and DEV on at least five datasets. However, our method outperforms each heuristic on at least five of seven datasets.

Adam: A Method For Stochastic Optimization

Abstract: We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.


The JCS system also benefits from mutual sharing of sensing information for improved reliability performance, e.g., using the range and Doppler knowledge to assist beamforming and channel prediction #REFR .

By 2030, the 6G MTC system intends to fulfill tremendous IoT connections for intelligent precise control applications to support the future IoT, such as advanced driving automation (from L4 to L5) and accurate vehicle swarm control in the industrial scenarios #OTHEREFR .

These applications aim to conduct customized and precise control missions in mixed sensing/actuation/haptics scenarios, and require sensing accuracy of 0.1 m indoors or 1 m outdoors and communication reliability of 99.9999% for the critical applications #OTHEREFR .

The tremendous connections and high function requirements will result in great spectrum congestion, and demand high spectrum and energy utilization efficiency #OTHEREFR - #OTHEREFR , which can not be fully addressed in 5G cellular MTC scenarios #OTHEREFR .

The state-of-the-art joint communication and sensing (JCS) technique is a promising technique to confront the aforementioned challenges.

The JCS system can share the unified transceiver, the same spectrum and digital signal processing hardware to exploit the reflected echo of communication beam to achieve additional radar sensing function #OTHEREFR , and achieve immediate improvement in both the spectrum and energy efficiency, compared with the conventional communication system where the echo signal is not well used for sensing #OTHEREFR .

Code-Division Ofdm Joint Communication And Sensing System For 6G Machine-Type Communication

Abstract: The joint communication and sensing (JCS) system can provide higher spectrum efficiency and load-saving for 6G machine-type communication (MTC) applications by merging necessary communication and sensing abilities with unified spectrum and transceivers. In order to suppress the mutual interference between the communication and radar sensing signals to improve the communication reliability and radar sensing accuracy, we propose a novel code-division orthogonal frequency division multiplex (CD-OFDM) JCS MTC system, where MTC users can simultaneously and continuously conduct communication and sensing with each other. We propose a novel CD-OFDM JCS signal and corresponding successive-interferencecancellation (SIC) based signal processing technique that obtains code-division multiplex (CDM) gain, which is compatible with the prevalent orthogonal frequency division multiplex (OFDM) communication system. To model the unified JCS signal transmission and reception process, we propose a novel unified JCS channel model. Finally, the simulation and numerical results are shown to verify the feasibility of the CD-OFDM JCS MTC system and the error propagation performance. We show that the CD-OFDM JCS MTC system can achieve not only more reliable communication but also comparably robust radar sensing compared with the precedent OFDM JCS system, especially in low signal-to-interference-and-noise ratio (SINR) regime.

Multibeam For Joint Communication And Radar Sensing Using Steerable Analog Antenna Arrays

Abstract: Beamforming has a great potential for joint communication and radar sensing (JCAS), which is becoming a demanding feature on many emerging platforms, such as unmanned aerial vehicles and smart cars. Although beamforming has been extensively studied for communication and radar sensing respectively, its application in the joint system is not straightforward due to different beamforming requirements by communication and sensing. In this paper, we propose a novel multibeam framework using steerable analog antenna arrays, which allows seamless integration of communication and sensing. Different to conventional JCAS schemes that support JCAS using a single beam, our framework is based on the key innovation of multibeam technology: providing fixed subbeam for communication and packet-varying scanning subbeam for sensing, simultaneously from a single transmitting array. We provide a system architecture and protocols for the proposed framework, complying well with modern packet communication systems with multicarrier modulation. We also propose low-complexity and effective multibeam design and generation methods, which offer great flexibility in meeting different communication and sensing requirements. We further develop sensing parameter estimation algorithms using conventional digital Fourier transform and one-dimensional compressive sensing techniques, matching well with the multibeam framework. Simulation results are provided and validate the effectiveness of our proposed framework, beamforming design methods, and the sensing algorithms.


In the past two decades, sparsity-based processing methods have been attracting a growing interest in statistical signal processing applications #REFR .

Quadratic compressed sensing (QCS) is a widely used formulation in sparse signal recovery;

examples include when imaging a sparse object using partially and spatially incoherent illumination #OTHEREFR , or phase retrieval for sparse signals #OTHEREFR .

To approach the global optimum, the QCS problem was relaxed as a semidefinite programming (SDP) problem, which involves minimizing the rank of a lifted matrix while satisfying both the recovery constraints and the row sparsity constraints on the signal #OTHEREFR .

To retrieve the sparse solution, an iterative thresholding algorithm was proposed that leverages a sequence of SDPs.

This approach is similar to the recent developments in the field of phase retrieval, where similar semidefinite programming-based ideas have been utilized #OTHEREFR .

One-Bit Quadratic Compressed Sensing: From Sample Abundance To Linear Feasibility

Abstract: One-bit quantization with time-varying sampling thresholds has recently found significant utilization potential in statistical signal processing applications due to its relatively low power consumption and low implementation cost. In addition to such advantages, an attractive feature of one-bit analog-to-digital converters (ADCs) is their superior sampling rates as compared to their conventional multi-bit counterparts. This characteristic endows one-bit signal processing frameworks with what we refer to as sample abundance. On the other hand, many signal recovery and optimization problems are formulated as (possibly non-convex) quadratic programs with linear feasibility constraints in the one-bit sampling regime. We demonstrate, with a particular focus on quadratic compressed sensing, that the sample abundance paradigm allows for the transformation of such quadratic problems to merely a linear feasibility problem by forming a large-scale overdetermined linear system; thus removing the need for costly optimization constraints and objectives. To efficiently tackle the emerging overdetermined linear feasibility problem, we further propose an enhanced randomized Kaczmarz algorithm, called Block SKM. Several numerical results are presented to illustrate the effectiveness of the proposed methodologies.

Sparsity Constrained Nonlinear Optimization: Optimality Conditions And Algorithms

Abstract: This paper treats the problem of minimizing a general continuously differentiable function subject to sparsity constraints. We present and analyze several different optimality criteria which are based on the notions of stationarity and coordinate-wise optimality. These conditions are then used to derive three numerical algorithms aimed at finding points satisfying the resulting optimality criteria: the iterative hard thresholding method and the greedy and partial sparse-simplex methods. The first algorithm is essentially a gradient projection method while the remaining two algorithms are of coordinate descent type. The theoretical convergence of these methods and their relations to the derived optimality conditions are studied. The algorithms and results are illustrated by several numerical examples.


Following the default settings in the field of natural language processing task #REFR , here we also randomly mask a certain fraction (e.g.

In fact, most of the previous visual-language modeling approaches focus on the masked language modeling of the texts in multimodal input by reconstructing the mask labeling unit, while neglects the mask feature learning at the pixel level for single image modality.

In this work, we introduce a Masked Multimodal Modeling strategy to perform pretraining based on the MMET model, which can effectively achieve the goal of performing mask loss calculations on image and text modality. More details are depicted in Fig. 3 .

More specifically, we mainly mask the pedestrian region instead of whole part of input images, since the human part contains more discriminative feature for representation learning, and it has no effect on the prediction performance of the model when the masked region mainly focus on the background of input samples.

In this work, masked multimodal modeling is a dynamic strategy which can be applied to pretraining tasks on text encoders for monolingual datasets.

masking ratio: 15%) of image blocks and text labeling units respectively.

Then we adopt the classifier to reconstruct from other image blocks and tokenized units on the hidden state output h M of the image-text modality.

In contrast to denoising auto-encoders #OTHEREFR , we only predict the masked image and words rather than reconstructing the entire text inputs.

During the pretraining, masked image modeling is applied into the image encoder over unpaired image data, while masked language modeling is applied into the text encoder over unpaired text data.

Specifically, given a image and text data, we perform masking on image encoder and text encoder respectively.

Learning Robust Visual-Semantic Embedding For Generalizable Person Re-Identification

Abstract: Generalizable person re-identification (Re-ID) is a very hot research topic in machine learning and computer vision, which plays a significant role in realistic scenarios due to its various applications in public security and video surveillance. However, previous methods mainly focus on the visual representation learning, while neglect to explore the potential of semantic features during training, which easily leads to poor generalization capability when adapted to the new domain. In this paper, we propose a Multi-Modal Equivalent Transformer called MMET for more robust visual-semantic embedding learning on visual, textual and visualtextual tasks respectively. To further enhance the robust feature learning in the context of transformer, a dynamic masking mechanism called Masked Multimodal Modeling strategy (MMM) is introduced to mask both the image patches and the text tokens, which can jointly works on multimodal or unimodal data and significantly boost the performance of generalizable person Re-ID. Extensive experiments on benchmark datasets demonstrate the competitive performance of our method over previous approaches. We hope this method could advance the research towards visual-semantic representation learning.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


We note a similarity to monad-comonad interaction laws #REFR , here adjusted for call-by-push-value.

(return : A) + (enqueue : P × E) + (dequeue : U ((E + 1) ⇒ F P ))

An element of program E (A) is a finite sequence of queue instructions terminated by returning a value of type A.

We may evaluate a program on a queue, by induction on the program:

ψ : − program E (A) × U (queue E ) ⇒ F A

This expresses the usual notion of running a sequence of operations on a data structure; the code is in Listing 6.

Amortized Analysis Via Coinduction

Abstract: Amortized analysis is a program cost analysis technique for data structures in which the cost of operations is specified in aggregate, under the assumption of continued sequential use. Typically, amortized analyses are presented inductively, in terms of finite sequences of operations; we demonstrate that coinduction provides an equivalent but more natural characterization. We describe a classic amortized data structure, the batched queue, and outline a coinductive proof of its amortized efficiency in calf , a type theory for cost analysis.

Interaction Laws Of Monads And Comonads

Abstract: We introduce and study functor-functor and monad-comonad interaction laws as mathematical objects to describe interaction of effectful computations with behaviors of effect-performing machines. Monad-comonad interaction laws are monoid objects of the monoidal category of functor-functor interaction laws. We show that, for suitable generalizations of the concepts of dual and Sweedler dual, the greatest functor resp. monad interacting with a given functor or comonad is its dual while the greatest comonad interacting with a given monad is its Sweedler dual. We relate monad-comonad interaction laws to stateful runners. We show that functor-functor interaction laws are Chu spaces over the category of endofunctors taken with the Day convolution monoidal structure. Hasegawa's glueing endows the category of these Chu spaces with a monoidal structure whose monoid objects are monad-comonad interaction laws.


BN was initially proposed to alleviate internal covariate shift, defined as the shift in the distribution of network activations caused by the change in network parameters during training #REFR .

The latter showed that adding a 2-layers MLPs with a non-linear activation function to the ResNet #OTHEREFR outperformed other choices such as MLPs with a linear activation function or the one without postprocessing for vision tasks #OTHEREFR .

(2) Column-wise Postprocessing -and more specifically, column-wise standardization, in which case each dimension of the raw embeddings is centered and rescaled.

Therefore, each node embedding is processed along with others in a batch, and no longer independently.

This type of processing mechanism is referred to as "batch normalization" (BN) in the deep learning literature #OTHEREFR .

When training deep neural networks (DNNs), BN is typically used before the activation function between layers to standardize inputs.

BN was subsequently found to improve speed and stability of the model's training and generalization accuracy by smoothing the optimization landscape #OTHEREFR and allowing faster learning rates #OTHEREFR .

Here, we propose a new way of using BN as a postprocessing step in self-supervised contrastive learning.

We posit indeed that the row-wise MLPs postprocessing used in usual methods overly complexifies the architecture of the algorithm with no clear gain in performance.

Rather, in Sections 3 and 4, we will show that the column-wise BN postprocessing results in better representations and this result is supported by recent findings in the contrastive learning literature of alignment and uniformity paradigm #OTHEREFR .

We refer to the contrastive learning framework with this choice of postprocessing as Contrastive Learning method for Node Representations (CLNR). The final node embeddings can be written as follows:

A Simplified Framework For Contrastive Learning For Node Representations

Abstract: Contrastive learning has recently established itself as a powerful self-supervised learning framework for extracting rich and versatile data representations. Broadly speaking, contrastive learning relies on a data augmentation scheme to generate two versions of the input data and learns low-dimensional representations by maximizing a normalized temperature-scaled cross entropy loss (NT-Xent) to identify augmented samples corresponding to the same original entity (Chen et al., 2020). In this paper, we investigate the potential of deploying contrastive learning in combination with Graph Neural Networks (Scarselli et al., 2008) for embedding nodes in a graph. Specifically, we show that the quality of the resulting embeddings and training time can be significantly improved by a simple column-wise postprocessing of the embedding matrix, instead of the row-wise postprocessing via multilayer perceptrons (MLPs) that is adopted by the majority of peer methods. This modification yields improvements in downstream classification tasks of up to 1.5% and even beats existing state-of-the-art approaches on 6 out of 8 different benchmarks. We justify our choices of postprocessing by revisiting the "alignment vs. uniformity paradigm" of Wang and Isola (2020), and show that column-wise post-processing improves both "alignment" and "uniformity" of the embeddings.

Batch Normalization: Accelerating Deep Network Training By Reducing Internal Covariate Shift

Abstract: Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.

network activations
internal covariate shift


ShanghaiTech #REFR is a challenging multi-scene dataset containing 13 campus scenes with various light conditions and camera angles.

We evaluate the proposed method on three public datasets: UCSD Ped2 #OTHEREFR , Avenue #OTHEREFR , and Shang-haiTech #OTHEREFR .

UCSD Ped2 #OTHEREFR is a single-scene dataset collected from pedestrian walkways, including anomalies such as bikers, skaters, small carts across a walkway. Avenue #OTHEREFR is a single-scene dataset as well.

It is captured in CUHK campus avenue, containing anomalies like running, bicycling, etc. It also contains some rare normal patterns #OTHEREFR .

The statistics of these datasets are summarized in Table 1 .

However, these three datasets contain very few scenedependent anomalies.

And as far as we know, there is no public scene-dependent anomaly dataset available.

In order to investigate the performance of our method on scenedependent anomaly detection, we additionally create three mixture datasets based on ShanghaiTech.

The mixture set [01, 02] consists of videos taken from scenes 01 and 02.

Hierarchical Semantic Contrast For Scene-Aware Video Anomaly Detection

Abstract: Increasing scene-awareness is a key challenge in video anomaly detection (VAD). In this work, we propose a hierarchical semantic contrast (HSC) method to learn a sceneaware VAD model from normal videos. We first incorporate foreground object and background scene features with highlevel semantics by taking advantage of pre-trained video parsing models. Then, building upon the autoencoderbased reconstruction framework, we introduce both scenelevel and object-level contrastive learning to enforce the encoded latent features to be compact within the same semantic classes while being separable across different classes. This hierarchical semantic contrast strategy helps to deal with the diversity of normal patterns and also increases their discrimination ability. Moreover, for the sake of tackling rare normal activities, we design a skeleton-based motion augmentation to increase samples and refine the model further. Extensive experiments on three public datasets and scene-dependent mixture datasets validate the effectiveness of our proposed method.

Future Frame Prediction For Anomaly Detection - A New Baseline

Abstract: Anomaly detection in videos refers to the identification of events that do not conform to expected behavior. However, almost all existing methods tackle the problem by minimizing the reconstruction errors of training data, which cannot guarantee a larger reconstruction error for an abnormal event. In this paper, we propose to tackle the anomaly detection problem within a video prediction framework. To the best of our knowledge, this is the first work that leverages the difference between a predicted future frame and its ground truth to detect an abnormal event. To predict a future frame with higher quality for normal events, other than the commonly used appearance (spatial) constraints on intensity and gradient, we also introduce a motion (temporal) constraint in video prediction by enforcing the optical flow between predicted frames and ground truth frames to be consistent, and this is the first work that introduces a temporal constraint into the video prediction task. Such spatial and motion constraints facilitate the future frame prediction for normal events, and consequently facilitate to identify those abnormal events that do not conform the expectation. Extensive experiments on both a toy dataset and some publicly available datasets validate the effectiveness of our method in terms of robustness to the uncertainty in normal events and the sensitivity to abnormal events


Thus, overall, the proposed approach RoLNiP outperforms the state-of-the-art LCNiP #REFR on different datasets and various noise rates.

• RoLNiP Versus LCNiP: We observe that with actual as well as estimated noise rates, RoLNiP outperforms LCNiP for Adult, Breast Cancer, Phishing, and W8a datasets for all combinations of noise rates.

For the Ionosphere dataset, RoLNiP outperforms LCNiP for all noise combinations except the (20%,10%) case.

For the (20%,10%) case, the performance of RoLNiP is comparable to LCNiP.

For the CIFAR-10 dataset, RolNiP and LCNiP perform comparably to each other except for the (30%,25%) noise case.

• RoLNiP with Sigmoid Loss, Ramp Loss and Absolute Error Loss: We observe that sigmoid loss, ramp loss and absolute error loss show robust behavior with SD conditional SD label noise.

Rolnip: Robust Learning Using Noisy Pairwise Comparisons

Abstract: This paper presents a robust approach for learning from noisy pairwise comparisons. We propose sufficient conditions on the loss function under which the risk minimization framework becomes robust to noise in the pairwise similar dissimilar data. Our approach does not require the knowledge of noise rate in the uniform noise case. In the case of conditional noise, the proposed method depends on the noise rates. For such cases, we offer a provably correct approach for estimating the noise rates. Thus, we propose an end-to-end approach to learning robust classifiers in this setting. We experimentally show that the proposed approach RoLNiP outperforms the robust state-of-the-art methods for learning with noisy pairwise comparisons.

Learning From Noisy Similar And Dissimilar Data

Abstract: With the widespread use of machine learning for classification, it becomes increasingly important to be able to use weaker kinds of supervision for tasks in which it is hard to obtain standard labeled data. One such kind of supervision is provided pairwise---in the form of Similar (S) pairs (if two examples belong to the same class) and Dissimilar (D) pairs (if two examples belong to different classes). This kind of supervision is realistic in privacy-sensitive domains. Although this problem has been looked at recently, it is unclear how to learn from such supervision under label noise, which is very common when the supervision is crowd-sourced. In this paper, we close this gap and demonstrate how to learn a classifier from noisy S and D labeled data. We perform a detailed investigation of this problem under two realistic noise models and propose two algorithms to learn from noisy S-D data. We also show important connections between learning from such pairwise supervision data and learning from ordinary class-labeled data. Finally, we perform experiments on synthetic and real world datasets and show our noise-informed algorithms outperform noise-blind baselines in learning from noisy pairwise data.


The unlabeled images are sampled from the unpaired data in the EUVP benchmark #REFR , which cover a variety of underwater scenes, water types and lighting conditions.

Our training set contains 1600 labeled image pairs and 1600 unlabeled images.

The labeled image pairs are randomly sampled from #OTHEREFR and UIEB #OTHEREFR with a ratio of 1:1.

#OTHEREFR provides a synthesized underwater image dataset in indoor scene.

The UIEB #OTHEREFR dataset contains 890 real underwater images with corresponding ground truths.

Test set is built with full-reference and non-reference benchmarks.

Full-reference test set includes 110 pairs from #OTHEREFR and 90 pairs from UIEB #OTHEREFR , namely testS and testR.

Non-reference test set includes nearly 700 real world underwater images without ground truths from benchmarks such as UIEB, EUVP, RUIE #OTHEREFR and Seathru #OTHEREFR .

Contrastive Semi-Supervised Learning For Underwater Image Restoration Via Reliable Bank

Abstract: Despite the remarkable achievement of recent underwater image restoration techniques, the lack of labeled data has become a major hurdle for further progress. In this work, we propose a mean-teacher based Semi-supervised Underwater Image Restoration (Semi-UIR) framework to incorporate the unlabeled data into network training. However, the naive mean-teacher method suffers from two main problems: (1) The consistency loss used in training might become ineffective when the teacher's prediction is wrong. (2) Using L1 distance may cause the network to overfit wrong labels, resulting in confirmation bias. To address the above problems, we first introduce a reliable bank to store the "best-ever" outputs as pseudo ground truth. To assess the quality of outputs, we conduct an empirical analysis based on the monotonicity property to select the most trustworthy NR-IQA method. Besides, in view of the confirmation bias problem, we incorporate contrastive regularization to prevent the overfitting on wrong labels. Experimental results on both full-reference and nonreference underwater benchmarks demonstrate that our algorithm has obvious improvement over SOTA methods quantitatively and qualitatively. Code has been released at https://github.com/Huang-ShiRui/Semi-UIR.

Fast Underwater Image Enhancement For Improved Visual Perception

Abstract: In this paper, we present a conditional generative adversarial network-based model for underwater image enhancement in real-time. In order to supervise the adversarial training, we formulate an objective function that evaluates the perceptual image quality based on its global content, color, and local style information. In addition, we present EUVP, a large-scale dataset of a paired and an unpaired collection of underwater images (of poor and good quality) that are captured using seven different cameras over various visibility conditions during oceanic explorations and human-robot collaborative experiments. Furthermore, we perform a number of qualitative and quantitative evaluations which suggest that the proposed model can learn to enhance the quality of underwater images from both paired and unpaired training. More importantly, the enhanced images provide improved performances for several standard models for underwater object detection and human pose estimation; hence, the proposed model can be used as an image processing pipeline by visually-guided underwater robots in real-time applications.