Inference Stabilizer As Implicit Regularizer

Similar results have been obtained by assuming the deep learning algorithms have been driven by an Ornstein-Uhlenbeck like process #REFR , while our work does not rely on such assumption but is all based on our proposed Doubly Stochastic Models.

The regularization effects of unbiased random label noises should be

where ∇ θ f (x, θ) refers to the gradient of f over θ and the effects are controlled by the batch size B and the variance of label noises σ 2 .

Stochastic Gradient Descent With Random Label Noises: Doubly Stochastic Models And Inference Stabilizer

Abstract: Random label noises (or observational noises) widely exist in practical machine learning settings. While previous studies primarily focus on the affects of label noises to the performance of learning, our work intends to investigate the implicit regularization effects of the label noises, under mini-batch sampling settings of stochastic gradient descent (SGD), with assumptions that label noises are unbiased. Specifically, we analyze the learning dynamics of SGD over the quadratic loss with unbiased label noises, where we model the dynamics of SGD as a stochastic differentiable equation (SDE) with two diffusion terms (namely a Doubly Stochastic Model). While the first diffusion term is caused by mini-batch sampling over the (label-noiseless) loss gradients as many other works on SGD [1, 2], our model investigates the second noise term of SGD dynamics, which is caused by mini-batch sampling over the label noises, as an implicit regularizer. Our theoretical analysis finds such implicit regularizer would favor some convergence points that could stabilize model outputs against perturbation of parameters (namely inference stability). Though similar phenomenon have been investigated in [3], our work

Implicit Regularization For Deep Neural Networks Driven By An Ornstein-Uhlenbeck Like Process

Abstract: We consider deep networks, trained via stochastic gradient descent to minimize L2 loss, with the training labels perturbed by independent noise at each iteration. We characterize the behavior of the training dynamics near any parameter vector that achieves zero training error, in terms of an implicit regularization term corresponding to the sum over the data points, of the squared L2 norm of the gradient of the model with respect to the parameter vector, evaluated at each data point. We then leverage this general characterization, which holds for networks of any connectivity, width, depth, and choice of activation function, to show that for 2-layer ReLU networks of arbitrary width and L2 loss, when trained on one-dimensional labeled data $(x_1,y_1),\ldots,(x_n,y_n),$ the only stable solutions with zero training error correspond to functions that: 1) are linear over any set of three or more co-linear training points (i.e. the function has no extra "kinks"); and 2) change convexity the minimum number of times that is necessary to fit the training data. Additionally, for 2-layer networks of arbitrary width, with tanh or logistic activations, we show that when trained on a single $d$-dimensional point $(x,y)$ the only stable solutions correspond to networks where the activations of all hidden units at the datapoint, and all weights from the hidden units to the output, take at most two distinct values, or are zero. In this sense, we show that when trained on "simple" data, models corresponding to stable parameters are also "simple"; in short, despite fitting in an over-parameterized regime where the vast majority of expressible functions are complicated and badly behaved, stable parameters reached by training with noise express nearly the "simplest possible" hypothesis consistent with the data. These results shed light on the mystery of why deep networks generalize so well in practice.


C. Optimal Control With Generic Basis Functions

Instead of the actual basis functions, we use the reduced-rank GP approximation proposed in #REFR to systematically determine basis the basis functions ϕ(x, u) and the parameter V of the prior.

In the following, we show that the proposed optimal control approach can yield good results even if no parametric model is known, which might be the case in practice.

We choose a GP with a squared exponential kernel and Fig. 2 .

Optimal control with generic basis functions: The red area shows the output constraints, the gray area encompasses the 100 scenarios that were used to determine the input trajectory, the green line shows the mean prediction, and the blue line shows one realization of the output of the actual system when the input trajectory u ⋆ 0:H is applied from time t = 0.

select the hyperparameters of the GP and the approximation based on the training data. These parameters are given in Table II .

Afterwards, K = 100 models are sampled using the PG sampler, and the resulting OCP is solved as in the previous example.

As Figure 2 shows, the results are similar to the case with known basis functions.

Learning-Based Optimal Control With Performance Guarantees For Unknown Systems With Latent States

Abstract: As control engineering methods are applied to increasingly complex systems, data-driven approaches for system identification appear as a promising alternative to physicsbased modeling. While many of these approaches rely on the availability of state measurements, the states of a complex system are often not directly measurable. It may then be necessary to jointly estimate the dynamics and a latent state, making it considerably more challenging to design controllers with performance guarantees. This paper proposes a novel method for the computation of an optimal input trajectory for unknown nonlinear systems with latent states. Probabilistic performance guarantees are derived for the resulting input trajectory, and an approach to validate the performance of arbitrary control laws is presented. The effectiveness of the proposed method is demonstrated in a numerical simulation.

Hilbert Space Methods For Reduced-Rank Gaussian Process Regression

Abstract: This paper proposes a novel scheme for reduced-rank Gaussian process regression. The method is based on an approximate series expansion of the covariance function in terms of an eigenfunction expansion of the Laplace operator in a compact subset of $$\mathbb {R}^d$$ ::: . On this approximate eigenbasis, the eigenvalues of the covariance function can be expressed as simple functions of the spectral density of the Gaussian process, which allows the GP inference to be solved under a computational cost scaling as $$\mathcal {O}(nm^2)$$ ::: (initial) and $$\mathcal {O}(m^3)$$ ::: (hyperparameter learning) with m basis functions and n data points. Furthermore, the basis functions are independent of the parameters of the covariance function, which allows for very fast hyperparameter learning. The approach also allows for rigorous error analysis with Hilbert space theory, and we show that the approximation becomes exact when the size of the compact subset and the number of eigenfunctions go to infinity. We also show that the convergence rate of the truncation error is independent of the input dimensionality provided that the differentiability order of the covariance function increases appropriately, and for the squared exponential covariance function it is always bounded by $${\sim }1/m$$ ::: regardless of the input dimensionality. The expansion generalizes to Hilbert spaces with an inner product which is defined as an integral over a specified input density. The method is compared to previously proposed methods theoretically and through empirical tests with simulated and real data.


In contrast, the decoder block incorporates a pixel shuffle layer #REFR between every two stages to increase the feature map's size by a factor of two. Next, we explain each component in more detail.

In order to reconstruct the robot, our model uses an image of the robot to predict the 3D coordinates along its centerline.

Subsequently, a weighted linear least squares algorithm is employed to derive a 3D curve that parametrizes the center of the robot. As shown in Fig.

2 , we design a network with a shared encoder and three decoders that are composed of four stages each.

These stages consist of a residual block #OTHEREFR with two convolutional layers that are connected through BatchNorm #OTHEREFR and Leaky ReLU activations #OTHEREFR .

The encoder block uses a maxpooling layer between every two stages to decrease the feature map's size by a factor of two.

a) Encoder: To incorporate location information, we add the 2D image indices to I RGB .

This results in a 5channel image, represented as I in ∈ R H×W ×5 .

The encoder then extracts multi-scale features from the input image, which are subsequently passed through three decoders for further processing.

b) Decoders: Given that the image includes background, not every pixel is relevant in determining the robot's shape.

To address this, the importance decoder learns the significance of each pixel in shape reconstruction.

Moss: Monocular Shape Sensing For Continuum Robots

Abstract: Continuum robots are promising candidates for interactive tasks in various applications due to their unique shape, compliance, and miniaturization capability. Accurate and real-time shape sensing is essential for such tasks yet remains a challenge. Embedded shape sensing has high hardware complexity and cost, while vision-based methods require stereo setup and struggle to achieve real-time performance. This paper proposes the first eye-to-hand monocular approach to continuum robot shape sensing. Utilizing a deep encoder-decoder network, our method, MoSSNet, eliminates the computation cost of stereo matching and reduces requirements on sensing hardware. In particular, MoSSNet comprises an encoder and three parallel decoders to uncover spatial, length, and contour information from a single RGB image, and then obtains the 3D shape through curve fitting. A two-segment tendon-driven continuum robot is used for data collection and testing, demonstrating accurate (mean shape error of 0.91 mm, or 0.36% of robot length) and real-time (70 fps) shape sensing on real-world data. Additionally, the method is optimized end-to-end and does not require fiducial markers, manual segmentation, or camera calibration. Code and datasets will be made available at https: //github.com/ContinuumRoboticsLab/MoSSNet. * indicates equal contribution.

Real-Time Single Image And Video Super-Resolution Using An Efficient Sub-Pixel Convolutional Neural Network

Abstract: Recently, several models based on deep neural networks have achieved great success in terms of both reconstruction accuracy and computational performance for single image super-resolution. In these methods, the low resolution (LR) input image is upscaled to the high resolution (HR) space using a single filter, commonly bicubic interpolation, before reconstruction. This means that the super-resolution (SR) operation is performed in HR space. We demonstrate that this is sub-optimal and adds computational complexity. In this paper, we present the first convolutional neural network (CNN) capable of real-time SR of 1080p videos on a single K2 GPU. To achieve this, we propose a novel CNN architecture where the feature maps are extracted in the LR space. In addition, we introduce an efficient sub-pixel convolution layer which learns an array of upscaling filters to upscale the final LR feature maps into the HR output. By doing so, we effectively replace the handcrafted bicubic filter in the SR pipeline with more complex upscaling filters specifically trained for each feature map, whilst also reducing the computational complexity of the overall SR operation. We evaluate the proposed approach using images and videos from publicly available datasets and show that it performs significantly better (+0.15dB on Images and +0.39dB on Videos) and is an order of magnitude faster than previous CNN-based methods.


CycleGAN #REFR utilises cycle-consistency loss with a GAN model for bidirectional image-to-image translation.

We compare our proposed framework with various state-of-the-art (SOTA) methods and two selfdesigned baselines.

Among those, pix2pix #OTHEREFR uses a conditional generative model for sketch-to-photo translation.

MUNIT #OTHEREFR aims to produce diverse outputs given one input sketch.

It tries to decompose an image into a content and a style code followed by learning those codes simultaneously.

U-GAT-IT #OTHEREFR uses an attention module for image translation while focusing on the domain-discriminative parts.

Moreover, employing a pre-trained StyleGAN #OTHEREFR we compare with the baseline B-Sketch Mapper which is equivalent to the baseline sketch mapper described in Sec. 5.1.

Following optimisation-based GAN inversion #OTHEREFR , we design B-Sketch Optimiser where we iteratively optimise the latent code using input sketch as a ground-truth with perceptual loss #OTHEREFR .

For a fair comparison, we trained all competing methods in a supervised manner with sketch-photo pairs from ShoeV2, ChairV2, and Handbag datasets.

Picture That Sketch: Photorealistic Image Generation From Abstract Sketches

Abstract: Edgemap Sketch Sketch Sketch Sketch Sketch Existing Methods Proposed Method Figure 1. (a) Set of photos generated by the proposed method. (b) While existing methods can generate faithful photos from perfectly pixel-aligned edgemaps, they fall short drastically in case of highly deformed and sparse free-hand sketches. In contrast, our autoregressive sketch-to-photo generation model produces highly photorealistic outputs from highly abstract sketches.

Unpaired Image-To-Image Translation Using Cycle-Consistent Adversarial Networks

Abstract: Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. We present an approach for learning to translate an image from a source domain $X$ to a target domain $Y$ in the absence of paired examples. Our goal is to learn a mapping $G: X \rightarrow Y$ such that the distribution of images from $G(X)$ is indistinguishable from the distribution $Y$ using an adversarial loss. Because this mapping is highly under-constrained, we couple it with an inverse mapping $F: Y \rightarrow X$ and introduce a cycle consistency loss to push $F(G(X)) \approx X$ (and vice versa). Qualitative results are presented on several tasks where paired training data does not exist, including collection style transfer, object transfiguration, season transfer, photo enhancement, etc. Quantitative comparisons against several prior methods demonstrate the superiority of our approach.


Then the output of the dropout layer is fed to the output layer followed by ReLU activation and gives a vector encoding HEL's (DA, action) output pair. For implementation, we used the Pythorch library #REFR .

The HEL agent network includes two fully connected layers followed by a dropout layer (ratio=0.1).

Multimodal Reinforcement Learning For Robots Collaborating With Humans

Abstract: Robot assistants for older adults and people with disabilities need to interact with their users in collaborative tasks. The core component of these systems is an interaction manager whose job is to observe and assess the task, and infer the state of the human and their intent to choose the best course of action for the robot. Due to the sparseness of the data in this domain, the policy for such multi-modal systems is often crafted by hand; as the complexity of interactions grows this process is not scalable. In this paper, we propose a reinforcement learning (RL) approach to learn the robot policy. In contrast to the dialog systems, our agent is trained with a simulator developed by using human data and can deal with multiple modalities such as language and physical actions. We conducted a human study to evaluate the performance of the system in the interaction with a user. Our designed system shows promising preliminary results when it is used by a real user.

Pytorch: An Imperative Style, High-Performance Deep Learning Library

Abstract: Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it provides an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. ::: In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. ::: We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several common benchmarks.


We use Monte Carlo Dropout for this, conducting multiple passes through the network with dropout enabled at inference time #REFR .

Our task is to aggregate multiple samples collected from the neural network in such a way that we obtain both an acceptable panoptic segmentation result and a corresponding pixel-level uncertainty estimate.

This occurs in three main parts, detailed in Algorithm 4 with Fig. 1 providing a highlevel overview of the process.

The first is collecting and processing the samples from the network.

However, these samples must be at least minimally processed, with many of the detected proposals being nothing more than noise (labeled as "background") #OTHEREFR and requiring removal.

Failing to do so introduces both needless computational overhead at later steps, and introduces a significant degree of noise to any generated segmentation.

This part is straightforward, being an an adaptation of the baseline approach #OTHEREFR and described in Algorithm 1.

For each sample, we first examine each proposal, and verify that the associated class label obtained through the argmax and softmax operators on the logits are not "background".

Then, applying the softmax and argmax operators to the masks yields a proposal map, thus linking each pixel to an instance ID and a classification.

Uncertainty Estimation In Deep Learning For Panoptic Segmentation

Abstract: As deep learning-based computer vision algorithms continue to improve and advance the state of the art, their robustness to real-world data continues to lag their performance on datasets. This makes it difficult to bring an algorithm from the lab to the real world. Ensemble-based uncertainty estimation approaches such as Monte Carlo Dropout have been successfully used in many applications in an attempt to address this robustness issue. Unfortunately, it is not always clear if such ensemble-based approaches can be applied to a new problem domain. This is the case with panoptic segmentation, where the structure of the problem and architectures designed to solve it means that unlike image classification or even semantic segmentation, the typical solution of using a mean across samples cannot be directly applied. In this paper, we demonstrate how ensemblebased uncertainty estimation approaches such as Monte Carlo Dropout can be used in the panoptic segmentation domain with no changes to an existing network, providing both improved performance and more importantly a better measure of uncertainty for predictions made by the network. Results are demonstrated quantitatively and qualitatively on the COCO, KITTI-STEP and VIPER datasets.

Dropout As A Bayesian Approximation: Representing Model Uncertainty In Deep Learning

Abstract: Deep learning tools have gained tremendous attention in applied machine learning. However such tools for regression and classification do not capture model uncertainty. In comparison, Bayesian models offer a mathematically grounded framework to reason about model uncertainty, but usually come with a prohibitive computational cost. In this paper we develop a new theoretical framework casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes. A direct result of this theory gives us tools to model uncertainty with dropout NNs -- extracting information from existing models that has been thrown away so far. This mitigates the problem of representing uncertainty in deep learning without sacrificing either computational complexity or test accuracy. We perform an extensive study of the properties of dropout's uncertainty. Various network architectures and non-linearities are assessed on tasks of regression and classification, using MNIST as an example. We show a considerable improvement in predictive log-likelihood and RMSE compared to existing state-of-the-art methods, and finish by using dropout's uncertainty in deep reinforcement learning.


However, the SQuAD results of MSG significantly outperform the baselines in all cases (Table 3, #REFR , which can be converted to additional advantages in pre-training time.

We fine-tune for 5 epochs for small datasets including CoLA, MRPC, STS-B, and RTE, (we exclude WNLI following most related work #OTHEREFR ), and 3 epochs for other tasks.

For SQuAD, we fine-tune with a batch size of 12 and learning rate of 3e-5 for 2 epochs.

SQuAD metrics are very sensitive to sequence length, and most models work well with only more than 384 sequence length.

Thus, we continue pre-training after the whole schedule with a sequence length of 512 for 100k steps for all the methods compared.

This yields slight drop on speed-up ratios (120% to 100% on Bert-large, disappears if pre-training with a large sequence length from scratch).

2X Faster Language Model Pre-Training Via Masked Structural Growth

Abstract: Acceleration of large language model pre-training is a critical issue in present NLP research. In this paper, we focus on speeding up pre-training by progressively growing from a small Transformer structure to a large one. There are two main research problems related to progressive growth: growth schedule and growth operator. For growth schedule, existing work has explored multi-stage expansion of depth and feedforward layers. However, the impact of each dimension on the schedule's efficiency is still an open question. For growth operator, existing work relies on the initialization of new weights to inherit knowledge, and achieve only non-strict function preservation, limiting further optimization of training dynamics. To address these issues, we propose Masked Structural Growth (MSG), including growth schedules involving all possible dimensions and strictly functionpreserving growth operators that is independent of the initialization of new weights. Experiments show that MSG is significantly faster than related work: we achieve a speed-up of 80% for Bert-base and 120% for Bert-large pre-training. Moreover, MSG is able to improve fine-tuning performances at the same time. 1 * Corresponding author 1 We will release our code for maximum reproducibility. Preprint. Under review.

Net2Net: Accelerating Learning Via Knowledge Transfer

Abstract: We introduce techniques for rapidly transferring the information stored in one neural net into another neural net. The main purpose is to accelerate the training of a significantly larger neural net. During real-world workflows, one often trains very many different neural networks during the experimentation and design process. This is a wasteful process in which each new model is trained from scratch. Our Net2Net technique accelerates the experimentation process by instantaneously transferring the knowledge from a previous network to each new deeper or wider network. Our techniques are based on the concept of function-preserving transformations between neural network specifications. This differs from previous approaches to pre-training that altered the function represented by a neural net when adding layers to it. Using our knowledge transfer mechanism to add depth to Inception modules, we demonstrate a new state of the art accuracy rating on the ImageNet dataset.


In accordance to the classification in #REFR , the above-mentioned algorithms were designed based on general SSI.

This approach integrates Lp-norm regularization factor of filter weight into objection functions of the AFAs directly to quicken convergence process of non-zero elements of underlying systems (where 0, 1 p 

Furthermore, several sparsity-aware LMS-type algorithms and corresponding improved variants have also been designed #OTHEREFR - #OTHEREFR .

Likewise, the L1-norm NSAF (L1-NSAF) and reweighted L1-NSAF (L1-RNSAF) algorithms were presented by employing L1-and reweighted L1-norms respectively to address highly correlated signals #OTHEREFR .

Similarly, through introducing L0-norm penalty factor, the resulting A L0-norm constraint NSAF (L0-NSAF) algorithm realized higher filtering accurateness and faster convergence #OTHEREFR , #OTHEREFR .

However, for block-sparse systems that dominant coefficients are clustered into several groups, in which are generally encountered in satellite-linked and MIMO communications, they may produce evident decrease in learning performance.

With regards to this, some block-sparsity-induced-type AFAs were developed #OTHEREFR - #OTHEREFR .

It is indisputable that AFAs with constant step-size generate conflicting requirements between convergence rate and filtering accuracy.

To eliminate this limitation, based on NSAF, various variable step-size (VSS) variants have been established one after another #OTHEREFR - #OTHEREFR .

The variable step-size matrix NSAF (VSSM-NSAF) was firstly designed by assuming subband noise powers are the same as the posteriori error and acquired excellent estimation accurateness and quicker convergence behavior in comparison with NSAF #OTHEREFR .

Abstract: Limited by fixed step-size and sparsity penalty factor, the conventional sparsity-aware normalized subband adaptive filtering (NSAF) type algorithms suffer from trade-off requirements of high filtering accurateness and quicker convergence behavior. To deal with this problem, this paper proposes variable step-size L0-norm constraint NSAF algorithms (VSS-L0-NSAFs) for sparse system identification. We first analyze mean-square-deviation (MSD) statistics behavior of the L0-NSAF algorithm innovatively in according to a novel recursion form and arrive at corresponding expressions for the cases that background noise variance is available and unavailable, where correlation

Block-Sparsity-Induced Adaptive Filter For Multi-Clustering System Identification

Abstract: In order to improve the performance of least mean square (LMS)-based adaptive filtering for identifying block-sparse systems, a new adaptive algorithm called block-sparse LMS (BS-LMS) is proposed in this paper. The basis of the proposed algorithm is to insert a penalty of block-sparsity, which is a mixed \$l_{2, 0}\$ norm of adaptive tap-weights with equal group partition sizes, into the cost function of traditional LMS algorithm. To describe a block-sparse system response, we first propose a Markov-Gaussian model, which can generate a kind of system responses of arbitrary average sparsity and arbitrary average block length using given parameters. Then we present theoretical expressions of the steady-state misadjustment and transient convergence behavior of BS-LMS with an appropriate group partition size for white Gaussian input data. Based on the above results, we theoretically demonstrate that BS-LMS has much better convergence behavior than \$l_0\$-LMS with the same small level of misadjustment. Finally, numerical experiments verify that all of the theoretical analysis agrees well with simulation results in a large range of parameters.


Therefore, we think the adaptation of CT→MRI is more difficult than MRI→CT, which is consistent with discussions in #REFR . We think it is probably because that MRI Fig. 12 . Visualization of the training process of FVP.

For the experiments of three dataset in Table I, Table II and Table IV with two adaptation directions, the proposed FVP yields the best Dice values in 5 cases over all 6 cases, and the best ASD values in all 6 cases.

We think that the experiments clearly demonstrate that our FVP perform generally better compared with other DA methods.

In Table I , the number of CT samples (30) is larger than the number of MRI samples #OTHEREFR .

While the Dice score by source-only in CT→MRI (0.517) is much lower than that in MRI→CT (0.647).

In Table II , the Dice score by source-only in CT→MRI (0.412) is much lower than that in MRI→CT (0.714).

The first and second rows are visualization of the real and imaginary parts of the prompt in the frequency domain, respectively.

Note that the size of the prompt is 16 × 16.

The third row is the prompt in the spatial domain with a size of 256 × 256.

provides more texture details in organs, compared with CT, which makes training the source segmentation model in CT results in a worse model for adaptation than training in MRI.

FVP does not work well on Dice in MRI→CT adaptation in Table II , probably because MRI→CT is the least discriminating task among all four tasks in Table I and II.

Fvp: Fourier Visual Prompting For Source-Free Unsupervised Domain Adaptation Of Medical Image Segmentation

Abstract: Medical image segmentation methods normally perform poorly when there is a domain shift between training and testing data. Unsupervised Domain Adaptation (UDA) addresses the domain shift problem by training the model using both labeled data from the source domain and unlabeled data from the target domain. Source-Free UDA (SFUDA) was recently proposed for UDA without requiring the source data during the adaptation, due to data privacy or data transmission issues, which normally adapts the pre-trained deep model in the testing stage. However, in real clinical scenarios of medical image segmentation, the trained model is normally frozen in the testing stage. In this paper, we propose Fourier Visual Prompting (FVP) for SFUDA of medical image segmentation. Inspired by prompting learning in natural language processing, FVP steers the frozen pre-trained model to perform well in the target domain by adding a visual prompt to the input target data. In FVP, the visual prompt is parameterized using only a small amount of low-frequency learnable parameters in the input frequency space, and is learned by minimizing the segmentation loss between the predicted segmentation of the prompted target image and reliable pseudo segmentation label of the target image under the frozen model. To our knowledge, FVP is the first work to apply visual prompts to SFUDA for medical image segmentation. The proposed FVP is validated using three public datasets, and experiments demonstrate that FVP yields better segmentation results, compared with various existing methods.

Unsupervised Bidirectional Cross-Modality Adaptation Via Deeply Synergistic Image And Feature Alignment For Medical Image Segmentation

Abstract: Unsupervised domain adaptation has increasingly gained interest in medical image computing, aiming to tackle the performance degradation of deep neural networks when being deployed to unseen data with heterogeneous characteristics. In this work, we present a novel unsupervised domain adaptation framework, named as Synergistic Image and Feature Alignment (SIFA), to effectively adapt a segmentation network to an unlabeled target domain. Our proposed SIFA conducts synergistic alignment of domains from both image and feature perspectives. In particular, we simultaneously transform the appearance of images across domains and enhance domain-invariance of the extracted features by leveraging adversarial learning in multiple aspects and with a deeply supervised mechanism. The feature encoder is shared between both adaptive perspectives to leverage their mutual benefits via end-to-end learning. We have extensively evaluated our method with cardiac substructure segmentation and abdominal multi-organ segmentation for bidirectional cross-modality adaptation between MRI and CT images. Experimental results on two different tasks demonstrate that our SIFA method is effective in improving segmentation performance on unlabeled target images, and outperforms the state-of-the-art domain adaptation approaches by a large margin.


We used the Adam optimizer #REFR with an L2 regularization penalty for performing stochastic gradient descent in all models.

HiSSNet and ProtoNet models were trained using episodic batches, where each batch contains a random subset of 12 sound classes from the taxonomy and 5 recordings for each class.

To sufficiently balance training on both SED and SID classes, we created three different batch configurations: SED only, SED & SID, and SID only.

The configuration for each batch was randomly selected during training, with a weight distribution of 60%/20%/20%, respectively.

Each epoch contained 100 episodic batches, and the model was trained for 1000 epochs.

All HiSSNet models were trained on the full SEID dataset, while SOTA baseline models were trained on SED-specific or SID-specific subsets of the dataset.

For the SED baselines, we implemented a dilated convolutional recurrent neural network (CRNN) #OTHEREFR and a non-hierarchical ProtoNet #OTHEREFR , and for the SID baselines we implemented a non-hierarchical ProtoNet #OTHEREFR .

The SED baselines were trained on the data subset from ESC50, TUT, TAU, FSD50K and BBC, while the SID baselines were trained on the data subset from VCTK and LibriSpeech.

The dilated CRNN was trained using standard batch processing with a batch size of 128.

Hissnet: Sound Event Detection And Speaker Identification Via Hierarchical Prototypical Networks For Low-Resource Headphones

Abstract: Modern noise-cancelling headphones have significantly improved users' auditory experiences by removing unwanted background noise, but they can also block out sounds that matter to users. Machine learning (ML) models for sound event detection (SED) and speaker identification (SID) can enable headphones to selectively pass through important sounds; however, implementing these models for a user-centric experience presents several unique challenges. First, most people spend limited time customizing their headphones, so the sound detection should work reasonably well out of the box. Second, the models should be able to learn over time the specific sounds that are important to users based on their implicit and explicit interactions. Finally, such models should have a small memory footprint to run on low-power headphones with limited on-chip memory. In this paper, we propose addressing these challenges using HiSSNet (Hierarchical SED and SID Network). HiSSNet is an SEID (SED and SID) model that uses a hierarchical prototypical network to detect both general and specific sounds of interest and characterize both alarm-like and speech sounds. We show that HiSSNet outperforms an SEID model trained using non-hierarchical prototypical networks by 6.9-8.6%. When compared to state-of-the-art (SOTA) models trained specifically for SED or SID alone, HiSSNet achieves similar or better performance while reducing the memory footprint required to support multiple capabilities on-device.

Adam: A Method For Stochastic Optimization

Abstract: We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.


First, in a similar fashion to #REFR we report the coverage of the actual safe region and the overlap with the unsafe one.

In this section, we carry out simulations to quantify the performance of IGCI.

We utilize a two dimensional navigation task and a robotic arm environment and consider scenaria of perfect and noisy state observations, as well as suboptimal trajectories.

To evaluate the quality of the inferred constraints, we utilize a number of metrics.

More specifically, we define the constrained and unconstrained regions as

The corresponding regions constructed using the estimated parametersc i , i = 1, . . .

, N c are designated withÃ andÃ c , respectively.

Constraint Inference In Control Tasks From Expert Demonstrations Via Inverse Optimization

Abstract: Inferring unknown constraints is a challenging and crucial problem in many robotics applications. When only expert demonstrations are available, it becomes essential to infer the unknown domain constraints to deploy additional agents effectively. In this work, we propose an approach to infer affine constraints in control tasks after observing expert demonstrations. We formulate the constraint inference problem as an inverse optimization problem, and we propose an alternating optimization scheme that infers the unknown constraints by minimizing a KKT residual objective. We demonstrate the effectiveness of our method in a number of simulations, and show that our method can infer less conservative constraints than a recent baseline method while maintaining comparable safety guarantees.

Learning Constraints From Locally-Optimal Demonstrations Under Cost Function Uncertainty

Abstract: We present an algorithm for learning parametric constraints from locally-optimal demonstrations, where the cost function being optimized is uncertain to the learner. Our method uses the Karush-Kuhn-Tucker (KKT) optimality conditions of the demonstrations within a mixed integer linear program (MILP) to learn constraints which are consistent with the local optimality of the demonstrations, by either using a known constraint parameterization or by incrementally growing a parameterization that is consistent with the demonstrations. We provide theoretical guarantees on the conservativeness of the recovered safe/unsafe sets and analyze the limits of constraint learnability when using locally-optimal demonstrations. We evaluate our method on high-dimensional constraints and systems by learning constraints for 7-DOF arm and quadrotor examples, show that it outperforms competing constraint-learning approaches, and can be effectively used to plan new constraint-satisfying trajectories in the environment.


Table 3 required between 0.49 and 0.8 GPU days to be discovered by FMAS, which is negligible compared to the 2,590 GPU days required by DPC #REFR .

Similar to Table 3 , Table 4 reports results when using the Mo-bileNetV2 backbone and searching for 25 generations.

In addition to FLOPs and parameters, we also report inference latency on the GAP8 for the original model, FCN-VGG16, and selected search results.

Note that while FCN-VGG16 uses only GAP8-supported operations, making it a suitable baseline, it requires more than 8× more RAM than the GAP8 has, and therefore cannot be deployed. Table 1 reports their hyperparameters.

FMAS-F1 cuts the number of FLOPs by 43% with respect to DL3+, and network parameters by 7.9%, for a relative increase of 5.2% in MIoU error; it was discovered in 0.68 GPU days (generation 17).

FMAS-F2 trades off only 2.5% of the MIoU error of DL3+ for reducing FLOPs by 10%, and network parameters by 20%, in 0.52 GPU days (generation 13).

Although DPC outperforms the MIoU of FMAS-F2 by 6.1%, FMAS-F2 cuts FLOPs and parameters by 9 and 22% respectively in only 0.65 GPU days.

Fmas: Fast Multi-Objective Supernet Architecture Search For Semantic Segmentation

Abstract: We present FMAS, a fast multi-objective neural architecture search framework for semantic segmentation. FMAS subsamples the structure and pre-trained parameters of DeepLabV3+, without finetuning, dramatically reducing training time during search. To further reduce candidate evaluation time, we use a subset of the validation dataset during the search. Only the final, Pareto non-dominated, candidates are ultimately fine-tuned using the complete training set. We evaluate FMAS by searching for models that effectively trade accuracy and computational cost on the PASCAL VOC 2012 dataset. FMAS finds competitive designs quickly, e.g., taking just 0.5 GPU days to discover a DeepLabV3+ variant that reduces FLOPs and parameters by 10% and 20% respectively, for less than 3% increased error. We also search on an edge device called GAP8 and use its latency as the metric. FMAS is capable of finding 2.2× faster network with 7.61% MIoU loss.

Searching For Efficient Multi-Scale Architectures For Dense Image Prediction

Abstract: The design of neural network architectures is an important component for achieving state-of-the-art performance with machine learning systems across a broad array of tasks. Much work has endeavored to design and build architectures automatically through clever construction of a search space paired with simple learning algorithms. Recent progress has demonstrated that such meta-learning methods may exceed scalable human-invented architectures on image classification tasks. An open question is the degree to which such methods may generalize to new domains. In this work we explore the construction of meta-learning techniques for dense image prediction focused on the tasks of scene parsing, person-part segmentation, and semantic image segmentation. Constructing viable search spaces in this domain is challenging because of the multi-scale representation of visual information and the necessity to operate on high resolution imagery. Based on a survey of techniques in dense image prediction, we construct a recursive search space and demonstrate that even with efficient random search, we can identify architectures that outperform human-invented architectures and achieve state-of-the-art performance on three dense prediction tasks including 82.7\% on Cityscapes (street scene parsing), 71.3\% on PASCAL-Person-Part (person-part segmentation), and 87.9\% on PASCAL VOC 2012 (semantic image segmentation). Additionally, the resulting architecture is more computationally efficient, requiring half the parameters and half the computational cost as previous state of the art systems.


In this article, we employ the BERT model #REFR as one of the most recent PLM approaches.

The objective of word embedding [7] is to map words to semantic vectors to be used in algorithms of machine learning.

It has been demonstrated to be a reliable method for extracting meaningful word representations based on their context #OTHEREFR .

Diverse word embedding techniques, including Skip-gram #OTHEREFR and matrix factorization techniques such as GloVe #OTHEREFR , have been suggested to produce meaningful word representations for neural network models.

Pre-trained language models (PLMs) typically employ unlabelled data to learn model parameters [3] .

BERT is a The general BERT architecture is shown in Figure 2 .

BERT uses a bi-directional transformer, in which representations are jointly conditioned on both the left and right context in all layers #OTHEREFR .

This distinguishes BERT from Word2Vec and GloVe models which produce an embedding in one direction to ignore its contextual differences.

A Novel Plagiarism Detection Approach Combining Bert-Based Word Embedding, Attention-Based Lstms And An Improved Differential Evolution Algorithm

Abstract: Detecting plagiarism involves finding similar items in two different sources. In this article, we propose a novel method for detecting plagiarism that is based on attention mechanism-based long short-term memory (LSTM) and bidirectional encoder representations from transformers (BERT) word embedding, enhanced with optimized differential evolution (DE) method for pre-training and a focal loss function for training. BERT could be included in a downstream task and fine-tuned as a task-specific BERT can be included in a downstream task and fine-tuned as a task-specific structure, while the trained BERT model is capable of detecting various linguistic characteristics. Unbalanced classification is one of the primary issues with plagiarism detection. We suggest a focal loss-based training technique that carefully learns minority class instances to solve this. Another issue that we tackle is the training phase itself, which typically employs gradient-based methods like back-propagation for the learning process and thus suffers from some drawbacks, including sensitivity to initialization. To initiate the BP process, we suggest a novel DE algorithm that makes use of a clustering-based mutation operator. Here, a winning cluster is identified for the current DE population, and a fresh updating method is used to produce potential answers. We evaluate our proposed approach on three benchmark datasets (MSRP, SNLI, and SemEval2014) and demonstrate that it performs well when compared to both conventional and population-based methods.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


We will now rewrite the formula from #REFR by adding a global disjunction over all values of these numbers g.

In particular, if for a, b ∈ N we use Q a,b to denote the set of all numbers q of granularity b ∈ N such that 0 ≤ q ≤ a, then

• |X 1 | ≤ |X 2 | (for some X 1 , X 2 ⊆ V

Moreover, the same also holds for t 2 (XȲ ) t 1 (X) since we can assume g(t 2 t 1 ) = t 1 (X), irrespective of t 2 (XȲ ) or the instantiation of Y .

The weight comparison g(t 2 t 1 ) t 1 (X) may thus occur outside the quantification of Y .

To avoid any confusion, we remark that this indeed increases the total length of the formula by a factor of O(N ), but the length of each individual subformula forming an atom of this disjunction remains upper-bounded by O(|ϕ|).

S∪{t 2 t 1 }→Q N +1,γ t≤t ∈S t(X) ≤ g(t ≤ t ) ∧ g(t ≤ t ) ≤ t (X) (14) ∧ g(t 2 t 1 ) t 1 (X) ∧ ∀Ȳ χ t 2 t 1 S (XȲ ) ∧ (t 2 (XȲ ) g(t 2 t 1 ) ∨ χ t 2 t 1 S (XȲ ) .

To truly transform #OTHEREFR into a disjunction of table formulas, for g : S ∪{t 2 t 1 } → Q N +1,γ and t ∈ τ 1 , we introduce a set of thresholds that will apply to individual weight terms occurring in #OTHEREFR rather than weight comparisons.

For this we will want to choose the strictest threshold for an individual weight term implied by a comparison involving it, which in the case of t 1 may include a threshold introduced for t 1 t 2 .

Approximate Evaluation Of Quantitative Second Order Queries

Abstract: Courcelle's theorem and its adaptations to cliquewidth have shaped the field of exact parameterized algorithms and are widely considered the archetype of algorithmic meta-theorems. In the past decade, there has been growing interest in developing parameterized approximation algorithms for problems which are not captured by Courcelle's theorem and, in particular, are considered not fixed-parameter tractable under the associated widths. We develop a generalization of Courcelle's theorem that yields efficient approximation schemes for any problem that can be captured by an expanded logic we call ∀ CMSO, capable of making logical statements about the sizes of set variables via so-called weight comparisons. The logic controls weight comparisons via the quantifier-alternation depth of the involved variables, allowing full comparisons for zero-alternation variables and limited comparisons for one-alternation variables. We show that the developed framework threads the very needle of tractability: on one hand it can describe a broad range of approximable problems, while on the other hand we show that the restrictions of our logic cannot be relaxed under well-established complexity assumptions. The running time of our approximation scheme is polynomial in 1/ε, allowing us to fully interpolate between faster approximate algorithms and slower exact algorithms. This provides a unified framework to explain the tractability landscape of graph problems parameterized by treewidth and cliquewidth, as well as classical non-graph problems such as Subset Sum and Knapsack. * acknowledges support by Austrian Science fund (FWF) START project Y1329 † acknowledges support by Austrian Science fund (FWF) project J4651-N 1

Structural Rounding: Approximation Algorithms For Graphs Near An Algorithmically Tractable Class

Abstract: We develop a new framework for generalizing approximation algorithms from the structural graph algorithm literature so that they apply to graphs somewhat close to that class (a scenario we expect is common when working with real-world networks) while still guaranteeing approximation ratios. The idea is to $\textit{edit}$ a given graph via vertex- or edge-deletions to put the graph into an algorithmically tractable class, apply known approximation algorithms for that class, and then $\textit{lift}$ the solution to apply to the original graph. We give a general characterization of when an optimization problem is amenable to this approach, and show that it includes many well-studied graph problems, such as Independent Set, Vertex Cover, Feedback Vertex Set, Minimum Maximal Matching, Chromatic Number, ($\ell$-)Dominating Set, Edge ($\ell$-)Dominating Set, and Connected Dominating Set. To enable this framework, we develop new editing algorithms that find the approximately-fewest edits required to bring a given graph into one of several important graph classes (in some cases, also approximating the target parameter of the family). For bounded degeneracy, we obtain a bicriteria $(4,4)$-approximation which also extends to a smoother bicriteria trade-off. For bounded treewidth, we obtain a bicriteria $(O(\log^{1.5} n), O(\sqrt{\log w}))$-approximation, and for bounded pathwidth, we obtain a bicriteria $(O(\log^{1.5} n), O(\sqrt{\log w} \cdot \log n))$-approximation. For treedepth $2$ (also related to bounded expansion), we obtain a $4$-approximation. We also prove complementary hardness-of-approximation results assuming $\mathrm{P} \neq \mathrm{NP}$: in particular, these problems are all log-factor inapproximable, except the last which is not approximable below some constant factor ($2$ assuming UGC).


C. Model Selection For Different Testing Modes

However, for C8 of #REFR , ViT-B showed better performance than ViT-H (S 2 : 69.3% (B) vs. 64.8% (H)).

Since the everything mode is the bright and key function of SAM, we evaluated it using both two models (everything with ViT-B and ViT-H, called S 1B and S 1H ).

For the prompt mode, we first conducted preexperiments on part of our whole dataset #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR , as shown in Table III .

Specifically, we tested two typical strategies (S 2 and S 5 ) using the DICE and HD metrics.

The experiments showed that the two models had no absolute advantage in different datasets or objects.

For example, in #OTHEREFR , ViT-H outperformed ViT-B on the DICE score for C2 (S 2 : 59.1% (B) vs. 73.2% (H)).

For different structures in the same dataset, e.g., C2, C3, and C4 in #OTHEREFR , there was no winner between ViT-B and ViT-H.

Even for the same structure in different datasets such as the A4 structures in two datasets #OTHEREFR , #OTHEREFR , the performance of these two models remained indistinguishable.

Based on the experiments in Table III, our conclusion is that the larger ViT-H model does not show a significant advantage over the smaller ViT-B model in MIS task under different prompt modes.

In other words, we can assume that the two models should achieve similar average results over a large number of medical images.

Thus, we chose ViT-B as the backbone to test our whole medical dataset to speed up the testing while ensuring that the result reflected SAM's segmentation ability, which is similar to #OTHEREFR .

Segment Anything Model For Medical Images?

Abstract: The Segment Anything Model (SAM) is the first foundation model for general image segmentation. It designed a novel promotable segmentation task, ensuring zero-shot image segmentation using the pre-trained model via two main modes including automatic everything and manual prompt (e.g., points and boxes). SAM has achieved impressive results on various natural image segmentation tasks. However, medical image segmentation (MIS) is more challenging due to the complex modalities, fine anatomical structures, uncertain and complex object boundaries, and wide-range object scales. Meanwhile, zero-shot and efficient MIS can well reduce the annotation time and boost the development of medical image analysis. Hence, SAM seems to be a potential tool and its performance on large medical datasets should be further validated. We collected and sorted 52 opensource datasets, and built a large medical segmentation dataset with 16 modalities, 68 objects, and 553K slice. We conducted a comprehensive analysis of different SAM testing strategies on the so-called COSMOS 553K dataset. Extensive experiments validate that SAM performs better with manual hints like points and boxes for object perception in medical images, leading to better performance in prompt mode compared to everything mode. Additionally, SAM shows remarkable performance in some specific objects and modalities, but is imperfect or even totally fails in other situations. Finally, we analyze the influence of different factors (e.g., the Fourier-based boundary complexity and size of the segmented objects) on SAM's segmentation performance. Extensive experiments validate that SAM's zeroshot segmentation capability is not sufficient to ensure its direct application to the MIS.

A Large Annotated Medical Image Dataset For The Development And Evaluation Of Segmentation Algorithms

Abstract: Semantic segmentation of medical images aims to associate a pixel with a label in a medical image without human initialization. The success of semantic segmentation algorithms is contingent on the availability of high-quality imaging data with corresponding labels provided by experts. We sought to create a large collection of annotated medical image datasets of various clinically relevant anatomies available under open source license to facilitate the development of semantic segmentation algorithms. Such a resource would allow: 1) objective assessment of general-purpose segmentation methods through comprehensive benchmarking and 2) open and free access to medical image data for any researcher interested in the problem domain. Through a multi-institutional effort, we generated a large, curated dataset representative of several highly variable segmentation tasks that was used in a crowd-sourced challenge - the Medical Segmentation Decathlon held during the 2018 Medical Image Computing and Computer Aided Interventions Conference in Granada, Spain. Here, we describe these ten labeled image datasets so that these data may be effectively reused by the research community.


Current graph-based data mining tasks mainly model the relationships between nodes from the perspective of topology and attribute content #REFR , making nodes of the same class more closely embedded in the embedding space and dissimilar nodes further away.

With the help of network data processing platform, operators can mine user Call Detail Records (CDR) to detect fraudsters, thereby assisting mobile network operation decisions.

Data analysis is the most crucial aspect of the whole process, and it is also full of challenges.

Subscribers' communication behaviors naturally constitute graphs, and the use of graph mining techniques for data arXiv:2303.17334v1 [cs.

LG] 29 Mar 2023 analysis has become an important trend.

In recent years, graph neural network (GNN) #OTHEREFR , #OTHEREFR , #OTHEREFR has gradually become the mainstream technology for graph data mining.

A typical semi-supervised node classification task is performed as follows #OTHEREFR : given a large graph with a small scale of node labels, a classifier is trained on those labeled nodes and used to classify other nodes during the testing process.

These related works include graph convolutional networks (GCN) #OTHEREFR and many of its variants proposed in recent years #OTHEREFR , which effectively utilize features in the spectral domain by using simplified first-order approximations.

GraphSage #OTHEREFR and Graph Attention Network(GAT) #OTHEREFR utilize features in the spatial domain to better adapt to different graph topologies.

GNNs have achieved remarkable performance in many application domains, such as text classification #OTHEREFR , image recognition #OTHEREFR , and recommender systems #OTHEREFR .

GNN-based graph data anomaly detection has also made great progress #OTHEREFR .

Gat-Cobo: Cost-Sensitive Graph Neural Network For Telecom Fraud Detection

Abstract: Along with the rapid evolution of mobile communication technologies, such as 5G, there has been a drastically increase in telecom fraud, which significantly dissipates individual fortune and social wealth. In recent years, graph mining techniques are gradually becoming a mainstream solution for detecting telecom fraud. However, the graph imbalance problem, caused by the Pareto principle, brings severe challenges to graph data mining. This is a new and challenging problem, but little previous work has been noticed. In this paper, we propose a Graph ATtention network with COst-sensitive BOosting (GAT-COBO) for the graph imbalance problem. First, we design a GAT-based base classifier to learn the embeddings of all nodes in the graph. Then, we feed the embeddings into a well-designed cost-sensitive learner for imbalanced learning. Next, we update the weights according to the misclassification cost to make the model focus more on the minority class. Finally, we sum the node embeddings obtained by multiple cost-sensitive learners to obtain a comprehensive node representation, which is used for the downstream anomaly detection task. Extensive experiments on two real-world telecom fraud detection datasets demonstrate that our proposed method is effective for the graph imbalance problem, outperforming the state-of-the-art GNNs and GNN-based fraud detectors. In addition, our model is also helpful for solving the widespread over-smoothing problem in GNNs. The GAT-COBO code and datasets are available at https://github.com/xxhu94/GAT-COBO.

Network Representation Learning: A Survey

Abstract: With the widespread use of information technologies, information networks are becoming increasingly popular to capture complex relationships across various disciplines, such as social networks, citation networks, telecommunication networks, and biological networks. Analyzing these networks sheds light on different aspects of social life such as the structure of societies, information diffusion, and communication patterns. In reality, however, the large scale of information networks often makes network analytic tasks computationally expensive or intractable. Network representation learning has been recently proposed as a new learning paradigm to embed network vertices into a low-dimensional vector space, by preserving network topology structure, vertex content, and other side information. This facilitates the original network to be easily handled in the new vector space for further analysis. In this survey, we perform a comprehensive review of the current literature on network representation learning in the data mining and machine learning field. We propose new taxonomies to categorize and summarize the state-of-the-art network representation learning techniques according to the underlying learning mechanisms, the network information intended to preserve, as well as the algorithmic designs and methodologies. We summarize evaluation protocols used for validating network representation learning including published benchmark datasets, evaluation methods, and open source algorithms. We also perform empirical studies to compare the performance of representative algorithms on common datasets, and analyze their computational complexity. Finally, we suggest promising research directions to facilitate future study.


This is consistent with the work in #REFR and allows sampling of healthy tissue patches that can be used as meaningful negative examples.

2) PAIP2019 dataset: The PAIP 2019 dataset #OTHEREFR contains 50 WSIs of liver cancer from 50 patients who underwent resection for hepatocellular carcinoma (HCC) at the Seoul National University Hospital.

The slides were stained by H&E and digitalised with an Aperio AT2 scanner at 20× power and 0.5021µm/px resolution, resulting in image sizes between 35, 855 × 39, 407 and 64, 768 × 47, 009 pixels.

Two types of annotation are provided: viable regions of cancer cells for continuous tumour areas, as well as whole cancer regions for boundary between the non-tumorous hepatic lobules and the viable tumour (including peritumoral fibrosis, capsules, and inflammation).

The initial annotations were provided by a pathologist with 11 years of experience in liver histopathology and reviewed by another expert pathologist.

Additionally, we also generated annotations for "tissue area" which indicates healthy tissue pixels by threshold of (R, G, B) ≤ (235, 210, 235).

For the data pre-processing, we generated context (5× magnification) and target patches (20× magnification) consistent with the settings used in the BCSS dataset.

We randomly selected 10 out of 50 WSIs as the validation set for our CV.

A Dual-Branch Self-Supervised Representation Learning Framework For Tumour Segmentation In Whole Slide Images

Abstract: Supervised deep learning methods have achieved considerable success in medical image analysis, owing to the availability of large-scale and well-annotated datasets. However, creating such datasets for whole slide images (WSIs) in histopathology is a challenging task due to their gigapixel size. In recent years, self-supervised learning (SSL) has emerged as an alternative solution to reduce the annotation overheads in WSIs, as it does not require labels for training. These SSL approaches, however, are not designed for handling multi-resolution WSIs, which limits their performance in learning discriminative image features. In this paper, we propose a Dual-branch SSL Framework for WSI tumour segmentation (DSF-WSI) that can effectively learn image features from multi-resolution WSIs. Our DSF-WSI connected two branches and jointly learnt low and high resolution WSIs in a self-supervised manner. Moreover, we introduced a novel Context-Target Fusion Module (CTFM) and a masked jigsaw pretext task to align the learnt multi-resolution features. Furthermore, we designed a Dense SimSiam Learning (DSL) strategy to maximise the similarity of different views of WSIs, enabling the learnt representations to be more efficient and discriminative. We evaluated our method using two public datasets on breast and liver cancer segmentation tasks. The experiment results demonstrated that our DSF-WSI can effectively extract robust and efficient representations, which we validated through subsequent fine-tuning and semi-supervised settings. Our proposed method achieved better accuracy than other state-of-the-art approaches. Code is available at https://github.com/Dylan-H-Wang/dsf-wsi.

Multi-Scale Fully Convolutional Neural Networks For Histopathology Image Segmentation: From Nuclear Aberrations To The Global Tissue Architecture

Abstract: Histopathologic diagnosis is dependent on simultaneous information from a broad range of scales, ranging from nuclear aberrations ($\approx \mathcal{O}(0.1 \mu m)$) over cellular structures ($\approx \mathcal{O}(10\mu m)$) to the global tissue architecture ($\gtrapprox \mathcal{O}(1 mm)$). Bearing in mind which information is employed by human pathologists, we introduce and examine different strategies for the integration of multiple and widely separate spatial scales into common U-Net-based architectures. Based on this, we present a family of new, end-to-end trainable, multi-scale multi-encoder fully-convolutional neural networks for human modus operandi-inspired computer vision in histopathology.


At runtime, a query is an encoder into a latent space, and the k documents are retrieved using a nearest neighbor algorithm such as HNSW #REFR .

Transformer Based Language Models such as BERT provide contextual language representations built on the Transformer architecture (Vaswani et al., 2017) which can be specialized and adapted for specific tasks and domains .

Using contextual word representations, it becomes relatively easy to excel at a broad range of natural language processing tasks such as Question Answering, Text Classification, and sentiment analysis.

Bi-Encoders, commonly called dual-encoders or dense retrievers, decompose ranking by leveraging the inner product of query and document representations to produce a relevance score for query document pairs.

While not as accurate at cross-encoders #OTHEREFR , they are more efficient for inference and easier to deploy.

Bi-encoder document representations are query invariant, allowing them to be pre-computed and loaded into an Approximate Nearest Neighbor (ANN) such as FAISS #OTHEREFR .

Since the entire document index has been pre-computed, the retrieval latency is limited to a single call of the document encoder.

Bi-encoders commonly leverage LLM such as BERT to retrieve short passages of text leading to the task descriptor of Dense Passage Retrievers (DPR) (Karpukhin et al., 2020) .

Driven by their efficiency in deployment and relevance performance, DPR-based models have rapidly become the building blocks for systems doing product search #OTHEREFR , open domain question answering (Karpukhin et al., 2020) and customer support #OTHEREFR .

Efficient Inference study methods and models which decrease the model execution cost while minimizing the losses to model performance.

Knowledge Distillation #OTHEREFR ) is a training method where a model, called the student, learns to emulate a teacher model, which is commonly larger or better performing than the student.

Quick Dense Retrievers Consume Kale: Post Training Kullback-Leibler Alignment Of Embeddings For Asymmetrical Dual Encoders

Abstract: In this paper, we consider the problem of improving the inference latency of language model-based dense retrieval systems by introducing structural compression and model size asymmetry between the context and query encoders. First, we investigate the impact of pre and post-training compression on the MSMARCO, Natural Questions, TriviaQA, SQUAD, and SCIFACT, finding that asymmetry in the dual-encoders in dense retrieval can lead to improved inference efficiency. Knowing this, we introduce Kullback-Leibler Alignment of Embeddings (KALE), an efficient and accurate method for increasing the inference efficiency of dense retrieval methods by pruning and aligning the query encoder after training. Specifically, KALE extends traditional Knowledge Distillation after bi-encoder training, allowing for effective query encoder compression without full retraining or index generation. Using KALE and asymmetric training, we can generate models which exceed the performance of DistilBERT despite having 3x faster inference.

Efficient And Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs

Abstract: We present a new approach for the approximate K-nearest neighbor search based on navigable small world graphs with controllable hierarchy (Hierarchical NSW, HNSW). The proposed solution is fully graph-based, without any need for additional search structures, which are typically used at the coarse search stage of the most proximity graph techniques. Hierarchical NSW incrementally builds a multi-layer structure consisting from hierarchical set of proximity graphs (layers) for nested subsets of the stored elements. The maximum layer in which an element is present is selected randomly with an exponentially decaying probability distribution. This allows producing graphs similar to the previously studied Navigable Small World (NSW) structures while additionally having the links separated by their characteristic distance scales. Starting search from the upper layer together with utilizing the scale separation boosts the performance compared to NSW and allows a logarithmic complexity scaling. Additional employment of a heuristic for selecting proximity graph neighbors significantly increases performance at high recall and in case of highly clustered data. Performance evaluation has demonstrated that the proposed general metric space search index is able to strongly outperform previous opensource state-of-the-art vector-only approaches. Similarity of the algorithm to the skip list structure allows straightforward balanced distributed implementation.


There, inspired by the Pointer Network #REFR , we directly use the decoder's attention score α j as the final output policy π θ , from which the agent chooses the next node to move to.

In doing so, each encoder, conditioned on the previous one's output, allows each spatio-temporal node feature h ST G j to be informed by the dependencies across all targets and their predictions, history beliefs, and features from all other nodes.

4) Decoder: The decoder is used to yield the final policy based on the spatio-temporal node features.

We first associate each node v j with the shortest distance to the current node ψ t , which is pre-solved by Dijkstra #OTHEREFR . Then, a fully-connected layer maps the concatenated feature

Concat(h ST G j , dist(v j , ψ t )) toh ST G j .

From these node features, we extract the current node feature (features of the current agent's node) as query (i.e., h q =h ST G t ) and its neighboring nodes' features as key-value pairs h k,v for the decoder unit.

This scheme relaxes the requirement of a fixed policy size, instead adapting the policy's dimension dynamically to the number of neighboring nodes.

Together with spatial positional encoding, this endows our network with the ability to generalize to arbitrary graphs and topologies.

Spatio-Temporal Attention Network For Persistent Monitoring Of Multiple Mobile Targets

Abstract: This work focuses on the persistent monitoring problem, where a set of targets moving based on an unknown model must be monitored by an autonomous mobile robot with a limited sensing range. To keep each target's position estimate as accurate as possible, the robot needs to adaptively plan its path to (re-)visit all the targets and update its belief from measurements collected along the way. In doing so, the main challenge is to strike a balance between exploitation, i.e., re-visiting previously-located targets, and exploration, i.e., finding new targets or re-acquiring lost ones. Encouraged by recent advances in deep reinforcement learning, we introduce an attention-based neural solution to the persistent monitoring problem, where the agent can learn the inter-dependencies between targets, i.e., their spatial and temporal correlations, conditioned on past measurements. This endows the agent with the ability to determine which target, time, and location to attend to across multiple scales, which we show also helps relax the usual limitations of a finite target set. We experimentally demonstrate that our method outperforms other baselines in terms of number of targets visits and average estimation error in complex environments. Finally, we implement and validate our model in a drone-based simulation experiment to monitor mobile ground targets in a high-fidelity simulator.

Pointer Networks

Abstract: We introduce a new neural architecture to learn the conditional probability of an output sequence with elements that are discrete tokens corresponding to positions in an input sequence. Such problems cannot be trivially addressed by existent approaches such as sequence-to-sequence [1] and Neural Turing Machines [2], because the number of target classes in each step of the output depends on the length of the input, which is variable. Problems such as sorting variable sized sequences, and various combinatorial optimization problems belong to this class. Our model solves the problem of variable size output dictionaries using a recently proposed mechanism of neural attention. It differs from the previous attention attempts in that, instead of using attention to blend hidden units of an encoder to a context vector at each decoder step, it uses attention as a pointer to select a member of the input sequence as the output. We call this architecture a Pointer Net (Ptr-Net). We show Ptr-Nets can be used to learn approximate solutions to three challenging geometric problems - finding planar convex hulls, computing Delaunay triangulations, and the planar Travelling Salesman Problem - using training examples alone. Ptr-Nets not only improve over sequence-to-sequence with input attention, but also allow us to generalize to variable size output dictionaries. We show that the learnt models generalize beyond the maximum lengths they were trained on. We hope our results on these tasks will encourage a broader exploration of neural learning for discrete problems.


Consider the epigraph formulation in #REFR and define c j = − G g=1 c A g α gj as a shorthand.

As discussed in Section V-B, the largest sum of s co ji is attained if λ co j = 0, in which case s co ji = c j ξ j , ∀i per (14c), i.e., the worst case, and the smallest sum of s co ji is attained if λ co j = c j , in which case s co ji = c j ξ j , ∀i as (14b)-(14d) are equal.

As (14b) is always dominated by (14c) and (14d), write s co ji = c j ξ j +mc j (ξ j − ξ ji ) where we set λ co j = mc j with m = [0, 1].

Using this definition the relevant part of (14a) can be written as mc j j + (N ) −1 N i=1 (c j ξ j − mc j ( ξ ji − ξ j ) = mc j j − (N ) −1 N i=1 ( ξ ji − ξ j ) + c j ξ j .

If the term in rectangular parentheses is positive (case 1 in Proposition 3), m should be minimized leading to m = 0 ⇒ λ co j = 0.

If term (A) is negative (case 2 in Proposition 3), m should be maximized leading to m = 1 ⇒ λ co j = c j .

Data Valuation From Data-Driven Optimization

Abstract: With the ongoing investment in data collection and communication technology in power systems, data-driven optimization has been established as a powerful tool for system operators to handle stochastic system states caused by weatherand behavior-dependent resources. However, most methods are ignorant to data quality, which may differ based on measurement and underlying privacy-protection mechanisms. This paper addresses this shortcoming by (i) proposing a practical data quality metric based on Wasserstein distance, (ii) leveraging a novel modification of distributionally robust optimization using information from multiple data sets with heterogeneous quality to valuate data, (iii) applying the proposed optimization framework to an optimal power flow problem, and (iv) showing a direct method to valuate data from the optimal solution. We conduct numerical experiments to analyze and illustrate the proposed model and publish the implementation open-source.

Constrained Thompson Sampling For Real-Time Electricity Pricing With Grid Reliability Constraints

Abstract: We consider the problem of an aggregator attempting to learn customers' load flexibility models while implementing a load shaping program by means of broadcasting daily dispatch signals. We adopt a multi-armed bandit formulation to account for the stochastic and unknown nature of customers' responses to dispatch signals. We propose a constrained Thompson sampling heuristic, Con-TS-RTP, that accounts for various possible aggregator objectives (e.g., to reduce demand at peak hours, integrate more intermittent renewable generation, track a desired daily load profile, etc) and takes into account the operational constraints of a distribution system to avoid potential grid failures as a result of uncertainty in the customers' response. We provide a discussion on the regret bounds for our algorithm as well as a discussion on the operational reliability of the distribution system's constraints being upheld throughout the learning process.


However, we found that the projection in CMPM can be regarded as a variable weight that adjusts the distribution of softmax output logits, similar to the temperature parameter #REFR for knowledge distillation.

Our main innovation is the design of a multimodal interaction encoder that can efficiently fuse visual and textual representations, align cross-modal fine-grained features through the MLM task.

This design helps the backbone network to extract more discriminative global image-text representations without requiring additional supervision.

To guide the image-text matching, commonly used loss functions include ranking loss and cross-modal projection matching (CMPM) #OTHEREFR loss.

Compared to ranking loss, the CMPM loss does not require the selection of specific triplets or margin parameter tuning.

It exhibits great stability with varying batch sizes, making it widely used in textto-image person retrieval #OTHEREFR .

Nevertheless, limited by the varying projection length, CMPM therefore cannot precisely control the projection probability distribution, making it difficult to focus on hard-negative samples during model updates.

To explore more effective cross-modal matching objective, we further propose an image-text similarity distribution matching (SDM) loss.

The SDM loss minimizes the KL divergence between the normalized image-text similarity score distributions and the normalized ground truth label matching distributions.

Additionally, we introduce a temperature hyperparameter to precisely control the similarity distribution compactness, which enables the model updates focus on hard-negative samples and effectively enlarges the variance between non-matching pairs and the correlation between matching pairs.

To address the limitations of separate pre-trained models on unimodal datasets, we leverage the Contrastive Language-Image Pre-training (CLIP) #OTHEREFR as the initialization of our model.

Cross-Modal Implicit Relation Reasoning And Aligning For Text-To-Image Person Retrieval

Abstract: Text-to-image person retrieval aims to identify the target person based on a given textual description query. The primary challenge is to learn the mapping of visual and textual modalities into a common latent space. Prior works have attempted to address this challenge by leveraging separately pre-trained unimodal models to extract visual and textual features. However, these approaches lack the necessary underlying alignment capabilities required to match multimodal data effectively. Besides, these works use prior information to explore explicit part alignments, which may lead to the distortion of intra-modality information. To alleviate these issues, we present IRRA: a cross-modal Implicit Relation Reasoning and Aligning framework that learns relations between local visual-textual tokens and enhances global image-text matching without requiring additional prior supervision. Specifically, we first design an Implicit Relation Reasoning module in a masked language modeling paradigm. This achieves cross-modal interaction by integrating the visual cues into the textual tokens with a cross-modal multimodal interaction encoder. Secondly, to globally align the visual and textual embeddings, Similarity Distribution Matching is proposed to minimize the KL divergence between image-text similarity distributions and the normalized label matching distributions. The proposed method achieves new state-of-the-art results on all three public datasets, with a notable margin of about 3%-9% for Rank-1 accuracy compared to prior methods.

Distilling The Knowledge In A Neural Network

Abstract: A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.

softmax output logits
knowledge distillation


Also 3D-MPA #REFR reaches a similar performance, confirming the difference between the indoor room and outdoor street scenarios.

The semantic segmentation results are almost the same (except for the weaker PointNet++ backbone).

In terms of instance segmentation, PointGroup without the additional ScoreNet is significantly worse (>8 percent points difference).

With ScoreNet, but without the specialisation to individual rooms, the performance is practically the same as for our pipeline.

The table also shows the result given in the original PointGroup paper, and several other recent instance segmentation works.

When tuned to indoor environments the full PointGroup does have the upper hand.

A Review Of Panoptic Segmentation For Mobile Mapping Point Clouds

Abstract: 3D point cloud panoptic segmentation is the combined task to (i) assign each point to a semantic class and (ii) separate the points in each class into object instances. Recently there has been an increased interest in such comprehensive 3D scene understanding, building on the rapid advances of semantic segmentation due to the advent of deep 3D neural networks. Yet, to date there is very little work about panoptic segmentation of outdoor mobile-mapping data, and no systematic comparisons. The present paper tries to close that gap. It reviews the building blocks needed to assemble a panoptic segmentation pipeline and the related literature. Moreover, a modular pipeline is set up to perform comprehensive, systematic experiments to assess the state of panoptic segmentation in the context of street mapping. As a byproduct, we also provide the first public dataset for that task, by extending the NPM3D dataset to include instance labels.

3D-Mpa: Multi Proposal Aggregation For 3D Semantic Instance Segmentation

Abstract: We present 3D-MPA, a method for instance segmentation on 3D point clouds. Given an input point cloud, we propose an object-centric approach where each point votes for its object center. We sample object proposals from the predicted object centers. Then, we learn proposal features from grouped point features that voted for the same object center. A graph convolutional network introduces inter-proposal relations, providing higher-level feature learning in addition to the lower-level point features. Each proposal comprises a semantic label, a set of associated points over which we define a foreground-background mask, an objectness score and aggregation features. Previous works usually perform non-maximum-suppression (NMS) over proposals to obtain the final object detections or semantic instances. However, NMS can discard potentially correct predictions. Instead, our approach keeps all proposals and groups them together based on the learned aggregation features. We show that grouping proposals improves over NMS and outperforms previous state-of-the-art methods on the tasks of 3D object detection and semantic instance segmentation on the ScanNetV2 benchmark and the S3DIS dataset.


Another way is to introduce the a-priors with the objective to preserve sharp features for surface reconstruction methods as done in DeepPrior #REFR .

Scanned 3D point sets are irregular and non-uniform, and need to be consolidated to enhance the surface reconstruction quality.

One possible solution is to introduce edgeawareness in the consolidation of point sets in a data-driven manner.

The EC-NET #OTHEREFR network processes patches of points and learns to consolidate points using an edge-aware joint loss function when learning from the data.

The performance of the model was demonstrated on a very limited set of 12 manually labelled scans.

In both cases, the presence of high-frequency features and noise in the input scanned data, makes it extremely challenging to recover the sharpness of the scans.

The inference of edges as parametric curves is an alternative that arises directly from CAD surface parametrization as boundary-representation (b-rep).

Following this direction, the PC2WF model [12] infers a wireframe of linear edges from a point cloud based on a vertex localization and an edge detector that identifies the pairs of vertices connected with an edge.

The work [14] proposes a parametric approach to extract a wireframe based on an estimated scalar distance field DEF #OTHEREFR that represents the proximity to the nearest sharp feature curve.

PIE-Net #OTHEREFR proposes to jointly detect edge and corner points, after which a curve proposal module generates an over-complete collection of curves that are further ranked.

Sepicnet: Sharp Edges Recovery By Parametric Inference Of Curves In 3D Shapes

Abstract: 3D scanning as a technique to digitize objects in reality and create their 3D models, is used in many fields and areas. Though the quality of 3D scans depends on the technical characteristics of the 3D scanner, the common drawback is the smoothing of fine details, or the edges of an object. We introduce SepicNet, a novel deep network for the detection and parametrization of sharp edges in 3D shapes as primitive curves. To make the network end-to-end trainable, we formulate the curve fitting in a differentiable manner. We develop an adaptive point cloud sampling technique that captures the sharp features better than uniform sampling. The experiments were conducted on a newly introduced large-scale dataset of 50k 3D scans, where the sharp edge annotations were extracted from their parametric CAD models, and demonstrate significant improvement over state-of-the-art methods.

Deep Geometric Prior For Surface Reconstruction

Abstract: The reconstruction of a discrete surface from a point cloud is a fundamental geometry processing problem that has been studied for decades, with many methods developed. We propose the use of a deep neural network as a geometric prior for surface reconstruction. Specifically, we overfit a neural network representing a local chart parameterization to part of an input point cloud using the Wasserstein distance as a measure of approximation. By jointly fitting many such networks to overlapping parts of the point cloud, while enforcing a consistency condition, we compute a manifold atlas. By sampling this atlas, we can produce a dense reconstruction of the surface approximating the input cloud. The entire procedure does not require any training data or explicit regularization, yet, we show that it is able to perform remarkably well: not introducing typical overfitting artifacts, and approximating sharp features closely at the same time. We experimentally show that this geometric prior produces good results for both man-made objects containing sharp features and smoother organic objects, as well as noisy inputs. We compare our method with a number of well-known reconstruction methods on a standard surface reconstruction benchmark.


Inspired by this, we apply a Squeeze-and-Excitation (SE) module #REFR in the mixing part of PPX.

Previous cross-modal fusion methods show that channel information is crucial #OTHEREFR .

This structure is crucial since some channels of certain modalities do capture more significant information than others.

It can further engage more spatially-holistic knowledge in the channels of the cross-modal complements in SQ-Hub.

Thus, the weighted feature f w is passed to a Feed-Forward Network (FFN) and a SE module #OTHEREFR for enhancing the channel information. The second part of PPX can be written as:

After the PPX block,f w is fused with RGB feature to form the final fused feature f l ∈{f 1 ,f 2 ,f 3 ,f 4 } by using FRM&FFM modules #OTHEREFR , as shown in Fig. 4 .

Delivering Arbitrary-Modal Semantic Segmentation

Abstract: Multimodal fusion can make semantic segmentation more robust. However, fusing an arbitrary number of modalities remains underexplored. To delve into this problem, we create the DELIVER arbitrary-modal segmentation benchmark, covering Depth, LiDAR, multiple Views, Events, and RGB. Aside from this, we provide this dataset in four severe weather conditions as well as five sensor failure cases to exploit modal complementarity and resolve partial outages. To make this possible, we present the arbitrary cross-modal segmentation model CMNEXT. It encompasses a Self-Query Hub (SQ-Hub) designed to extract effective information from any modality for subsequent fusion with the RGB representation and adds only negligible amounts of parameters (∼0.01M) per additional modality. On top, to efficiently and flexibly harvest discriminative cues from the auxiliary modalities, we introduce the simple Parallel Pooling Mixer (PPX). With extensive experiments on a total of six benchmarks, our CMNEXT achieves state-of-the-art performance on the DELIVER, KITTI-360, MFNet, NYU Depth V2, UrbanLF, and MCubeS datasets, allowing to scale from 1 to 81 modalities. On the freshly collected DELIVER, the quad-modal CMNEXT reaches up to 66.30% in mIoU with a +9.10% gain as compared to the mono-modal baseline. 1

Squeeze-And-Excitation Networks

Abstract: Convolutional neural networks are built upon the convolution operation, which extracts informative features by fusing spatial and channel-wise information together within local receptive fields. In order to boost the representational power of a network, much existing work has shown the benefits of enhancing spatial encoding. In this work, we focus on channels and propose a novel architectural unit, which we term the"Squeeze-and-Excitation"(SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channels. We demonstrate that by stacking these blocks together, we can construct SENet architectures that generalise extremely well across challenging datasets. Crucially, we find that SE blocks produce significant performance improvements for existing state-of-the-art deep architectures at slight computational cost. SENets formed the foundation of our ILSVRC 2017 classification submission which won first place and significantly reduced the top-5 error to 2.251%, achieving a 25% relative improvement over the winning entry of 2016.

mixing part
Squeeze-and-Excitation (SE) module


Last, we aim to tune BERT #REFR and its variants on the STS-B task under EXPECTED.

We pre-train a binary classifier on the records with the country of "U.S" and take "non-U.S" records as unobserved target data.

Amazon dataset is constructed from Amazon review data with two categories of products selected, i.e., "Electronics" and "Watches".

In our experiments, the datarich category "Electronics" is used to pre-train a prediction model which maps user comments to the rating score ranging from one to five, and "Watches" is treated as the target data.

The settings of Adult and Amazon follows the work #OTHEREFR .

In terms of CIFAR-10-C/CIFAR-100-C, the initial provided model is built on clean images, and it is then tuned to fit the disjoint corrupted images following the unsupervised tuning research #OTHEREFR , which mimics the unexpected distribution shift in the real world.

Following the research #OTHEREFR , the models are firstly trained on the sentence pairs from the genre of MSRvid and then tuned to fit the unknown target data which are extracted from Images, where the evaluation metric is Pearson's correlation coefficient.

On the task of corrupted image classification, we treat all the corrupted images (target data) as the tuning data for a fair comparison with the unsupervised tuning methods.

Throughout all the remaining datasets, the target data are split into two sets.

We do randomly equal splitting for Adult and Amazon and use the default splitting for STS-B.

One is the support set that is used for evaluating the query efficiency of tuning algorithms, and the other is the holdout set on which the model generalization is assessed.

Earning Extra Performance From Restrictive Feedbacks

Abstract: Many machine learning applications encounter a situation where model providers are required to further refine the previously trained model so as to gratify the specific need of local users. This problem is reduced to the standard model tuning paradigm if the target data is permissibly fed to the model. However, it is rather difficult in a wide range of practical cases where target data is not shared with model providers but commonly some evaluations about the model are accessible. In this paper, we formally set up a challenge named Earning eXtra PerformancE from restriCTive feEDdbacks (EXPECTED) to describe this form of model tuning problems. Concretely, EXPECTED admits a model provider to access the operational performance of the candidate model multiple times via feedback from a local user (or a group of users). The goal of the model provider is to eventually deliver a satisfactory model to the local user(s) by utilizing the feedbacks. Unlike existing model tuning methods where the target data is always ready for calculating model gradients, the model providers in EXPECTED only see some feedbacks which could be as simple as scalars, such as inference accuracy or usage rate. To enable tuning in this restrictive circumstance, we propose to characterize the geometry of the model performance with regard to model parameters through exploring the parameters' distribution. In particular, for the deep models whose parameters distribute across multiple layers, a more query-efficient algorithm is further tailor-designed that conducts layerwise tuning with more attention to those layers which pay off better. Our theoretical analyses justify the proposed algorithms from the aspects of both efficacy and efficiency. Extensive experiments on different applications demonstrate that our work forges a sound solution to the EXPECTED problem, which establishes the foundation for future studies towards this direction.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).