By inferring a pixelwise displacement field #REFR from the distorted image to the distortion-free one with a deep network, a distorted document image can be unwarped with the predicted flow field at a high speed.

Other methods #OTHEREFR - #OTHEREFR capture multiview images of a document as an alternative.

However, the extra hardwares and the demand for multiview images inevitably limit their usage in real applications.

To overcome these limitations, some other methods assume a parametric model on the deformed surface of documents and optimize the model by extracting specific attributes, such as shading #OTHEREFR , #OTHEREFR , boundaries #OTHEREFR , textlines #OTHEREFR - #OTHEREFR , or texture flow #OTHEREFR .

Nevertheless, the oversimplified parametric models usually lead to limited performance, and the optimization process introduces non-negligible computational cost.

Recently, deep learning based solutions #OTHEREFR - #OTHEREFR have been shown as an alternative to the traditional methods, with promising performance and efficiency.

However, most existing algorithms #OTHEREFR - #OTHEREFR are dedicated to processing restricted document images, i.e., the input images must contain complete document boundaries (see Fig. 1(a) ).

Once the distorted images do not include complete document boundaries (see Fig. 1(b, c) ), the rectification quality is degraded.

To overcome this issue, DocProj #OTHEREFR divides the input images into multiple patches and infers per-patch displacement field #OTHEREFR .

However, the stitching of the per-patch displacement field involves heavy computational cost #OTHEREFR and this method is limited to rectifying the document images without any background regions (see Fig. 1(c) ).

For document images with some background regions, we need to manually cut out the document region first before the rectification.

Deep Unrestricted Document Image Rectification

Abstract: In recent years, tremendous efforts have been made on document image rectification, but existing advanced algorithms are limited to processing restricted document images, i.e., the input images must incorporate a complete document. Once the captured image merely involves a local text region, its rectification quality is degraded and unsatisfactory. Our previously proposed DocTr, a transformer-assisted network for document image rectification, also suffers from this limitation. In this work, we present DocTr++, a novel unified framework for document image rectification, without any restrictions on the input distorted images. Our major technical improvements can be concluded in three aspects. Firstly, we upgrade the original architecture by adopting a hierarchical encoder-decoder structure for multi-scale representation extraction and parsing. Secondly, we reformulate the pixel-wise mapping relationship between the unrestricted distorted document images and the distortion-free counterparts. The obtained data is used to train our DocTr++ for unrestricted document image rectification. Thirdly, we contribute a real-world test set and metrics applicable for evaluating the rectification quality. To our best knowledge, this is the first learning-based method for the rectification of unrestricted document images. Extensive experiments are conducted, and the results demonstrate the effectiveness and superiority of our method. We hope our DocTr++ will serve as a strong baseline for generic document image rectification, prompting the further advancement and application of learningbased algorithms. The source code and the proposed dataset are publicly available at https://github.com/fh2019ustc/DocTr-Plus.

Raft: Recurrent All-Pairs Field Transforms For Optical Flow

Abstract: We introduce Recurrent All-Pairs Field Transforms (RAFT), a new deep network architecture for optical flow. RAFT extracts per-pixel features, builds multi-scale 4D correlation volumes for all pairs of pixels, and iteratively updates a flow field through a recurrent unit that performs lookups on the correlation volumes. RAFT achieves state-of-the-art performance, with strong cross-dataset generalization and high efficiency in inference time, training speed, and parameter count. Code is available \url{https://github.com/princeton-vl/RAFT}.


The evaluation was performed by applying the Residual-Unet backbone #REFR and a proposed Transformer-Encoder-based cross-attention method to image pairs from the StreeLearn dataset with large overlaps.

The expressive power of the Transformer-Encoder depends on the number of heads and layers. The more are used, the better the expressive power.

However, using an excessive number might lead to overfit, and the optimal constellation, in terms of accuracy in Table 4 , is given by h = 4, l = 2.

In particular, this constellation is a sweetspot such that increasing the number of heads or layers results in reduced accuracy.

3D rotation encoding and training losses.

The ablations of different rotation encodings and their corresponding training losses were evaluated and are presented in Table 6 .

For the discrete formulation, in line with Cai et al.

#OTHEREFR , the pitch and yaw angles were discretized into 360 bins ∈ [−180 • , 180 • ], and a cross-entropy loss was used to train the network.

These results were compared to those obtained using the L 2 regression loss, as described in Eq. 2 in our scheme.

The results in Table 6 demonstrate that our L 2 regression approach outperforms the discrete Euler angle approach proposed by Cai et al. #OTHEREFR .

Estimating Extreme 3D Image Rotation With Transformer Cross-Attention

Abstract: The estimation of large and extreme image rotation plays a key role in multiple computer vision domains, where the rotated images are related by a limited or a nonoverlapping field of view. Contemporary approaches apply convolutional neural networks to compute a 4D correlation volume to estimate the relative rotation between image pairs. In this work, we propose a cross-attentionbased approach that utilizes CNN feature maps and a Transformer-Encoder, to compute the cross-attention between the activation maps of the image pairs, which is shown to be an improved equivalent of the 4D correlation volume, used in previous works. In the suggested approach, higher attention scores are associated with image regions that encode visual cues of rotation. Our approach is end-to-end trainable and optimizes a simple regression loss. It is experimentally shown to outperform contemporary state-of-the-art schemes when applied to commonly used image rotation datasets and benchmarks, and establishes a new state-of-the-art accuracy on these datasets. We make our code publicly available 1 .

Road Extraction By Deep Residual U-Net

Abstract: Road extraction from aerial images has been a hot research topic in the field of remote sensing image analysis. In this letter, a semantic segmentation neural network, which combines the strengths of residual learning and U-Net, is proposed for road area extraction. The network is built with residual units and has similar architecture to that of U-Net. The benefits of this model are twofold: first, residual units ease training of deep networks. Second, the rich skip connections within the network could facilitate information propagation, allowing us to design networks with fewer parameters, however, better performance. We test our network on a public road data set and compare it with U-Net and other two state-of-the-art deep-learning-based road extraction methods. The proposed approach outperforms all the comparing methods, which demonstrates its superiority over recently developed state of the arts.


To reduce the number of parameters, we set a decay factor R, which means to reduce the number of convolution kernels in the first FC layer to #REFR R .

The last part is mainly composed of SENet #OTHEREFR , Feed Forward module.

Unlike self-attention, SENet weights the channel dimension of speech features to obtain the interdependence between its modeling channels. Fig.

6 shows its structure; it mainly includes Squeeze and Excitation.

The Squeeze operation is a global average pooling, after the compression operation, the feature map is compressed into 1 × C, where C is the number of channels, and it has a global receptive field.

Excitation is similar to the gate mechanism in RNN, utilizing the weight matrix W ∈ R 1×C to regenerate the weights for each feature channel.

Let the input of SENet be x, its formula is: 6 . SENet.

The Global Pooling is applied to condense the input into a 1D vector, which is then processed by a bottleneck structure formed by two fully connected (FC) layers with swish activation functions.

The output goes through a sigmoid function to be mapped to (0, 1), and then applied to the input vector using pointwise multiplications.

Pyramid Multi-Branch Fusion Dcnn With Multi-Head Self-Attention For Mandarin Speech Recognition

Abstract: As one of the major branches of automatic speech recognition, attention-based models greatly improves the feature representation ability of the model. In particular, the multi-head mechanism is employed in the attention, hoping to learn speech features of more aspects in different attention subspaces. For speech recognition of complex languages, on the one hand, a small head size will lead to an obvious shortage of learnable aspects. On the other hand, we need to reduce the dimension of each subspace to keep the size of the overall feature space unchanged when we increase the number of heads, which will significantly weaken the ability to represent the feature of each subspace. Therefore, this paper explores how to use a small attention subspace to represent complete speech features while ensuring many heads. In this work we propose a novel neural network architecture, namely, pyramid multi-branch fusion DCNN with multi-head self-attention. The proposed architecture is inspired by Dilated Convolution Neural Networks (DCNN), it uses multiple branches with DCNN to extract the feature of the input speech under different receptive fields. To reduce the number of parameters, every two branches are merged until all the branches are merged into one. Thus, its shape is like a pyramid rotated 90 degrees. We demonstrate that on Aishell-1, a widely used Mandarin speech dataset, our model achieves a character error rate (CER) of 6.45% on the test sets.

Deep Speech: Scaling Up End-To-End Speech Recognition

Abstract: We present a state-of-the-art speech recognition system developed using end-to-end deep learning. Our architecture is significantly simpler than traditional speech systems, which rely on laboriously engineered processing pipelines; these traditional systems also tend to perform poorly when used in noisy environments. In contrast, our system does not need hand-designed components to model background noise, reverberation, or speaker variation, but instead directly learns a function that is robust to such effects. We do not need a phoneme dictionary, nor even the concept of a "phoneme." Key to our approach is a well-optimized RNN training system that uses multiple GPUs, as well as a set of novel data synthesis techniques that allow us to efficiently obtain a large amount of varied data for training. Our system, called Deep Speech, outperforms previously published results on the widely studied Switchboard Hub5'00, achieving 16.0% error on the full test set. Deep Speech also handles challenging noisy environments better than widely used, state-of-the-art commercial speech systems.


This objective is accomplished by estimating word similarity through Word2Vec #REFR to convert words into vectors.

This data preview also functions as an indicator to detect any deviations in the subsequent generation process.

Furthermore, we extract the data annotation encompassing the x and y axes and the title into an SVG format file, making it available for subsequent user editing. Semantic context.

For the semantic context extraction, we employ a two-step approach, namely keyword extraction and relevant word retrieval.

Initially, we extract the keyword using MPNet #OTHEREFR , a pretrained model featuring a sentence transformer architecture.

However, since the keyword may not be sufficiently explicit to inspire a concrete visual element, we also provide relevant words to stimulate users' creativity, particularly for individuals with limited design expertise, in accordance with Fig. 2 .

To accomplish this, we employ the English Wikipedia Dump from November 2021, comprising a corpus of 199,430 words, each represented by a 300-dimensional vector.

In our experiments, we discovered that some top similar words are more closely associated with proper nouns, which could decrease the recognition capacity in visualization.

With the assumption that the occurrence frequency of the word in the corpus is related to its usage in everyday life, we integrate frequency and similarity as two criteria to rank the relevant semantic context, yielding the top 7 words as a result ( Fig. 5 (A 3 ) ).

Let The Chart Spark: Embedding Semantic Context Into Chart With Text-To-Image Generative Model

Abstract: Pictorial visualization seamlessly integrates data and semantic context into visual representation, conveying complex information in a manner that is both engaging and informative. Extensive studies have been devoted to developing authoring tools to simplify the creation of pictorial visualizations. However, mainstream works mostly follow a retrieving-and-editing pipeline that heavily relies on retrieved visual elements from a dedicated corpus, which often compromise the data integrity. Text-guided generation methods are emerging, but may have limited applicability due to its predefined recognized entities. In this work, we propose ChartSpark, a novel system that embeds semantic context into chart based on text-to-image generative model. ChartSpark generates pictorial visualizations conditioned on both semantic context conveyed in textual inputs and data information embedded in plain charts. The method is generic for both foreground and background pictorial generation, satisfying the design practices identified from an empirical research into existing pictorial visualizations. We further develop an interactive visual interface that integrates a text analyzer, editing module, and evaluation module to enable users to generate, modify, and assess pictorial visualizations. We experimentally demonstrate the usability of our tool, and conclude with a discussion of the potential of using text-to-image generative model combined with interactive interface for visualization design.

Efficient Estimation Of Word Representations In Vector Space

Abstract: We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities.


Another pattern of trigger #REFR uses a designed perturbation that can be directly added to the training data.

In the domain adaptation task, model providers randomly select a portion of the source data to poison and specify a targeted category label y b .

Blended #OTHEREFR uses a fixed image as a trigger whose size is the same as the training data x s i .

The backdoor source data x b,s i is calculated through a mixup manner as follows,

where I tri denotes the designed trigger and α controls the mix ratio.

This perturbation needs to have a bounded amplitude and the backdoor source data must be clamped to the meaningful interval as,

where ∆ tri represents the trigger perturbation and clamp denotes the clamp operation to make sure all pixel values are in the range between 0 and 255.

The providers replace the selected set with its poison version

Adaptguard: Defending Against Universal Attacks For Model Adaptation

Abstract: Model adaptation aims at solving the domain transfer problem under the constraint of only accessing the pretrained source models. With the increasing considerations of data privacy and transmission efficiency, this paradigm has been gaining recent popularity. This paper studies the vulnerability to universal attacks transferred from the source domain during model adaptation algorithms due to the existence of the malicious providers. We explore both universal adversarial perturbations and backdoor attacks as loopholes on the source side and discover that they still survive in the target models after adaptation. To address this issue, we propose a model preprocessing framework, named AdaptGuard, to improve the security of model adaptation algorithms. AdaptGuard avoids direct use of the risky source parameters through knowledge distillation and utilizes the pseudo adversarial samples under adjusted radius to enhance the robustness. AdaptGuard is a plug-and-play module that requires neither robust pretrained models nor any changes for the following model adaptation algorithms. Extensive results on three commonly used datasets and two popular adaptation methods validate that AdaptGuard can effectively defend against universal attacks and maintain clean accuracy in the target domain simultaneously. We hope this research will shed light on the safety and robustness of transfer learning.

A New Backdoor Attack In Cnns By Training Set Corruption Without Label Poisoning

Abstract: Backdoor attacks against CNNs represent a new threat against deep learning systems, due to the possibility of corrupting the training set so to induce an incorrect behaviour at test time. To avoid that the trainer recognises the presence of the corrupted samples, the corruption of the training set must be as stealthy as possible. Previous works have focused on the stealthiness of the perturbation injected into the training samples, however they all assume that the labels of the corrupted samples are also poisoned. This greatly reduces the stealthiness of the attack, since samples whose content does not agree with the label can be identified by visual inspection of the training set or by running a pre-classification step. In this paper we present a new backdoor attack without label poisoning Since the attack works by corrupting only samples of the target class, it has the additional advantage that it does not need to identify beforehand the class of the samples to be attacked at test time. Results obtained on the MNIST digits recognition task and the traffic signs classification task show that backdoor attacks without label poisoning are indeed possible, thus raising a new alarm regarding the use of deep learning in security-critical applications.


In these DNN life cycle models, development and operational stages are linked in a loop #REFR , aiming to assess and improve the accuracy the DNN according to the operating conditions.

Metamorphic testing #OTHEREFR and mutation testing #OTHEREFR , #OTHEREFR represent the most common strategies to evaluate the robustness of the DNN and to forecast the reliability of these systems in the operational environment.

However, the accuracy estimated before release can substantially diverge from the one obtained during operation (operational accuracy). Retch et al.

demonstrated how the accuracy scores of classifiers can significantly drop when completely new data are submitted #OTHEREFR .

This problem grows up when unexpected phenomena occur in operation, such as distribution shift or label shift #OTHEREFR .

Iterative life cycles specific for DNN -such as MLOps #OTHEREFR , #OTHEREFR -have been envisaged by companies like Google.

In particular, they may exploit operational data for remodeling and/or retraining the DNN before the new deployment (experimental stage), and for both the automatic evaluation of the accuracy and the automatic re-training of the models on the field (deployment stage).

The true label of operational data collected by monitoring the DNN is generally unknown.

This is a general issue in software testing, known as the oracle problem #OTHEREFR .

For ML systems, according to Murphy et al., "there is no reliable test oracle to indicate what the correct output should be for arbitrary input" #OTHEREFR .

The problem clearly affects also DNN accuracy estimation #OTHEREFR .

Iterative Assessment And Improvement Of Dnn Operational Accuracy

Abstract: Deep Neural Networks (DNN) are nowadays largely adopted in many application domains thanks to their human-like, or even superhuman, performance in specific tasks. However, due to unpredictable/unconsidered operating conditions, unexpected failures show up on field, making the performance of a DNN in operation very different from the one estimated prior to release. In the life cycle of DNN systems, the assessment of accuracy is typically addressed in two ways: offline, via sampling of operational inputs, or online, via pseudo-oracles. The former is considered more expensive due to the need for manual labeling of the sampled inputs. The latter is automatic but less accurate. We believe that emerging iterative industrial-strength life cycle models for Machine Learning systems, like MLOps, offer the possibility to leverage inputs observed in operation not only to provide faithful estimates of a DNN accuracy, but also to improve it through remodeling/retraining actions. We propose DAIC (DNN Assessment and Improvement Cycle), an approach which combines "low-cost" online pseudo-oracles and "high-cost" offline sampling techniques to estimate and improve the operational accuracy of a DNN in the iterations of its life cycle. Preliminary results show the benefits of combining the two approaches and integrating them in the DNN life cycle.

Assuring The Machine Learning Lifecycle: Desiderata, Methods, And Challenges

Abstract: Machine learning has evolved into an enabling technology for a wide range of highly successful applications. The potential for this success to continue and accelerate has placed machine learning (ML) at the top of research, economic and political agendas. Such unprecedented interest is fuelled by a vision of ML applicability extending to healthcare, transportation, defence and other domains of great societal importance. Achieving this vision requires the use of ML in safety-critical applications that demand levels of assurance beyond those needed for current ML applications. Our paper provides a comprehensive survey of the state-of-the-art in the assurance of ML, i.e. in the generation of evidence that ML is sufficiently safe for its intended use. The survey covers the methods capable of providing such evidence at different stages of the machine learning lifecycle, i.e. of the complex, iterative process that starts with the collection of the data used to train an ML component for a system, and ends with the deployment of that component within the system. The paper begins with a systematic presentation of the ML lifecycle and its stages. We then define assurance desiderata for each stage, review existing methods that contribute to achieving these desiderata, and identify open challenges that require further research.


Indeed, the best-known sub-packetization of MSR array code construction is exp(O(n)) while the sub-packetization of (high-rate) scalar MSR codes needs to be exp(Θ(n log(n)), according to a lower bound in #REFR .

In particular, Guruswami and Wootters viewed each codeword coordinate as a vector over a subfield of RS codes' symbol field and proposed repair schemes with smaller bandwidth than the naive repair scheme. The degree of field extension is called sub-packetization.

Building on the framework in #OTHEREFR , Tamo et al.

#OTHEREFR proposed an RS code construction with optimal repair bandwidth, which is a scalar MSR code.

In contrast, for an (n, k, ℓ) MSR array code, each codeword coordinate is a vector in F ℓ q for some field F q , where the parameter ℓ is sub-packetization.

Compared to scalar MSR codes, their array code counterparts allow more flexibility in the code construction, which results in a smaller sub-packetization.

For this reason, we only consider MSR array codes in this paper.

Msr Codes With Linear Field Size And Smallest Sub-Packetization For Any Number Of Helper Nodes

Abstract: An (n, k, ℓ) array code has k information coordinates and r = n − k parity coordinates, where each coordinate is a vector in F ℓ q for some field F q. An (n, k, ℓ) MDS array code has the additional property that any k out of n coordinates suffice to recover the whole codeword. Dimakis et al. considered the problem of repairing the erasure of a single coordinate and proved a lower bound on the amount of data transmission that is needed for the repair. A minimum storage regenerating (MSR) array code with repair degree d is an MDS array code that achieves this lower bound for the repair of any single erased coordinate from any d out of n − 1 remaining coordinates. An MSR code has the optimal access property if the amount of accessed data is the same as the amount of transmitted data in the repair procedure. The sub-packetization ℓ and the field size q are of paramount importance in the MSR array code constructions. For optimal-access MSR codes, Balaji et al. proved that ℓ ≥ s ⌈n/s⌉ , where s = d − k + 1. Rawat et al. showed that this lower bound is attainable for all admissible values of d when the field size is exponential in n. After that, tremendous efforts have been devoted to reducing the field size. However, till now, reduction to linear field size is only available for d ∈ {k + 1, k + 2, k + 3} and d = n − 1. In this paper, we construct the first class of explicit optimal-access MSR codes with the smallest sub-packetization ℓ = s ⌈n/s⌉ for all d between k + 1 and n−1, resolving an open problem in the survey (Ramkumar et al., Foundations and Trends in Communications and Information Theory: Vol. 19: No. 4). We further propose another class of explicit MSR code constructions (not optimal-access) with even smaller sub-packetization s ⌈n/(s+1)⌉ for all admissible values of d, making significant progress on another open problem in the survey. Previously, MSR codes with ℓ = s ⌈n/(s+1)⌉ and q = O(n) were only known for d = k + 1 and d = n − 1. The key insight that enables a linear field size in our construction is to reduce n r global constraints of non-vanishing determinants to O s (n) local ones, which is achieved by carefully designing the parity check matrices.

Optimal Repair Of Reed-Solomon Codes: Achieving The Cut-Set Bound

Abstract: Coding for distributed storage gives rise to a new set of problems in coding theory related to the need of reducing inter-node communication in the system. A large number of recent papers addressed the problem of optimizing the total amount of information downloaded for repair of a single failed node (the repair bandwidth) by accessing information on $d$ {\em helper nodes}, where $k\le d\le n-1.$ By the so-called cut-set bound (Dimakis et al., 2010), the repair bandwidth of an $(n,k=n-r)$ MDS code using $d$ helper nodes is at least $dl/(d+1-k),$ where $l$ is the size of the node. Also, a number of known constructions of MDS array codes meet this bound with equality. In a related but separate line of work, Guruswami and Wootters (2016) studied repair of Reed-Solomon (RS) codes, showing that these codes can be repaired using a smaller bandwidth than under the trivial approach. At the same time, their work as well as follow-up papers stopped short of constructing RS codes (or any scalar MDS codes) that meet the cut-set bound with equality, which has been an open problem in coding theory. In this work we present a solution to this problem, constructing RS codes of length $n$ over the field $q^l, l=\exp((1+o(1))n\log n)$ that meet the cut-set bound. We also prove an almost matching lower bound on $l$, showing that the super-exponential scaling is both necessary and sufficient for achieving the cut-set bound using linear repair schemes. More precisely, we prove that for scalar MDS codes (including the RS codes) to meet this bound, the sub-packetization $l$ must satisfy $l \ge \exp((1+o(1)) k\log k).$


Motivated by a possibility to cover by algebraic constructions more parameters than the parameters of linear codes, the concept of additive codes was suggested in #REFR .

However, the sphere-packing necessary condition on the existence of 1-perfect codes requires only the divisibility of the number 16 m • 4 n of the vertices of the graph by the cardinality 1 + 6m + 3n of a radius-1 ball.

The family of parameters satisfying this condition is much wider than the ones meeting #OTHEREFR .

For example, all 10 Doob graphs of diameter 21, D #OTHEREFR 19) , D(2, 17), . . .

, D(10, 1), have radius-1 balls of size 64, dividing 4 21 , while only one of them, D #OTHEREFR , falls into restrictions #OTHEREFR .

As was shown in #OTHEREFR , the sphere-packing divisibility condition is sufficient for the existence of unrestricted 1-perfect codes in a Doob graph, where "unrestricted" means that the code is not implied to have any algebraic structure.

A Doob graph D(m, n) inherits the structure of the module

where n = n ′ + n ′′ , and a code is additive if it is additively closed, or, equivalently, forms a submodule of this module. Necessary conditions

2m + n ′ + n ′′ = 2 Γ+2∆ − 1 3 , 3n ′ + n ′′ = 2 Γ+∆ − 1, 0 ≤ n ′′ ≤ 2 ∆ − 1, n ′′ = 1, Γ ≥ 0, ∆ ≥ 2 (2)

were given and first examples constructed in #OTHEREFR , while constructions developed in #OTHEREFR cover the whole three-parameter family of Doob graphs D(m, n ′ + n ′′ ) satisfying #OTHEREFR .

Quasi-Cyclic Perfect Codes In Doob Graphs And Special Partitions Of Galois Rings

Abstract: The Galois ring GR(4 ∆) is the residue ring Z4[x]/(h(x)), where h(x) is a basic primitive polynomial of degree ∆ over Z4. For any odd ∆ larger than 1, we construct a partition of GR(4 ∆)\{0} into 6-subsets of type {a, b, −a−b, −a, −b, a+b} and 3-subsets of type {c, −c, 2c} such that the partition is invariant under the multiplication by a nonzero element of the Teichmuller set in GR(4 ∆) and, if ∆ is not a multiple of 3, under the action of the automorphism group of GR(4 ∆). As a corollary, this implies the existence of quasi-cyclic additive 1-perfect codes of index (2 ∆ − 1) in D((2 ∆ − 1)(2 ∆ − 2)/6, 2 ∆ − 1) where D(m, n) is the Doob metric scheme on Z 2m+n .

Perfect Codes In Doob Graphs

Abstract: We study $$1$$1-perfect codes in Doob graphs $$D(m,n)$$D(m,n). We show that such codes that are linear over the Galois ring $$\mathrm {GR}(4^2)$$GR(42) exist if and only if there exist integers $$\gamma \ge 0$$źź0 and $$\delta >0$$ź>0 such that $$n=(4^{\gamma +\delta }-1)/3$$n=(4ź+ź-1)/3 and $$m=(4^{\gamma +2\delta }-4^{\gamma +\delta })/6$$m=(4ź+2ź-4ź+ź)/6. We also prove necessary conditions on $$(m,n)$$(m,n) for $$1$$1-perfect codes that are linear over $$Z_4$$Z4 (we call such codes additive) to exist in $$D(m,n)$$D(m,n) graphs; for some of these parameters, we show the existence of codes. For every $$m$$m and $$n$$n satisfying $$2m+n = (4^\mu - 1)/3$$2m+n=(4μ-1)/3 and $$m\le (4^\mu -5\cdot 2^{\mu -1}+1)/9$$m≤(4μ-5·2μ-1+1)/9, we construct $$1$$1-perfect codes in $$D(m,n)$$D(m,n), which do not necessarily have a group structure.


Though AC is not broadly tied to understanding people within organizations, examining fairness through this lens is highly illustrative of some of the difficulties and inherent trade-offs (cf. #REFR ) in fairness considerations within AC.

Fairness has no universal definition as it is a social, not psychometric, concept rooted in value judgments #OTHEREFR .

Fairness is a subjective evaluation (e.g., justice, morality), varying across cultures and societies, and in the context of organizations such as schools, hospitals, or corporations, organizational justice, has been the predominant theoretical concept used to recognize perceptions of unfairness #OTHEREFR .

Organizational justice involves three key dimensions: distributive fairness, interactional fairness, and procedural fairness #OTHEREFR .

Distributive fairness regards the perceived fairness of outcomes and allocations of important resources (e.g., jobs).

Interactional fairness regards how people perceive the explanations, rationales, and justifications for organizational decisions and how they perceive the interpersonal treatment they receive along the way.

Procedural fairness regards the perceived fairness of the elements of the decision-making process.

Procedural fairness is emphasized in the Standards because it is crucial that an assessment (e.g., ML predictions) does not generate different scores between subgroups if they have equivalent true scores.

Integrating Psychometrics And Computing Perspectives On Bias And Fairness In Affective Computing: A Case Study Of Automated Video Interviews

Abstract: We provide a psychometric-grounded exposition of bias and fairness as applied to a typical machine learning pipeline for affective computing. We expand on an interpersonal communication framework to elucidate how to identify sources of bias that may arise in the process of inferring human emotions and other psychological constructs from observed behavior. Various methods and metrics for measuring fairness and bias are discussed along with pertinent implications within the United States legal context. We illustrate how to measure some types of bias and fairness in a case study involving automatic personality and hireability inference from multimodal data collected in video interviews for mock job applications. We encourage affective computing researchers and practitioners to encapsulate bias and fairness in their research processes and products and to consider their role, agency, and responsibility in promoting equitable and just systems.

Inherent Trade-Offs In The Fair Determination Of Risk Scores

Abstract: Recent discussion in the public sphere about algorithmic classification has involved tension between competing notions of what it means for a probabilistic classification to be fair to different groups. We formalize three fairness conditions that lie at the heart of these debates, and we prove that except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously. Moreover, even satisfying all three conditions approximately requires that the data lie in an approximate version of one of the constrained special cases identified by our theorem. These results suggest some of the ways in which key notions of fairness are incompatible with each other, and hence provide a framework for thinking about the trade-offs between them.


Other datasets, such as Continuous Object Recognition (CORe50) #REFR , specifically designed for LL, still make the same assumptions to generate tasks.

The result is that the benchmark panorama of CL in computer vision is quite fragmented, and therefore it has become tougher to measure catastrophic forgetting and domain adaptation in a fair and homogeneous way for the many CL models we have in the literature these days.

Furthermore, all previous evaluation protocols are designed to highlight some specific model characteristics and, as such, are generally over-simplified w.r.t. real-world data #OTHEREFR .

For example, one of the most popular evaluation protocol for CL models in computer vision is to design different tasks to classify different (subsets of the) classes of a single dataset #OTHEREFR .

The most prominent example is splitMNIST in which the 10 digits from MNIST #OTHEREFR are (usually) divided into 5 tasks consisting of 2 digits each.

Similar approaches are proposed for CIFAR10 [20] , and TinyImagenet #OTHEREFR .

Clearly, these protocols are not suited to detect distribution shifts, due to the high inter-task similarity.

Consequently, catastrophic forgetting is much easier to prevent in these cases.

Therefore the reported metrics for models evaluated in this way can be overly optimistic. Table 1 Benchmarks comparison considering only multi-dataset benchmarks.

Columns refer to: i) supporting multiple heterogeneous tasks; ii) varying task complexity and quality; iii) evaluating curriculum strategies; iv) rigorous way to measure generalization and forgetting, and v) exactly reproducible out-of-the-box.

In addition, we consider the coverage of class (CI) and task-incremental (TI) learning settings.

From Mnist To Imagenet And Back: Benchmarking Continual Curriculum Learning

Abstract: Continual learning (CL) is one of the most promising trends in recent machine learning research. Its goal is to go beyond classical assumptions in machine learning and develop models and learning strategies that present high robustness in dynamic environments. This goal is realized by designing strategies that simultaneously foster the incorporation of new knowledge while avoiding forgetting past knowledge. The landscape of CL research is fragmented into several learning evaluation protocols, comprising different learning tasks, datasets, and evaluation metrics. Additionally, the benchmarks adopted so far are still distant from the complexity of real-world scenarios, and are usually tailored to highlight capabilities specific to certain strategies. In such a landscape, it is hard to clearly and objectively assess models and strategies. In this work, we fill this gap for CL on image data by introducing two novel CL benchmarks that involve multiple heterogeneous tasks from six image datasets, with varying levels of complexity and quality. Our aim is to fairly evaluate current state-of-the-art CL strategies on a common ground that is closer to complex real-world scenarios. We additionally structure our benchmarks so that tasks are presented in increasing and decreasing order of complexityaccording to a curriculum-in order to evaluate if current CL models are able to exploit structure across tasks. We devote particular emphasis to providing the CL community with a rigorous and reproducible evaluation protocol for measuring the ability of a model to generalize and not to forget while learning. Furthermore, we provide an extensive experimental evaluation showing that popular CL strategies,

Core50: A New Dataset And Benchmark For Continuous Object Recognition

Abstract: Continuous/Lifelong learning of high-dimensional data streams is a challenging research problem. In fact, fully retraining models each time new data become available is infeasible, due to computational and storage issues, while naive incremental strategies have been shown to suffer from catastrophic forgetting. In the context of real-world object recognition applications (e.g., robotic vision), where continuous learning is crucial, very few datasets and benchmarks are available to evaluate and compare emerging techniques. In this work we propose a new dataset and benchmark CORe50, specifically designed for continuous object recognition, and introduce baseline approaches for different continuous learning scenarios.


In Figure 5 and 9 (more samples only for our method), we compare the generated images for LSUN 2 #REFR bedroom dataset for 32 × 32 images; we find that the generated images are close to state-of-the-art.

Since for CIFAR10, a standard metric to compare is inception score, in subfigure 6c in Figure 6 , we plot the inception scores for these methods, we find that our method achieves high inception score much earlier than both ACGD and ConOpt for the same GAN architecture as mentioned above.

For reference, we have also plotted discriminator and generator losses in Figure 6a and 6b respectively.

However, we must mention here that the first order methods such as ADAM never achieved comparable inception scores as those of second order methods; this observation was also found in previous works such as #OTHEREFR , #OTHEREFR , where other first order methods were also compared.

Hence, existing literature and our comparison suggest that qualitative improvements are seen with second order methods.

Also, as shown before, our method is fastest to train and requires less memory among second order methods.

Lastly, in Figure 7c and 8 (more samples only for our method), we show images generated from FFHQ dataset #OTHEREFR ; this dataset offers a significant variety in terms of ethnicity, age, viewpoint, image background, and lighting

t ← t + 1 3: D x ← ∇ x f (x t , y t ) 4: D 2 xx ∆x ← ∇ x (D x )(x t − x t−1 )

{Update second order derivative} #OTHEREFR :

xx ∆x + D x {Update first order derivative} #OTHEREFR :

Adaptive Consensus Optimization Method For Gans

Abstract: We propose a second order gradient based method with ADAM and RMSprop for the training of generative adversarial networks. The proposed method is fastest to obtain similar accuracy when compared to prominent second order methods. Unlike state-of-the-art recent methods, it does not require solving a linear system, or it does not require additional mixed second derivative terms. We derive the fixed point iteration corresponding to proposed method, and show that the proposed method is convergent. The proposed method produces better or comparable inception scores, and comparable quality of images compared to other recently proposed state-of-the-art second order methods. Compared to first order methods such as ADAM, it produces significantly better inception scores. The proposed method is compared and validated on popular datasets such as FFHQ, LSUN, CIFAR10, MNIST, and Fashion MNIST for image generation tasks 1 .

Lsun: Construction Of A Large-Scale Image Dataset Using Deep Learning With Humans In The Loop

Abstract: While there has been remarkable progress in the performance of visual recognition algorithms, the state-of-the-art models tend to be exceptionally data-hungry. Large labeled training datasets, expensive and tedious to produce, are required to optimize millions of parameters in deep network models. Lagging behind the growth in model capacity, the available datasets are quickly becoming outdated in terms of size and density. To circumvent this bottleneck, we propose to amplify human effort through a partially automated labeling scheme, leveraging deep learning with humans in the loop. Starting from a large set of candidate images for each category, we iteratively sample a subset, ask people to label them, classify the others with a trained model, split the set into positives, negatives, and unlabeled based on the classification confidence, and then iterate with the unlabeled set. To assess the effectiveness of this cascading procedure and enable further progress in visual recognition research, we construct a new image dataset, LSUN. It contains around one million labeled images for each of 10 scene categories and 20 object categories. We experiment with training popular convolutional networks and find that they achieve substantial performance gains when trained on this dataset.


This is demonstrated by the fact that we achieve a PSNR gain of 7.98dB and a SSIM improvement of 0.07 compared to Cycle-GAN #REFR .

Star-Net achieves a PSNR gain of 4.12dB and an SSIM gain of 0.05 compared to Tran-sWeather #OTHEREFR .

In the SRRS dataset, Restormer #OTHEREFR shows a PSNR of 32.24dB and SSIM of 0.96.

Star-Net shows a PSNR of 32.41dB and SSIM of 0.97.

As shown in Table 6 , Star-Net performs best on the quantitative evaluation of the Snow100K dataset.

Specially, we score a PSNR of 34.79dB and a SSIM of 0.96.

Compared to All in One #OTHEREFR , Star-Net achieves a PSNR improvement of 8.72dB and SSIM gain of 0.08.

Furthermore, Star-Net achieves a PSNR gain of 0.28dB and an SSIM improvement of 0.01 compared to MPRNet #OTHEREFR .

Figure 15 shows the results of the qualitative comparison of Star-Net with several algorithms on CSD dataset.

Although these methods achieve considerable snow removal on the CSD dataset, they still have some shortcomings.

DesnowNet #OTHEREFR and JSTASR #OTHEREFR suffer from snow particle residual issue, and HDCW-Net #OTHEREFR suffers from image distortion.

Star-Net: Improving Single Image Desnowing Model With More Efficient Connection And Diverse Feature Interaction

Abstract: Compared to other severe weather image restoration tasks, single image desnowing is a more challenging task. This is mainly due to the diversity and irregularity of snow shape, which makes it extremely difficult to restore images in snowy scenes. Moreover, snow particles also have a veiling effect similar to haze or mist. Although current works can effectively remove snow particles with various shapes, they also bring distortion to the restored image. To address these issues, we propose a novel single image desnowing network called Star-Net. First, we design a Star type Skip Connection (SSC) to establish information channels for all different scale features, which can deal with the complex shape of snow particles. Second, we present a Multi-Stage Interactive Transformer (MIT) as the base module of Star-Net, which is designed to better understand snow particle shapes and to address image distortion by explicitly modeling a variety of important image recovery features. Finally, we propose a Degenerate Filter Module (DFM) to filter the snow particle and snow fog residual in the SSC on the spatial and channel domains. Extensive experiments show that our Star-Net achieves state-of-the-art snow removal performances on three standard snow removal datasets and retains the original sharpness of the images.

Unpaired Image-To-Image Translation Using Cycle-Consistent Adversarial Networks

Abstract: Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. We present an approach for learning to translate an image from a source domain $X$ to a target domain $Y$ in the absence of paired examples. Our goal is to learn a mapping $G: X \rightarrow Y$ such that the distribution of images from $G(X)$ is indistinguishable from the distribution $Y$ using an adversarial loss. Because this mapping is highly under-constrained, we couple it with an inverse mapping $F: Y \rightarrow X$ and introduce a cycle consistency loss to push $F(G(X)) \approx X$ (and vice versa). Qualitative results are presented on several tasks where paired training data does not exist, including collection style transfer, object transfiguration, season transfer, photo enhancement, etc. Quantitative comparisons against several prior methods demonstrate the superiority of our approach.


The network is trained with an initial learning rate of 2 × 10 −2 and cyclic learning rate scheduling #REFR .

Experimental setting 1) Training environment: We used PyTorch implementation of ESPNetv2 #OTHEREFR with modifications of adding an auxiliary classification branch, and normalizing the features and classification weights.

The models were trained and evaluated on one NVIDIA Quadro RTX 8000 with 48GB RAM.

2) Datasets: As source datasets, we use CamVid #OTHEREFR , Cityscapes #OTHEREFR , and Freiburg Forest #OTHEREFR .

As target datasets, we use Greenhouse A #OTHEREFR , TUT Park, a dataset of images around a in-campus park, and Toyohashi Trail, images in unstructured mountain paths in Toyohashi Nature Trail.

The datasets used in the experiments are summarized in Table I.

3) Baselines: In our comparative studies, we employ baseline methods as follows.

Training without DA As baselines without DA, we use two methods, namely supervised and ensemble.

Multi-Source Soft Pseudo-Label Learning With Domain Similarity-Based Weighting For Semantic Segmentation

Abstract: This paper describes a method of domain adaptive training for semantic segmentation using multiple source datasets that are not necessarily relevant to the target dataset. We propose a soft pseudo-label generation method by integrating predicted object probabilities from multiple source models. The prediction of each source model is weighted based on the estimated domain similarity between the source and the target datasets to emphasize contribution of a model trained on a source that is more similar to the target and generate reasonable pseudo-labels. We also propose a training method using the soft pseudo-labels considering their entropy to fully exploit information from the source datasets while suppressing the influence of possibly misclassified pixels. The experiments show comparative or better performance than our previous work and another existing multi-source domain adaptation method, and applicability to a variety of target environments.

Cyclical Learning Rates For Training Neural Networks

Abstract: It is known that the learning rate is the most important hyper-parameter to tune for training deep neural networks. This paper describes a new method for setting the learning rate, named cyclical learning rates, which practically eliminates the need to experimentally find the best values and schedule for the global learning rates. Instead of monotonically decreasing the learning rate, this method lets the learning rate cyclically vary between reasonable boundary values. Training with cyclical learning rates instead of fixed values achieves improved classification accuracy without a need to tune and often in fewer iterations. This paper also describes a simple way to estimate "reasonable bounds" -- linearly increasing the learning rate of the network for a few epochs. In addition, cyclical learning rates are demonstrated on the CIFAR-10 and CIFAR-100 datasets with ResNets, Stochastic Depth networks, and DenseNets, and the ImageNet dataset with the AlexNet and GoogLeNet architectures. These are practical tools for everyone who trains neural networks.


Our use of such simple detection methods contrasts with well-studied error detection methods in NLP #REFR Fomicheva et al., 2020) as a sign that tools are reasonable model-agnostic extensions even with simple detection heuristics.

Sequence log-probability (SeqLogProb) is the length-normalised sequence log-probability from a trained model F (

We expect low-confidence answers are likely to be incorrect.

Input length, as we have seen in Section 3, poses a challenge to all models irrespective of size and training objective.

We leverage the size of the table measured by the number of rows as a simple feature to decide when to apply a tool.

Generate, Transform, Answer: Question Specific Tool Synthesis For Tabular Data

Abstract: Tabular question answering (TQA) presents a challenging setting for neural systems by requiring joint reasoning of natural language with large amounts of semi-structured data. Unlike humans who use programmatic tools like filters to transform data before processing, language models in TQA process tables directly, resulting in information loss as table size increases. In this paper we propose Tool-Writer to generate query specific programs and detect when to apply them to transform tables and align them with the TQA model's capabilities. Focusing ToolWriter to generate rowfiltering tools improves the state-of-the-art for WikiTableQuestions and WikiSQL with the most performance gained on long tables. By investigating headroom, our work highlights the broader potential for programmatic tools combined with neural components to manipulate large amounts of structured data. Question: How many monarchs died before the age of 35? Deceased James I aged 68 Alfonso I aged 27 Sancho aged 28 James II aged 69 295 rows x 5 columns TQA Model James III aged 34 Alfonso I aged 27 Sancho aged 28 >>> table.apply(lambda row: float(row['Deceased'] .split('aged')[1]) < 35

Naver Labs Europe'S Systems For The Wmt19 Machine Translation Robustness Task

Abstract: This paper describes the systems that we submitted to the WMT19 Machine Translation robustness task. This task aims to improve MT's robustness to noise found on social media, like informal language, spelling mistakes and other orthographic variations. The organizers provide parallel data extracted from a social media website in two language pairs: French-English and Japanese-English (in both translation directions). The goal is to obtain the best scores on unseen test sets from the same source, according to automatic metrics (BLEU) and human evaluation. We proposed one single and one ensemble system for each translation direction. Our ensemble models ranked first in all language pairs, according to BLEU evaluation. We discuss the pre-processing choices that we made, and present our solutions for robustness to noise and domain adaptation.

NLP Fomicheva
well-studied error detection


While it is acknowledged that the targeted attack is a more common protocol in attacking the ImageNet dataset #REFR , however, we believe that our significant improvement on the untargeted attack is still worth noting.

We report experimental results with different settings in Table 19 , where RE/LE/-TAR/UNTAR represents Robust Ensemble/Less-Robust Ensemble/Targeted Attack/Untargeted Attack, respectively.

It can be seen that, in the robust ensemble setting (RE-TAR and RE-UNTAR), our MOO achieves a similar performance compared to the baseline, while TA-MOO has a further improvement over MOO.

The gap of SAR-All between TA-MOO and the uniform weighting strategy is 0.1% in the targeted attack setting (RE-TAR), while that in the untargeted attack setting is 1.2%.

In the less-robust ensemble setting (LE-TAR and LE-UNTAR), the improvement of our methods over the baseline is higher than in the robust ensemble setting.

With the gap of SAR-All between TA-MOO and the uniform strategy is 0.38% with the targeted attack setting (LE-TAR), while the gap in the untargeted setting (LE-UNTAR) is 15.22% a significantly higher.

We conduct an additional experiment on the EoT setting with the ImageNet dataset and report result in Table 20 .

In this experiment, we use the robust pretrained ResNet18 model (model ID: Salman2020Do_R18) as the victim model.

We use the standard attack setting, i.e., targeted attack with = 4/255, η = 1/255 with 20 steps.

It can be seen that both MOO and TA-MOO could obtain a better attack performance than the uniform strategy.

It is a worth noting that, in the experiment on the CIFAR10/CIFAR100 datasets (i.e., Table 6 in the main paper) the dominating issue of the vertical filliping exists and prevents MOO to obtain a better performance.

Generating Adversarial Examples With Task Oriented Multi-Objective Optimization

Abstract: Deep learning models, even the-state-of-the-art ones, are highly vulnerable to adversarial examples. Adversarial training is one of the most efficient methods to improve the model's robustness. The key factor for the success of adversarial training is the capability to generate qualified and divergent adversarial examples which satisfy some objectives/goals (e.g., finding adversarial examples that maximize the model losses for simultaneously attacking multiple models). Therefore, multi-objective optimization (MOO) is a natural tool for adversarial example generation to achieve multiple objectives/goals simultaneously. However, we observe that a naive application of MOO tends to maximize all objectives/goals equally, without caring if an objective/goal has been achieved yet. This leads to useless effort to further improve the goal-achieved tasks, while putting less focus on the goal-unachieved tasks. In this paper, we propose Task Oriented MOO to address this issue, in the context where we can explicitly define the goal achievement for a task. Our principle is to only maintain the goal-achieved tasks, while letting the optimizer spend more effort on improving the goal-unachieved tasks. We conduct comprehensive experiments for our Task Oriented MOO on various adversarial example generation schemes. The experimental results firmly demonstrate the merit of our proposed approach. Our code is available at https://github.com/tuananhbui89/TAMOO.

Obfuscated Gradients Give A False Sense Of Security: Circumventing Defenses To Adversarial Examples

Abstract: We identify obfuscated gradients as a phenomenon that leads to a false sense of security in defenses against adversarial examples. While defenses that cause obfuscated gradients appear to defeat optimization-based attacks, we find defenses relying on this effect can be circumvented. ::: For each of the three types of obfuscated gradients we discover, we describe indicators of defenses exhibiting this effect and develop attack techniques to overcome it. In a case study, examining all defenses accepted to ICLR 2018, we find obfuscated gradients are a common occurrence, with 7 of 8 defenses relying on obfuscated gradients. Using our new attack techniques, we successfully circumvent all 7 of them.


The importance of the temperature parameters for the continuous relaxation of discrete distributions is consistent with the results of #REFR , revealing that the success of the relaxation depends on the choice of the temperature for the different latent space sizes.

Concerning the continuous relaxation of discrete distributions, we used a different temperature for each distribution, as #OTHEREFR

(2017) pointed out that doing so is valuable to improve the results. We further followed the guidelines of #OTHEREFR

(2017) to choose the interval of temperatures and did not schedule any annealing scheme (in contrast to VAE-MDPs).

Essentially, the search reveals that the regularizer scale factors β¨(defining the optimization direction) as well as the encoder and latent transition temperatures are important to improve the performance of distilled policies.

For the encoder temperature, we found a nice spot in λ φι " 2 {3, which provides the best performance in general, whereas the choice of λ P θ and β¨are (latent-) environment dependent.

We used the same labeling functions as those described by #OTHEREFR .

For completeness, we recall the labeling function used for each environment in Table 4 .

Let θ 1 , θ 2 P r0, 2πs be the angles of the two rotational joints,

Wasserstein Auto-Encoded Mdps Formal Verification Of Efficiently Distilled Rl Policies With Many-Sided Guarantees

Abstract: Although deep reinforcement learning (DRL) has many success stories, the largescale deployment of policies learned through these advanced techniques in safetycritical scenarios is hindered by their lack of formal guarantees. Variational Markov Decision Processes (VAE-MDPs) are discrete latent space models that provide a reliable framework for distilling formally verifiable controllers from any RL policy. While the related guarantees address relevant practical aspects such as the satisfaction of performance and safety properties, the VAE approach suffers from several learning flaws (posterior collapse, slow learning speed, poor dynamics estimates), primarily due to the absence of abstraction and representation guarantees to support latent optimization. We introduce the Wasserstein auto-encoded MDP (WAE-MDP), a latent space model that fixes those issues by minimizing a penalized form of the optimal transport between the behaviors of the agent executing the original policy and the distilled policy, for which the formal guarantees apply. Our approach yields bisimulation guarantees while learning the distilled policy, allowing concrete optimization of the abstraction and representation model quality. Our experiments show that, besides distilling policies up to 10 times faster, the latent model quality is indeed better in general. Moreover, we present experiments from a simple time-to-failure verification algorithm on the latent space. The fact that our approach enables such simple verification techniques highlights its applicability.

The Concrete Distribution: A Continuous Relaxation Of Discrete Random Variables

Abstract: The reparameterization trick enables optimizing large scale stochastic computation graphs via gradient descent. The essence of the trick is to refactor each stochastic node into a differentiable function of its parameters and a random variable with fixed distribution. After refactoring, the gradients of the loss propagated by the chain rule through the graph are low variance unbiased estimators of the gradients of the expected loss. While many continuous random variables have such reparameterizations, discrete random variables lack useful reparameterizations due to the discontinuous nature of discrete states. In this work we introduce Concrete random variables -- continuous relaxations of discrete random variables. The Concrete distribution is a new family of distributions with closed form densities and a simple reparameterization. Whenever a discrete stochastic node of a computation graph can be refactored into a one-hot bit representation that is treated continuously, Concrete stochastic nodes can be used with automatic differentiation to produce low-variance biased gradients of objectives (including objectives that depend on the log-probability of latent stochastic nodes) on the corresponding discrete graph. We demonstrate the effectiveness of Concrete relaxations on density estimation and structured prediction tasks using neural networks.

continuous relaxation
discrete distributions


Following ProHMR, we use 6D representation #REFR to model rotations and the L orth loss to force the 6D representations of the samples drawn from the distribution to be close to the orthonormal 6D representation.

For each image I, the mode θ * I of the output distribution corresponds to the transformation of z = 0.

We do this by explicitly supervising θ * I with all the available annotations as in a standard regression framework and minimize

The loss weights are defined as: λ nll = 0.001, λ exp,2D = 0.001, λ exp,adv = λ mode,adv = 0.01, λ mode,2D = 0.01, λ mode,3D = 0.05, λ mode,θ = 0.001, λ mode,β = 0.0005 and λ orth = 0.1.

Tape: Temporal Attention-Based Probabilistic Human Pose And Shape Estimation

Abstract: Reconstructing 3D human pose and shape from monocular videos is a well-studied but challenging problem. Common challenges include occlusions, the inherent ambiguities in the 2D to 3D mapping and the computational complexity of video processing. Existing methods ignore the ambiguities of the reconstruction and provide a single deterministic estimate for the 3D pose. In order to address these issues, we present a Temporal Attention based Probabilistic human pose and shape Estimation method (TAPE) that operates on an RGB video. More specifically, we propose to use a neural network to encode video frames to temporal features using an attention-based neural network. Given these features, we output a per-frame but temporally-informed probability distribution for the human pose using Normalizing Flows. We show that TAPE outperforms state-of-the-art methods in standard benchmarks and serves as an effective video-based prior for optimizationbased human pose and shape estimation.

On The Continuity Of Rotation Representations In Neural Networks

Abstract: In neural networks, it is often desirable to work with various representations of the same space. For example, 3D rotations can be represented with quaternions or Euler angles. In this paper, we advance a definition of a continuous representation, which can be helpful for training deep neural networks. We relate this to topological concepts such as homeomorphism and embedding. We then investigate what are continuous and discontinuous representations for 2D, 3D, and n-dimensional rotations. We demonstrate that for 3D rotations, all representations are discontinuous in the real Euclidean spaces of four or fewer dimensions. Thus, widely used representations such as quaternions and Euler angles are discontinuous and difficult for neural networks to learn. We show that the 3D rotations have continuous representations in 5D and 6D, which are more suitable for learning. We also present continuous representations for the general case of the n-dimensional rotation group SO(n). While our main focus is on rotations, we also show that our constructions apply to other groups such as the orthogonal group and similarity transforms. We finally present empirical results, which show that our continuous rotation representations outperform discontinuous ones for several practical problems in graphics and vision, including a simple autoencoder sanity test, a rotation estimator for 3D point clouds, and an inverse kinematics solver for 3D human poses.


D. Metrics Of Faithfulness To The Source

Such observations mirror the conclusion of #REFR , about the effectiveness of deep features as perceptual metrics.

On the contrary, the Type-2 examples suggest that I-L 2 , the L 2 on Inception latent features, might be a better measurement of semantic consistency.

While EGSDE fails to maintain features such as background, facial expression (neural v.s.

smile), eye movement, and prominent bone structure and produces over-generalized translations, UVC-GANv2 translations with significantly lower I-L 2 manage to preserve those features and appear more individualized.

These examples illustrate that the pixel-wise L 2 faithfulness metric may be in poor agreement with a human judgment of image faithfulness.

They also point to a possibility that the I-L 2 distances, based on deep features of Inception-v3, may better capture the perceived image faithfulness.

However, we stress again, that the main purpose of this paper is to improve the performance of the classic Cycle-GAN architecture, not the development of better faithfulness metrics.

While this section points to a possibility of I-L 2 being a better faithfulness metric, a full-scale investigation needs to be conducted to conclusively establish this. We leave such a study for future work.

Rethinking Cyclegan: Improving Quality Of Gans For Unpaired Image-To-Image Translation

Abstract: An unpaired image-to-image (I2I) translation technique seeks to find a mapping between two domains of data in a fully unsupervised manner. While the initial solutions to the I2I problem were provided by the generative adversarial neural networks (GANs), currently, diffusion models (DM) hold the state-of-the-art status on the I2I translation benchmarks in terms of FID. Yet, they suffer from some limitations, such as not using data from the source domain during the training, or maintaining consistency of the source and translated images only via simple pixel-wise errors. This work revisits the classic CycleGAN model and equips it with recent advancements in model architectures and model training procedures. The revised model is shown to significantly outperform other advanced GAN-and DM-based competitors on a variety of benchmarks. In the case of Male-to-Female translation of CelebA, the model achieves over 40% improvement in FID score compared to the stateof-the-art results. This work also demonstrates the ineffectiveness of the pixel-wise I2I translation faithfulness metrics and suggests their revision. The code and trained models are available at https://github.com/LS4GAN/ uvcgan2.

The Unreasonable Effectiveness Of Deep Features As A Perceptual Metric

Abstract: While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on the ImageNet classification task has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called"perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new Full Reference Image Quality Assessment (FR-IQA) dataset of perceptual human judgments, orders of magnitude larger than previous datasets. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by huge margins. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.


Two example alignments between source and target tokens induced by attention values in the work of #REFR .

The brightness of each cell is proportional to the value of s(h i , c j ), where the columns i correspond to source tokens and the rows j correspond to target tokens. . . . . . . . .

Abstract: Recent progress in NLP can largely be attributed to progress in Deep Representation Learning techniques, which are based on opaque statistical learning methods. The success of these methods is due to the availability of large volumes of data which they leverage in pre-training schemes to improve generalization when fine-tuned on specific NLP tasks. Producing such high performance representations with an understandable meaning most often calls for the availability of annotated data. However, annotated data comes at great costs, and incurs recurring annotation effort which is impractical when deploying models at scale, i.e. for different languages, different tasks, and on different domains.

Neural Machine Translation By Jointly Learning To Align And Translate

Abstract: Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.


CMAE #REFR creates an exploration compartment for each agent that is not shared with other agents, drastically reducing the searching space via cooperative behavior.

Although these methods are effective in the single-agent domain, naïvely applying them separately and independently to each agent in the MARL setting is bound to result in suboptimal performance.

This is because the agents mutually influence one another, creating additional uncertainty that has to be taken into account.

One way of accounting for such uncertainty is to require cooperative behavior between agents.

Recently, there have been some works on ensuring cooperation between the agents.

MAVEN #OTHEREFR uses a hierarchical architecture to generate a shared latent vector for each agent to explore the space cooperatively.

However, none of these algorithms account for inherent uncertainty and are applied to other MARL algorithms.

Toward Risk-Based Optimistic Exploration For Cooperative Multi-Agent Reinforcement Learning

Abstract: The multi-agent setting is intricate and unpredictable since the behaviors of multiple agents influence one another. To address this environmental uncertainty, distributional reinforcement learning algorithms that incorporate uncertainty via distributional output have been integrated with multi-agent reinforcement learning (MARL) methods, achieving state-of-the-art performance. However, distributional MARL algorithms still rely on the traditional-greedy, which does not take cooperative strategy into account. In this paper, we present a risk-based exploration that leads to collaboratively optimistic behavior by shifting the sampling region of distribution. Initially, we take expectations from the upper quantiles of state-action values for exploration, which are optimistic actions, and gradually shift the sampling region of quantiles to the full distribution for exploitation. By ensuring that each agent is exposed to the same level of risk, we can force them to take cooperatively optimistic actions. Our method shows remarkable performance in multi-agent settings requiring cooperative exploration based on quantile regression appropriately controlling the level of risk.

Pic: Permutation Invariant Critic For Multi-Agent Deep Reinforcement Learning

Abstract: Sample efficiency and scalability to a large number of agents are two important goals for multi-agent reinforcement learning systems. Recent works got us closer to those goals, addressing non-stationarity of the environment from a single agent's perspective by utilizing a deep net critic which depends on all observations and actions. The critic input concatenates agent observations and actions in a user-specified order. However, since deep nets aren't permutation invariant, a permuted input changes the critic output despite the environment remaining identical. To avoid this inefficiency, we propose a 'permutation invariant critic' (PIC), which yields identical output irrespective of the agent permutation. This consistent representation enables our model to scale to 30 times more agents and to achieve improvements of test episode reward between 15% to 50% on the challenging multi-agent particle environment (MPE).


In this regard, we further propose a teacher-student #REFR approach to generate pseudo labels effectively and fine-tune g(x) simultaneously.

However, the domain shift between WSI and natural images may lead to sub-optimal representations, so recently there have been methods proposed to fine-tune g(x) using self-supervised techniques #OTHEREFR or weakly-supervised techniques #OTHEREFR (as shown in Fig.2(c) ).

Nevertheless, since these two processes are still trained separately with different supervision signals, they lack joint optimization and may still leads to inconsistencies within the entire MIL pipeline.

To address the challenges mentioned above, we propose a novel MIL framework called ICMIL, which can iteratively couple the patch feature embedding process with the bag-level classification process to enhance the effectiveness of MIL training (as illustrated in Fig. 2(d) ).

Unlike previous works that mainly focused on designing sophisticated instance aggregators a(x) #OTHEREFR and bag classifiers f (x) #OTHEREFR , we aim to bridge the loss back-propagation process from f (x) to g(x) to improve g(x)'s ability to perceive slide-level labels.

Specifically, we propose to use the bag-level classifier f (x) to initialize an instance-level classifier f (x), enabling f (x) to use the category knowledge learned from bag-level features to determine each instance's category.

After fine-tuning, the domain shift problem is alleviated in g(x), leading to better patch representations.

The new representations can be used to train a better bag-level classifier in return for the next round of iteration.

In summary, our contributions are: #OTHEREFR We propose ICMIL which bridges the loss propagation from the bag classifier to the patch embedder by iteratively coupling them during training.

This framework fine-tunes the patch embedder based on the bag-level classifier, and the refined embeddings, in turn, help train a more accurate bag-level classifier.

2We propose a teacher-student approach to achieve effective and robust knowledge transfer from the bag-level classifier f (x) to the instance-level representation embedder g(x).

Iteratively Coupled Multiple Instance Learning From Instance To Bag Classifier For Whole Slide Image Classification

Abstract: Whole Slide Image (WSI) classification remains a challenge due to their extremely high resolution and the absence of fine-grained labels. Presently, WSIs are usually classified as a Multiple Instance Learning (MIL) problem when only slide-level labels are available. MIL methods involve a patch embedding process and a bag-level classification process, but they are prohibitively expensive to be trained end-to-end. Therefore, existing methods usually train them separately, or directly skip the training of the embedder. Such schemes hinder the patch embedder's access to slide-level labels, resulting in inconsistencies within the entire MIL pipeline. To overcome this issue, we propose a novel framework called Iteratively Coupled MIL (ICMIL), which bridges the loss back-propagation process from the bag-level classifier to the patch embedder. In ICMIL, we use category information in the bag-level classifier to guide the patch-level fine-tuning of the patch feature extractor. The refined embedder then generates better instance representations for achieving a more accurate bag-level classifier. By coupling the patch embedder and bag classifier at a low cost, our proposed framework enables information exchange between the two processes, benefiting the entire MIL classification model. We tested our framework on two datasets using three different backbones, and our experimental results demonstrate consistent performance improvements over state-of-the-art MIL methods. Code will be made available upon acceptance.

Distilling The Knowledge In A Neural Network

Abstract: A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.


The library is built entirely on Py-Torch #REFR and comprises five modules: Configuration, Data, Model, Evaluation, and Execution.

Based on this, we introduce LibCity #OTHEREFR , an open-source library that supports standardized measurement of models.

Using LibCity, we conducted a series of experiments to validate the effectiveness of different models and components.

By providing a standard library for urban spatial-temporal prediction, we aim to enhance reproducibility, comparability, and the advancement of this field.

The main features of LibCity can be summarized in four aspects:

• Unified and Modular Framework Design: LibCity adopts a comprehensive and standardized approach to implementing, deploying, and evaluating spatial-temporal prediction models.

Each module has a well-defined scope and collaborates seamlessly with others to deliver the complete functionality of the library.

We design basic spatial-temporal data storage, unified model instantiation interfaces, and standardized evaluation procedure within these modules.

Users can effortlessly train and evaluate existing models with simple configurations.

On the other hand, developers can concentrate solely on the interfaces that are relevant to their models without worrying about the implementation details of other modules.

• General and Extensible Data Storage Format: Opensource spatial-temporal datasets are available in various storage formats.

Towards Efficient And Comprehensive Urban Spatial-Temporal Prediction: A Unified Library And Performance Benchmark

Abstract: As deep learning technology advances and more urban spatial-temporal data accumulates, an increasing number of deep learning models are being proposed to solve urban spatial-temporal prediction problems. However, there are limitations in the existing field, including open-source data being in various formats and difficult to use, few papers making their code and data openly available, and open-source models often using different frameworks and platforms, making comparisons challenging. A standardized framework is urgently needed to implement and evaluate these methods. To address these issues, we provide a comprehensive review of urban spatial-temporal prediction and propose a unified storage format for spatialtemporal data called atomic files. We also propose LibCity, an open-source library that offers researchers a credible experimental tool and a convenient development framework. In this library, we have reproduced 65 spatial-temporal prediction models and collected 55 spatial-temporal datasets, allowing researchers to conduct comprehensive experiments conveniently. Using LibCity, we conducted a series of experiments to validate the effectiveness of different models and components, and we summarized promising future technology developments and research directions for spatial-temporal prediction. By enabling fair model comparisons, designing a unified data storage format, and simplifying the process of developing new models, LibCity is poised to make significant contributions to the spatial-temporal prediction field.

Pytorch: An Imperative Style, High-Performance Deep Learning Library

Abstract: Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it provides an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. ::: In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. ::: We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several common benchmarks.


Specifically, we derive greedy selection rules for ℓ 0 w -based variants of the SR-LASSO (Appendix B.1) and LAD-LASSO (Appendix B.2), extending the ℓ 0 w -based LASSO setting considered in #REFR .

l(t) = m i=1 |r i − tA ij | + λw j |x j + t| = m+1 i=1 r j i − t A ij

and, thanks to Lemma 1, l(t) is minimized at t = r ĵ i(j) / Aî (j),j , whereî(j) = τ (k(j)) andk(j) is defined as in #OTHEREFR B Greedy selection rules for ℓ 0 w -based loss functions

In this appendix we show how to derive greedy selection rules for ℓ 0 w -regularized loss functions.

The corresponding weighted OMP algorithms are numerically tested in §4, Experiments I and II.

The Greedy Side Of The Lasso: New Algorithms For Weighted Sparse Recovery Via Loss Function-Based Orthogonal Matching Pursuit

Abstract: We propose a class of greedy algorithms for weighted sparse recovery by considering new loss function-based generalizations of Orthogonal Matching Pursuit (OMP). Given a (regularized) loss function, the proposed algorithms alternate the iterative construction of the signal support via greedy index selection and a signal update based on solving a local data-fitting problem restricted to the current support. We show that greedy selection rules associated with popular weighted sparsity-promoting loss functions admit explicitly computable and simple formulas. Specifically, we consider ℓ 0-and ℓ 1-based versions of the weighted LASSO (Least Absolute Shrinkage and Selection Operator), the Square-Root LASSO (SR-LASSO) and the Least Absolute Deviations LASSO (LAD-LASSO). Through numerical experiments on Gaussian compressive sensing and high-dimensional function approximation, we demonstrate the effectiveness of the proposed algorithms and empirically show that they inherit desirable characteristics from the corresponding loss functions, such as SR-LASSO's noise-blind optimal parameter tuning and LAD-LASSO's fault tolerance. In doing so, our study sheds new light on the connection between greedy sparse recovery and convex relaxation.

Sparse Approximation Of Multivariate Functions From Small Datasets Via Weighted Orthogonal Matching Pursuit

Abstract: We show the potential of greedy recovery strategies for the sparse approximation of multivariate functions from a small dataset of pointwise evaluations by considering an extension of the orthogonal matching pursuit to the setting of weighted sparsity. The proposed recovery strategy is based on a formal derivation of the greedy index selection rule. Numerical experiments show that the proposed weighted orthogonal matching pursuit algorithm is able to reach accuracy levels similar to those of weighted l1 minimization programs while considerably improving the computational efficiency.


B. Evaluations Under The Carlini Wagner (Cw) Attack

In fact, it reaches a 100% fooling rate for all three datasets. Similar findings were also made in #REFR .

For our estimations, we set a maximum of 10 iterations for the MNIST and 5 iterations for the CIFAR10 dataset.

In addition, for the attacks on CIFAR10, the algorithm has been forced to set the confidence of the targeted class to 0.5 higher than the confidence of the true label.

Table II reports the fooling rate and the mean distortion for both the datasets.

CNNs: The CW attack is very effective for traditional CNNs.

On the other hand, both CapsNets show greater robustness (i.e., lower fooling rate) than CNNs, for the MNIST dataset (and also for the GTSRB, even if the fooling rate of the DeepCaps is just a little bit lower than 100%).

The CapsNets also require a higher mean distortion than the CNNs, which makes the resulting adversarial example more perceptible.

For the CIFAR10 dataset, the CW attack shows its effectiveness because, for all the networks, the fooling rate is 100%.

However, we can notice that CapsNets are more robust due to a higher mean distortion. DeepCaps vs.

ShallowCaps: The DeepCaps appears to be more robust than the ShallowCaps, because of a lower fooling rate, despite having slightly lower mean distortion.

Robcaps: Evaluating The Robustness Of Capsule Networks Against Affine Transformations And Adversarial Attacks

Abstract: Capsule Networks (CapsNets) are able to hierarchically preserve the pose relationships between multiple objects for image classification tasks. Other than achieving high accuracy, another relevant factor in deploying CapsNets in safety-critical applications is the robustness against input transformations and malicious adversarial attacks. In this paper, we systematically analyze and evaluate different factors affecting the robustness of CapsNets, compared to traditional Convolutional Neural Networks (CNNs). Towards a comprehensive comparison, we test two CapsNet models and two CNN models on the MNIST, GTSRB, and CIFAR10 datasets, as well as on the affine-transformed versions of such datasets. With a thorough analysis, we show which properties of these architectures better contribute to increasing the robustness and their limitations. Overall, CapsNets achieve better robustness against adversarial examples and affine transformations, compared to a traditional CNN with a similar number of parameters. Similar conclusions have been derived for deeper versions of CapsNets and CNNs. Moreover, our results unleash a key finding that the dynamic routing does not contribute much to improving the CapsNets' robustness. Indeed, the main generalization contribution is due to the hierarchical feature learning through capsules.

Towards Evaluating The Robustness Of Neural Networks

Abstract: Neural networks provide state-of-the-art results for most machine learning tasks. Unfortunately, neural networks are vulnerable to adversarial examples: given an input $x$ and any target classification $t$, it is possible to find a new input $x'$ that is similar to $x$ but classified as $t$. This makes it difficult to apply neural networks in security-critical areas. Defensive distillation is a recently proposed approach that can take an arbitrary neural network, and increase its robustness, reducing the success rate of current attacks' ability to find adversarial examples from $95\%$ to $0.5\%$. ::: In this paper, we demonstrate that defensive distillation does not significantly increase the robustness of neural networks by introducing three new attack algorithms that are successful on both distilled and undistilled neural networks with $100\%$ probability. Our attacks are tailored to three distance metrics used previously in the literature, and when compared to previous adversarial example generation algorithms, our attacks are often much more effective (and never worse). Furthermore, we propose using high-confidence adversarial examples in a simple transferability test we show can also be used to break defensive distillation. We hope our attacks will be used as a benchmark in future defense attempts to create neural networks that resist adversarial examples.


Evaluation On Music Conditioned Dance Generation

And for DanceNet #REFR , it generates dance in a repeat motion pattern, nearly zero freezing but with low diversity.

The existing evaluation metrics are not sufficient to reflect the quality of generated dance in practice, which motivates us to propose a PFF and AUC f to evaluate the percentage of freezing frames.

It is worth noting that our architecture outperforms FACT #OTHEREFR and Bailando #OTHEREFR in terms of PFF and AUC f , which shows that our method rarely generates frozen frames. Meanwhile, Li et al.

#OTHEREFR and DanceNet #OTHEREFR also gain nearly zero freezing in PFF and AUC f .

The reason is that the generated dances of Li et al.

#OTHEREFR are highly jittery making its velocity variation extremely high, which is also reported in #OTHEREFR , and leads to the non-freezing issue.

Furthermore, the results of user study show that the generated 3D dance is more visually realistic compared to other methods.

Even in comparison to the ground truth, 41.9% of our generated dance is voted as Table 3 : The result of text2motion, only mean reported.

We also report the result trained with dance-only data, which shows comparable performance.

In addition to the evaluation on the test set of AIST++, we also show the experimental results on our in-the-wild dataset in Table 1 .

Fig-1: Demo

Abstract: Audio input Effect range: 6s ~ 8s A person does a cartwheel. Effect range : 16s ~ 18s A person does a spin.

Music2Dance: Dancenet For Music-Driven Dance Generation

Abstract: Synthesize human motions from music, i.e., music to dance, is appealing and attracts lots of research interests in recent years. It is challenging due to not only the requirement of realistic and complex human motions for dance, but more importantly, the synthesized motions should be consistent with the style, rhythm and melody of the music. In this paper, we propose a novel autoregressive generative model, DanceNet, to take the style, rhythm and melody of music as the control signals to generate 3D dance motions with high realism and diversity. To boost the performance of our proposed model, we capture several synchronized music-dance pairs by professional dancers, and build a high-quality music-dance pair dataset. Experiments have demonstrated that the proposed method can achieve the state-of-the-art results.


On LiTS17 #REFR , as shown in Table 2, our model achieves 91.50% in Dice and 84.59% in IoU, outperforming SOTA methods by 0.26% and 0.45%, respectively.

We compare our CTO with the state-of-the-art (SOTA) methods including U-Net #OTHEREFR , ResUNet #OTHEREFR with ResNet-50 #OTHEREFR as the backbone, VNet #OTHEREFR , ViT #OTHEREFR , TransUNet #OTHEREFR , and Swin-Unet #OTHEREFR .

On ISIC 2016 #OTHEREFR in IoU, which outperforms the SOTA methods by 0.05% and 0.88%, respectively.

On ISIC 2018 #OTHEREFR , our CTO achieves 91.2% in Dice and 84.5% in IoU by 5-fold crossvalidation, which outperforms the SOTA methods by 1.8% and 2.3%, respectively.

On CoNIC #OTHEREFR , results are shown in Table 2 , we can observe that our CTO achieves 79.77%, 66.42%, and 65.58% in Dice, IoU, and PQ, respectively, consistently outperforming other methods. Qualitative result comparisons are illustrated in Figure 2 .

We can observe that our CTO delineates more accurate object contours than other methods regarding diverse shapes and sizes of nuclei, especially on some blurred nuclei objects. We also conduct experiments on 3D MISeg tasks.

On BTCV #OTHEREFR , as shown in Table 3 , our CTO achieves 81.10% in Dice and 18.75% in HD, which outperforms the SOTA methods.

In particular, as for Dice, our CTO outperforms the second, third, and fourth best methods by 1.98%, 3.33%, and 3.62, respectively. Besides, the distinct improvements Table 2 .

Comparisons with other methods on CoNIC #OTHEREFR and LiTS17 #OTHEREFR .

can be markedly observed for organs with blurry boundaries, e.g., the "pancreas" and the "stomach", where our model achieves significant gains over the SOTA methods, i.e., 4.70% and 3.60% in Dice, respectively.

As for the model efficiency, we can observe that CTO achieves competitive performance improvements with comparable FLOPs and parameters. Ablation Study.

Rethinking Boundary Detection In Deep Learning Models For Medical Image Segmentation

Abstract: Medical image segmentation is a fundamental task in the community of medical image analysis. In this paper, a novel network architecture, referred to as Convolution, Transformer, and Operator (CTO), is proposed. CTO employs a combination of Convolutional Neural Networks (CNNs), Vision Transformer (ViT), and an explicit boundary detection operator to achieve high recognition accuracy while maintaining an optimal balance between accuracy and efficiency. The proposed CTO follows the standard encoder-decoder segmentation paradigm, where the encoder network incorporates a popular CNN backbone for capturing local semantic information, and a lightweight ViT assistant for integrating long-range dependencies. To enhance the learning capacity on boundary, a boundary-guided decoder network is proposed that uses a boundary mask obtained from a dedicated boundary detection operator as explicit supervision to guide the decoding learning process. The performance of the proposed method is evaluated on six challenging medical image segmentation datasets, demonstrating that CTO achieves state-of-the-art accuracy with a competitive model complexity.

The Liver Tumor Segmentation Benchmark (Lits)

Abstract: In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LITS) organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2016 and International Conference On Medical Image Computing Computer Assisted Intervention (MICCAI) 2017. Twenty four valid state-of-the-art liver and liver tumor segmentation algorithms were applied to a set of 131 computed tomography (CT) volumes with different types of tumor contrast levels (hyper-/hypo-intense), abnormalities in tissues (metastasectomie) size and varying amount of lesions. The submitted algorithms have been tested on 70 undisclosed volumes. The dataset is created in collaboration with seven hospitals and research institutions and manually reviewed by independent three radiologists. We found that not a single algorithm performed best for liver and tumors. The best liver segmentation algorithm achieved a Dice score of 0.96(MICCAI) whereas for tumor segmentation the best algorithm evaluated at 0.67(ISBI) and 0.70(MICCAI). The LITS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.