By comparison, the image generated by the Bicubic interpolation contains serious noises, while the image predicted by SRCNN #REFR is more blurry than others.

2 , there exist dozens of obvious reddish and blackish noisy points on the white vertical ridge of the building in images predicted by the Bicubic and CNN-based methods.

In contrast, for those images generated by Transformer-based methods, the existence of such severe noisy points is much less frequent.

Specifically, images predicted by SwinIR #OTHEREFR and LSwinSR contain only two and one such noisy points, respectively.

With the increase of the upsample scale, the available information for accurate reconstruction becomes more and more scarce, leading to great challenges for super-resolution algorithms. For example, in the top part of Fig.

4 , only SwinIR #OTHEREFR and the proposed LSwinSR provide smooth results for the text in the image when upsampling ×4 scale.

For LP-KPN #OTHEREFR and NLSA #OTHEREFR , the problem is the unnecessary wrinkles in the reconstructed texts.

When it comes to ×8 super-resolution, the super-resolution algorithms can only rely on very limited input information. As shown in Fig.

7 , the details of the building structure are nearly indistinguishable for all methods, but the Transformer-based methods still provide a clearer contour.

As the proposed LSwinSR is an improved and simplified version of SwinIR #OTHEREFR , the complexity and the inference speed between the two models are compared.

As can be seen in Table 2, the complexity and the inference speed of our LSwinSR are slightly better than SwinIR #OTHEREFR when the window size is eight.

Lswinsr: Uav Imagery Super-Resolution Based On Linear Swin Transformer

Abstract: Super-resolution, which aims to reconstruct high-resolution images from low-resolution images, has drawn considerable attention and has been intensively studied in computer vision and remote sensing communities. The super-resolution technology is especially beneficial for Unmanned Aerial Vehicles (UAV), as the amount and resolution of images captured by UAV are highly limited by physical constraints such as flight altitude and load capacity. In the wake of the successful application of deep learning methods in the super-resolution task, in recent years, a series of super-resolution algorithms have been developed. In this paper, for the super-resolution of UAV images, a novel network based on the state-of-the-art Swin Transformer is proposed with better efficiency and competitive accuracy. Meanwhile, as one of the essential applications of the UAV is land cover and land use monitoring, simple image quality assessments such as the Peak-Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index Measure (SSIM) are not enough to comprehensively measure the performance of an algorithm. Therefore, we further investigate the effectiveness of super-resolution methods using the accuracy of semantic segmentation. The code will be available at https://github.com/ lironui/GeoSR.

Image Super-Resolution Using Deep Convolutional Networks

Abstract: We propose a deep learning method for single image super-resolution (SR). Our method directly learns an end-to-end mapping between the low/high-resolution images. The mapping is represented as a deep convolutional neural network (CNN) that takes the low-resolution image as the input and outputs the high-resolution one. We further show that traditional sparse-coding-based SR methods can also be viewed as a deep convolutional network. But unlike traditional methods that handle each component separately, our method jointly optimizes all layers. Our deep CNN has a lightweight structure, yet demonstrates state-of-the-art restoration quality, and achieves fast speed for practical on-line usage. We explore different network structures and parameter settings to achieve trade-offs between performance and speed. Moreover, we extend our network to cope with three color channels simultaneously, and show better overall reconstruction quality.


Bounds On The P→Q Disagreement Coefficient

We can apply the whitening transform from Theorem 16 of #REFR provided (1/20 + c 1 ) 1/C 1 − c 2 1 ≤ 1/9, where C 1 = e β1 log 2 (d+1) and c 1 = e(C 1 − 1) √ 2C 1 .

Then for 0 ≤ β 1 , β 2 ≤ 1 56 log 2 (d+1) , we have Θ P→Q (ε) = O(d 1/2+ β 2 2 ln 2 log(d/ε)).

Our proof builds on and generalizes the arguments used in the proof of Theorem 14 in #OTHEREFR .

Let h ∈ B P (h * , r), i.e. d(h, h * ) ≤ r.

It may be verified that this condition holds for 0 ≤ β 1 ≤ 1 56 log 2 (d+1) .

Now, by Theorem 11 of #OTHEREFR we can bound the angle between their normal vectors as θ(w h , w h * ) ≤ cr where c is an absolute constant.

Now if x ∈ X has a large margin |w h * • x| ≥ crα and small norm ||x|| ≤ α, for some α > 0, we have

|w h • x − w h * • x| ≤ ||w h − w h * || • ||x|| < crα.

Now the large margin condition |w h * • x| ≥ crα implies w h , x w h * , x > 0, or h(x) = h * (x). Since h ∈ B P (h * , r)

Abstract: Machine learning algorithms are often used in environments which are not captured accurately even by the most carefully obtained training data, either due to the possibility of 'adversarial' test-time attacks, or on account of 'natural' distribution shift. For test-time attacks, we introduce and analyze a novel robust reliability guarantee, which requires a learner to output predictions along with a reliability radius η, with the meaning that its prediction is guaranteed to be correct as long as the adversary has not perturbed the test point farther than a distance η. We provide learners that are optimal in the sense that they always output the best possible reliability radius on any test point, and we characterize the reliable region, i.e. the set of points where a given reliability radius is attainable. We additionally analyze reliable learners under distribution shift, where the test points may come from an arbitrary distribution Q different from the training distribution P. For both cases, we bound the probability mass of the reliable region for several interesting examples, for linear separators under nearly log-concave and s-concave distributions, as well as for smooth boundary classifiers under smooth probability distributions.

Active And Passive Learning Of Linear Separators Under Log-Concave Distributions

Abstract: We provide new results concerning label efficient, polynomial time, passive and active learning of linear separators. We prove that active learning provides an exponential improvement over PAC (passive) learning of homogeneous linear separators under nearly log-concave distributions. Building on this, we provide a computationally efficient PAC algorithm with optimal (up to a constant factor) sample complexity for such problems. This resolves an open question concerning the sample complexity of efficient PAC algorithms under the uniform distribution in the unit ball. Moreover, it provides the first bound for a polynomial-time PAC algorithm that is tight for an interesting infinite class of hypothesis functions under a general and natural class of data-distributions, providing significant progress towards a longstanding open question. ::: We also provide new bounds for active and passive learning in the case that the data might not be linearly separable, both in the agnostic case and and under the Tsybakov low-noise condition. To derive our results, we provide new structural results for (nearly) log-concave distributions, which might be of independent interest as well.


We obtained embeddings of each target word in each of these contexts using a pretrained version of BERT (model version: bert-large-uncased; PyTorch Transformers library; Wolf et al., 2019; #REFR .

Starting from the sentence in which the target word occurs, we compared the length of the target sentence (current context) and the length of its immediate broader context extended to the sentence before and after (broader context), a total of three sentences in the first instance.

We then determined whether the current context or the broader context was closest to our desired 100-word window.

If the broader context was closest, then we repeated the process (adding an additional sentence before and after the current context) until the current context was closest to 100 words.

This process resulted in a set of naturalistic contexts with varying lengths of approximately 100-words (M=99.94; SD=16.34).

In total, we retrieved 2,589,921 contexts from the BNC (an average of 3,368 contexts per target word; see supplementary Table 1 in the OSF site for this project for distribution of contexts across conditions).

To derive contextual embeddings for our target words, we submitted all 2,589,921 contexts (as defined above) as input to the pre-trained model.

We then extracted multidimensional representations of each target word in each context (i.e.

the contextual embeddings) from the final layer of the language model.

Each of the resulting 2,589,921 contextual embeddings had 1,024 dimensions.

We ran simulation analyses of lexical ambiguity studies #OTHEREFR to determine whether the ambiguity structure of target words in these studies was apparent in the sample of BERT contextual embeddings obtained for each of these target words.

Shades Of Meaning 1 Shades Of Meaning: Uncovering The Geometry Of Ambiguous Word Representations Through Contextualised Language Models

Abstract: Lexical ambiguity presents a profound and enduring challenge to the language sciences. Researchers for decades have grappled with the problem of how language users learn, represent and process words with more than one meaning. Our work offers new insight into psychological understanding of lexical ambiguity through a series of simulations that capitalise on recent advances in contextual language models. These models have no grounded understanding of the meanings of words at all; they simply learn to predict words based on the surrounding context provided by other words. Yet, our analyses show that their representations capture fine-grained meaningful distinctions between unambiguous, homonymous, and polysemous words that align with lexicographic classifications and psychological theorising. These findings provide quantitative support for modern psychological conceptualisations of lexical ambiguity and raise new challenges for understanding of the way that contextual information shapes the meanings of words across different timescales.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


In this section, we present our method for anonymizing the identity of faces in a given real face dataset by optimizing the representations of the dataset's images in the latent space of a pre-trained StyleGAN2 #REFR .

More specifically, given a real dataset X R , we first create a fake dataset X F by randomly generating a large set (i.e., such that |X F | > |X R |) of fake images and obtaining the corresponding latent codes in the W + space of StyleGAN2, namely, W + F .

Additionally, we obtain the latent codes of the real dataset in the W + space by inverting its images using e4e #OTHEREFR , arriving at a set of latent codes W + R .

In order to obtain meaningful initial values for the latent codes that will be optimized to create the anonymized version of the real dataset, namely X A , we first pair the real images from the original set (i.e., X R ) with fake ones from the generated dataset (i.e., X F ) in the feature space of the ViT-based FaRL #OTHEREFR image encoder and use their latent codes for initializing the aforementioned trainable codes.

The latent codes of the anonymized dataset are then optimized under the following objectives via two novel loss functions: (a) to be similar to the corresponding real ones, up to a certain margin, using the proposed identity loss (L id ), and (b) to preserve the facial attributes of the corresponding real ones by being pulled closer in the Figure 2 .

Overview of the proposed method: optimizing the trainable portion of the latent code w iÃ ∈ R 5×512 to obfuscate the identity of the resulting synthetic image x i A with L id whilst preserving the facial attributes with Latt.

Attribute-Preserving Face Dataset Anonymization Via Latent Code Optimization

Abstract: This work addresses the problem of anonymizing the identity of faces in a dataset of images, such that the privacy of those depicted is not violated, while at the same time the dataset is useful for downstream task such as for training machine learning models. To the best of our knowledge, we are the first to explicitly address this issue and deal with two major drawbacks of the existing state-of-the-art approaches, namely that they (i) require the costly training of additional, purpose-trained neural networks, and/or (ii) fail to retain the facial attributes of the original images in the anonymized counterparts, the preservation of which is of paramount importance for their use in downstream tasks. We accordingly present a task-agnostic anonymization procedure that directly optimizes the images' latent representation in the latent space of a pre-trained GAN. By optimizing the latent codes directly, we ensure both that the identity is of a desired distance away from the original (with an identity obfuscation loss), whilst preserving the facial attributes (using a novel feature-matching loss in FaRL's [49] deep feature space). We demonstrate through a series of both qualitative and quantitative experiments that our method is capable of anonymizing the identity of the images whilstcrucially-better-preserving the facial attributes. We make the code and the pre-trained models publicly available at: https://github.com/chi0tzp/FALCO.

Analyzing And Improving The Image Quality Of Stylegan

Abstract: The style-based GAN architecture (StyleGAN) yields state-of-the-art results in data-driven unconditional generative image modeling. We expose and analyze several of its characteristic artifacts, and propose changes in both model architecture and training methods to address them. In particular, we redesign the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images. In addition to improving image quality, this path length regularizer yields the additional benefit that the generator becomes significantly easier to invert. This makes it possible to reliably attribute a generated image to a particular network. We furthermore visualize how well the generator utilizes its output resolution, and identify a capacity problem, motivating us to train larger models for additional quality improvements. Overall, our improved model redefines the state of the art in unconditional image modeling, both in terms of existing distribution quality metrics as well as perceived image quality.


Unlike most SDF-based methods such as DeepSDF #REFR which directly predicts the values of signed distance functions via neural networks, our model outputs the learnable parameters of curves that can be applied to handle complex glyph outlines containing concave curves. Fig. 2 shows the pipeline of our VecFontSDF.

In order to directly reconstruct and synthesize vector fonts from input glyph images via neural networks, we need a learnable shape representation that precisely preserves geometry information.

Previous methods that fully rely on sequential generative models such as SVG-VAE #OTHEREFR and DeepVecFont #OTHEREFR fail to perfectly handle this task mainly due to the ambiguity of control point selection.

To address this problem, we propose a vector graphic reconstructor based on an interpretable distance function with differentiable rendering.

For data preparation, we need to pre-calculate the values of signed distance functions (SDFs) for each input glyph using the method described in Sec. 3.1.

We have two types of precalculated SDFs: grid SDFs and contour SDFs.

Grid SDFs mean that the sampling points used to calculate SDFs are located at all grid positions (i.e., pixels in a raster image).

Contour SDFs mean that the sampling points are uniformly distributed near the contours of the input glyph.

Then, a CNN encoder is trained to extract the features from input raster images and an SDF decoder is followed to predict the parameters for every parabolic curve.

Vecfontsdf: Learning To Reconstruct And Synthesize High-Quality Vector Fonts Via Signed Distance Functions

Abstract: Font design is of vital importance in the digital content design and modern printing industry. Developing algorithms capable of automatically synthesizing vector fonts can significantly facilitate the font design process. However, existing methods mainly concentrate on raster image generation, and only a few approaches can directly synthesize vector fonts. This paper proposes an end-to-end trainable method, VecFontSDF, to reconstruct and synthesize high-quality vector fonts using signed distance functions (SDFs). Specifically, based on the proposed SDFbased implicit shape representation, VecFontSDF learns to model each glyph as shape primitives enclosed by several parabolic curves, which can be precisely converted to quadratic Bézier curves that are widely used in vector font products. In this manner, most image generation methods can be easily extended to synthesize vector fonts. Qualitative and quantitative experiments conducted on a publicly-available dataset demonstrate that our method obtains high-quality results on several tasks, including vector font reconstruction, interpolation, and few-shot vector font synthesis, markedly outperforming the state of the art.

Deep Parametric Shape Predictions Using Distance Fields

Abstract: Many tasks in graphics and vision demand machinery for converting shapes into consistent representations with sparse sets of parameters; these representations facilitate rendering, editing, and storage. When the source data is noisy or ambiguous, however, artists and engineers often manually construct such representations, a tedious and potentially time-consuming process. While advances in deep learning have been successfully applied to noisy geometric data, the task of generating parametric shapes has so far been difficult for these methods. Hence, we propose a new framework for predicting parametric shape primitives using deep learning. We use distance fields to transition between shape parameters like control points and input data on a pixel grid. We demonstrate efficacy on 2D and 3D tasks, including font vectorization and surface abstraction.


For each superpointŝ i , we construct a local patch G i using a point-to-node strategy #REFR .

In this paper, we supervise the superpoints to fall in the locally significant region, which may also lead to redundant proposals located in the same significant region.

Thus, we use a simple radius search-based filtering strategy to force only one proposal in the same region.

We iteratively perform a radius search for each proposal and filter out the ones close to the search center.

After that, we obtain the final superpointsŜ with associated featuresĤ.

Note that we limit the offsets ∆P to a certain range, which maintains the superpoints to be evenly distributed throughout the point cloud instead of only concentrated in so-called significant areas. It also avoids possible degeneracy #OTHEREFR .

Specifically, each point is assigned to its nearest superpoint by:

There are two advantages of this strategy.

First, it assigns every point to a specific superpoint without duplication or loss.

Second, it adapts to different densities, which is particularly suitable for our case since our superpoints break the uniformity of the original sampling after adding offsets.

Rdmnet: Reliable Dense Matching Based Point Cloud Registration For Autonomous Driving

Abstract: Point cloud registration is an important task in robotics and autonomous driving to estimate the ego-motion of the vehicle. Recent advances following the coarse-to-fine manner show promising potential in point cloud registration. However, existing methods rely on good superpoint correspondences, which are hard to be obtained reliably and efficiently, thus resulting in less robust and accurate point cloud registration. In this paper, we propose a novel network, named RDMNet, to find dense point correspondences coarse-to-fine and improve final pose estimation based on such reliable correspondences. Our RDMNet uses a devised 3D-RoFormer mechanism to first extract distinctive superpoints and generates reliable superpoints matches between two point clouds. The proposed 3D-RoFormer fuses 3D position information into the transformer network, efficiently exploiting point clouds' contextual and geometric information to generate robust superpoint correspondences. RDMNet then propagates the sparse superpoints matches to dense point matches using the neighborhood information for accurate point cloud registration. We extensively evaluate our method on multiple datasets from different environments. The experimental results demonstrate that our method outperforms existing state-of-the-art approaches in all tested datasets with a strong generalization ability.

Usip: Unsupervised Stable Interest Point Detection From 3D Point Clouds

Abstract: In this paper, we propose the USIP detector: an Unsupervised Stable Interest Point detector that can detect highly repeatable and accurately localized keypoints from 3D point clouds under arbitrary transformations without the need for any ground truth training data. Our USIP detector consists of a feature proposal network that learns stable keypoints from input 3D point clouds and their respective transformed pairs from randomly generated transformations. We provide degeneracy analysis of our USIP detector and suggest solutions to prevent it. We encourage high repeatability and accurate localization of the keypoints with a probabilistic chamfer loss that minimizes the distances between the detected keypoints from the training point cloud pairs. Extensive experimental results of repeatability tests on several simulated and real-world 3D point cloud datasets from Lidar, RGB-D and CAD models show that our USIP detector significantly outperforms existing hand-crafted and deep learning-based 3D keypoint detectors. Our code is available at the project website. this https URL


This new ca- #REFR The cost of training a Stable Diffusion model is around 600K USD.

Our model showed competitive improvements in FID score and user study ranking compared to the baselines but it still required finetuning for the overall performance boost.

Secondly, we aligned a multilingual language model, XLM-Roberta-L #OTHEREFR , using our approach, enabling multilingual text-to-image generation.

It achieved competitive results of translation-based models under a significantly lower training cost.

Finally, we demonstrated GlueNet's capability to bring new functionalities beyond text signals into existing T2I models.

The alignment of the AudioClip #OTHEREFR encoder enables sound-to-image generation without requiring any parameter finetuning of the image generator.

pability allows the existing Stable Diffusion model to generate high-quality images that correspond to sound signals such as dogs barking and street music.

This new capability goes beyond the traditional T2I generation and opens up new possibilities for creating multimedia content towards X-to-Image (X2I) generation.

Our contributions can be summarized as follows:

• To the best of our knowledge, this is the first work to consider the problem of efficiently aligning a pretrained audio model with a pre-trained T2I diffusion model for sound-to-image generation.

• Extensive experiments on text-to-image generation benchmarks demonstrate the superiority of our model over the baseline LDM method on both image quality and language controllability.

Gluegen: Plug And Play Multi-Modal Encoders For X-To-Image Generation

Abstract: Text-to-image (T2I) models based on diffusion processes have achieved remarkable success in controllable image generation using user-provided captions. However, the tight coupling between the current text encoder and image decoder in T2I models makes it challenging to replace or upgrade. Such changes often require massive fine-tuning or even training from scratch with the prohibitive expense. To address this problem, we propose GlueGen, which applies a newly proposed GlueNet model to align features from single-modal or multi-modal encoders with the latent space of an existing T2I model. The approach introduces a new training objective that leverages parallel corpora to align the representation spaces of different encoders. Empirical results show that GlueNet can be trained efficiently and enables various capabilities beyond previous state-of-the-art models: 1) multilingual language models such as XLM-Roberta can be aligned with existing T2I models, allowing for the generation of high-quality images from captions beyond English; 2) GlueNet can align multi-modal encoders such as AudioCLIP with the Stable Diffusion model, enabling sound-to-image generation; 3) it can also upgrade the current text encoder of the latent diffusion model for challenging case generation. By the alignment of various feature representations, the GlueNet allows for flexible and efficient integration of new functionality into existing T2I models and sheds light on X-to-image (X2I) generation. 1 * This work was done when Can Qin interned at Salesforce AI Research.

Fast, Accurate, And Lightweight Super-Resolution With Cascading Residual Network

Abstract: In recent years, deep learning methods have been successfully applied to single-image super-resolution tasks. Despite their great performances, deep learning methods cannot be easily applied to real-world applications due to the requirement of heavy computation. In this paper, we address this issue by proposing an accurate and lightweight deep network for image super-resolution. In detail, we design an architecture that implements a cascading mechanism upon a residual network. We also present variant models of the proposed cascading residual network to further improve efficiency. Our extensive experiments show that even with much fewer parameters and operations, our models achieve performance comparable to that of state-of-the-art methods.


Empirically, we found that for all strongly-connected and undirected graphs, the iterative algorithm in #REFR converges within N iterations, where N = |V| is the number of nodes.

x t+1 ∈ Q (j) ∞ that is reachable from x t . Since Q (j)

∞ ⊆ P req (y (j) ), the defender can also defend the time step t + 1.

Through mathematical induction, one can easily argue that being in Q (i) ∞ when y t−1 = y (i) guarantees indefinite defense for all no-splitting attacker strategies.

The conditions on the graph that guarantee convergence of the iterative algorithm in #OTHEREFR as well as the conditions for the existence of such fixed point(s) is an ongoing research.

Note that not all graphs have such a fixed point, for example, a sink graph (see Figure 13 in Appendix A) does not have one, since it requires infinite defender resources to guard indefinitely.

A follow-up work would be to establish the convergence guarantees.

Dynamic Adversarial Resource Allocation: The Ddab Game

Abstract: This work proposes a dynamic and adversarial resource allocation problem in a graph environment, which is referred to as the dynamic Defender-Attacker Blotto (dDAB) game. A team of defender robots is tasked to ensure numerical advantage at every node in the graph against a team of attacker robots. The engagement is formulated as a discrete-time dynamic game, where the two teams reallocate their robots in sequence and each robot can move at most one hop at each time step. The game terminates with the attacker's victory if any node has more attacker robots than defender robots. Our goal is to identify the necessary and sufficient number of defender robots to guarantee defense. Through a reachability analysis, we first solve the problem for the case where the attacker team stays as a single group. The results are then generalized to the case where the attacker team can freely split and merge into subteams. Crucially, our analysis indicates that there is no incentive for the attacker team to split, which significantly reduces the search space for the attacker's winning strategies and also enables us to design defender counter-strategies using superposition. We also present an efficient numerical algorithm to identify the necessary and sufficient number of defender robots to defend a given graph. Finally, we present illustrative examples to verify the efficacy of the proposed framework.

Characterizing The Interplay Between Information And Strength In Blotto Games

Abstract: In this paper, we investigate informational asymmetries in the Colonel Blotto game, a game-theoretic model of competitive resource allocation between two players over a set of battlefields. The battlefield valuations are subject to randomness. One of the two players knows the valuations with certainty. The other knows only a distribution on the battlefield realizations. However, the informed player has fewer resources to allocate. We characterize unique equilibrium payoffs in a two battlefield setup of the Colonel Blotto game. We then focus on a three battlefield setup in the General Lotto game, a popular variant of the Colonel Blotto game. We characterize the unique equilibrium payoffs and mixed equilibrium strategies. We quantify the value of information - the difference in equilibrium payoff between the asymmetric information game and complete information game. We find information strictly improves the informed player's performance guarantee. However, the magnitude of improvement varies with the informed player's strength as well as the game parameters. Our analysis highlights the interplay between strength and information in adversarial environments.


The authors of #REFR provide theoretical evidence for a strong relationship between small curvature and large robustness, proposing an efficient regularizer that encourages small curvatures and leads to significant boosts in neural network robustness.

Several recent works (e.g., [8, 12, 15, 19, 21-24, 31, 33, 36, 37, 50, 79, 81, 83] ) have explored the issue of deep learning model robustness from an image or text perspective.

In #OTHEREFR , the authors analyze model robustness by adding noise or perturbations to images and observing their impact on predicted results.

While most existing works focus on adding noise to the image input, our work instead focuses on adding noise to the text input #OTHEREFR .

Specifically, we consider the semantically related BQs of a given MQ as a type of noise for the MQ, using these BQs to evaluate the robustness of VQA models.

Improving Visual Question Answering Models Through Robustness Analysis And In-Context Learning With A Chain Of Basic Questions

Abstract: Deep neural networks have been critical in the task of Visual Question Answering (VQA), with research traditionally focused on improving model accuracy. Recently, however, there has been a trend towards evaluating the robustness of these models against adversarial attacks. This involves assessing the accuracy of VQA models under increasing levels of noise in the input, which can target either the image or the proposed query question, dubbed the main question. However, there is currently a lack of proper analysis of this aspect of VQA. This work proposes a new method that utilizes semantically related questions, referred to as basic questions, acting as noise to evaluate the robustness of VQA models. It is hypothesized that as the similarity of a basic question to the main question decreases, the level of noise increases. To generate a reasonable noise level for a given main question, a pool of basic questions is ranked based on their similarity to the main question, and this ranking problem is cast as a optimization problem. Additionally, this work proposes a novel robustness measure, , and two basic question datasets to standardize the analysis of VQA model robustness. The experimental results demonstrate that the proposed evaluation method effectively analyzes the robustness of VQA models. Moreover, the experiments show that in-context learning with a chain of basic questions can enhance model accuracy.

Robustness Via Curvature Regularization, And Vice Versa

Abstract: State-of-the-art classifiers have been shown to be largely vulnerable to adversarial perturbations. One of the most effective strategies to improve robustness is adversarial training. In this paper, we investigate the effect of adversarial training on the geometry of the classification landscape and decision boundaries. We show in particular that adversarial training leads to a significant decrease in the curvature of the loss surface with respect to inputs, leading to a drastically more "linear" behaviour of the network. Using a locally quadratic approximation, we provide theoretical evidence on the existence of a strong relation between large robustness and small curvature. To further show the importance of reduced curvature for improving the robustness, we propose a new regularizer that directly minimizes curvature of the loss surface, and leads to adversarial robustness that is on par with adversarial training. Besides being a more efficient and principled alternative to adversarial training, the proposed regularizer confirms our claims on the importance of exhibiting quasi-linear behavior in the vicinity of data points in order to achieve robustness.


Note that G(n, p) may not be an α-expander for fixed α > 0: If edge probability p = (1 + )/n, then G(n, p) contains an induced path of length Ω(log n) with high probability #REFR .

Indeed, initialize S as the empty set and keep adding components S i (that were not added already) until S has size at least (1/2 − )n.

Note that S has cardinality no larger than n/2 and has at most αn/4 neighbors.

For sufficiently small, this is a contradiction to G being an α-expander.

Observe that the assertion in Lemma 4 applies also if G contains a sub-graph with Ω(n) nodes which is an α-expander. Proof of Theorem 11.

Nevertheless, it is known that if p ≥ (1 + )/n, then G(n, p) contains an α-expander with Ω(n) nodes with high probability (see #OTHEREFR ). Lemmas 4 and 5 complete the proof.

† Observe that as C, L > 0 the cost is always strictly positive.

Inoculation Strategies For Bounded Degree Graphs

Abstract: We analyze a game-theoretic abstraction of epidemic containment played on an undirected graph G: each player is associated with a node in G and can either acquire protection from a contagious process or risk infection. After decisions are made, an infection starts at a random node v and propagates through all unprotected nodes reachable from v. It is known that the price of anarchy (PoA) in n-node graphs can be as large as Θ(n). Our main result is a tight bound of order √ n∆ on the PoA, where ∆ is the maximum degree of the graph. We also study additional factors that can reduce the PoA, such as higher thresholds for contagion and varying the costs of becoming infected vs. acquiring protection.

The Mixing Time Of The Giant Component Of A Random Graph

Abstract: We show that the total variation mixing time of the simple random walk on the giant component of supercritical Gn,p and Gn,m is i¾?log2n. This statement was proved, independently, by Fountoulakis and Reed. Our proof follows from a structure result for these graphs which is interesting in its own right. We show that these graphs are "decorated expanders" - an expander glued to graphs whose size has constant expectation and exponential tail, and such that each vertex in the expander is glued to no more than a constant number of decorations. © 2014 Wiley Periodicals, Inc. Random Struct. Alg., 45, 383-407, 2014


Moreover, we also calculate LPIPS #REFR , a popular deep-learning-based similarity metric for a more comprehensive evaluation.

To reduce computational consumption while maintaining image content as much as possible, we decrease the latter size to 49 × 288 × 288 by cropping.

Compared Methods: We compare our model with three SR methods: #OTHEREFR Cubic interpolation, a mathematical method based on cubic polynomials; (2) IREM, a self-supervised SR method based on INR.

(3) SMORE #OTHEREFR , a deep learning method based on self super-resolution (SRR) #OTHEREFR .

Here Cubic interpolation is based on the scikit-image library of Python, while IREM #OTHEREFR and SMORE #OTHEREFR are reproduced following the original papers.

Evaluation Metrics: PSNR and SSIM #OTHEREFR , two widely-used image quality evaluation metrics, are computed.

The slice-by-slice strategy is employed to calculate LPIPS #OTHEREFR since it is designed for 2D images.

Specifically, we first extract the most central 30 2D slices from SR volumes along three orthogonal views (i.e., ten images per view).

Then, we calculate LPIPS #OTHEREFR based on the 2D slices and average them to obtain the final scores.

Self-Supervised Arbitrary Scale Super-Resolution Framework For Anisotropic Mri

Abstract: In this paper, we propose an efficient self-supervised arbitraryscale super-resolution (SR) framework to reconstruct isotropic magnetic resonance (MR) images from anisotropic MRI inputs without involving external training data. The proposed framework builds a training dataset using "in-the-wild" anisotropic MR volumes with arbitrary image resolution. We then formulate the 3D volume SR task as a SR problem for 2D image slices. The anisotropic volume's high-resolution (HR) plane is used to build the HR-LR image pairs for model training. We further adapt the implicit neural representation (INR) network to implement the 2D arbitrary-scale image SR model. Finally, we leverage the welltrained proposed model to up-sample the 2D LR plane extracted from the anisotropic MR volumes to their HR views. The isotropic MR volumes thus can be reconstructed by stacking and averaging the generated HR slices. Our proposed framework has two major advantages: (1) It only involves the arbitrary-resolution anisotropic MR volumes, which greatly improves the model practicality in real MR imaging scenarios (e.g., clinical brain image acquisition); (2) The INR-based SR model enables arbitrary-scale image SR from the arbitrary-resolution input image, which significantly improves model training efficiency. We perform experiments on a simulated public adult brain dataset and a real collected 7T brain dataset. The results indicate that our current framework greatly outperforms two well-known self-supervised models for anisotropic MR image SR tasks.

The Unreasonable Effectiveness Of Deep Features As A Perceptual Metric

Abstract: While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on the ImageNet classification task has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called"perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new Full Reference Image Quality Assessment (FR-IQA) dataset of perceptual human judgments, orders of magnitude larger than previous datasets. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by huge margins. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.


The texts are a subset of the NoReC dataset #REFR , a multi-domain dataset of full-text professional reviews published in Norwegian online news sources.

NorQuAD A Norwegian question answering dataset for machine reading comprehension #OTHEREFR based on the SQuAD format #OTHEREFR .

For a given question, the model must predict the correct span in a provided passage that answers the question.

NorQuAD includes three domain splits: one sourced from the Norwegian Wikipedia (N = 2351), one from Norwegian news articles (N = 2398), and one split that combines both of them (N = 4749).

We use an 80 − 10 − 10 split on all three domains for training, validation, and testing.

NoReC fine A fine-grained sentiment analysis dataset for Norwegian (Øvrelid et al., 2020) .

Each sentence in NoReC fine is annotated for sentiment holders, targets, polar expressions, expression polarities, and polar intensities.

A version for targeted sentiment analysis (TSA) is released on GitHub where only the sentiment targets are labeled. 4

NoReC sent A sentence-level sentiment analysis dataset for Norwegian derived from NoReC fine #OTHEREFR .

This dataset is generated by aggregating the entity sentiments in each sentence.

The sentences are then labeled as either positive, negative, or neutral.

Brent: Bidirectional Retrieval Enhanced Norwegian Transformer

Abstract: Retrieval-based language models are increasingly employed in questionanswering tasks. These models search in a corpus of documents for relevant information instead of having all factual knowledge stored in its parameters, thereby enhancing efficiency, transparency, and adaptability. We develop the first Norwegian retrieval-based model by adapting the REALM framework and evaluate it on various tasks. After training, we also separate the language model, which we call the reader, from the retriever components, and show that this can be fine-tuned on a range of downstream tasks. Results show that retrieval augmented language modeling improves the reader's performance on extractive question-answering, suggesting that this type of training improves language models' general ability to use context and that this does not happen at the expense of other abilities such as part-of-speech tagging, dependency parsing, named entity recognition, and lemmatization. Code, trained models, and data are made publicly available. 1 * The authors contributed equally to this work 1 https://github.com/ltgoslo/brent query Gauss regnet ut den nøyaktige posisjonen hvor en kunne forvente å observere [MASK] igjen MLM Reader Database query encoder documents Retriever Allerede som barn viste Gauss seg som svaert begavet ... ledet Gauss til studiet av dvergplaneters bevegelse Gauss ble professor i astronomi i 1807 og var direktør for observatoriet i.

Norec: The Norwegian Review Corpus

Abstract: This paper presents the Norwegian Review Corpus (NoReC), created for training and evaluating models for document-level sentiment analysis. The full-text reviews have been collected from major Norwegian news sources and cover a range of different domains, including literature, movies, video games, restaurants, music and theater, in addition to product reviews across a range of categories. Each review is labeled with a manually assigned score of 1-6, as provided by the rating of the original author. This first release of the corpus comprises more than 35,000 reviews. It is distributed using the CoNLL-U format, pre-processed using UDPipe, along with a rich set of metadata. The work reported in this paper forms part of the SANT initiative (Sentiment Analysis for Norwegian Text), a project seeking to provide resources and tools for sentiment analysis and opinion mining for Norwegian. As resources for sentiment analysis have so far been unavailable for Norwegian, NoReC represents a highly valuable and sought-after addition to Norwegian language technology.


Moreover, the differences between the results of our model and others are significant. Table 5 . CR and SCR results on FashionGen #REFR .

FashionSAP gets the SOTA performance as the comparable results shown in Tab. 2.

We report the average result of 5 randomly chosen retrieval test sets and each of them contains 1k queries by following previous works.

For each query in test sets, only one candidate is matched (positive), while the other 100 candidates are mismatched (negative) and chosen from the same subcategory.

For the T2I task, there are 101 candidate images for each query text, and only one image in candidates is matched.

In order to test the performance of our model thoroughly, we also evaluate our model in the full test set of FashionGen #OTHEREFR in Tab. 3 following #OTHEREFR . Our model also gets the SOTA performance.

Fashionsap: Symbols And Attributes Prompt For Fine-Grained Fashion Vision-Language Pre-Training

Abstract: Fashion vision-language pre-training models have shown efficacy for a wide range of downstream tasks. However, general vision-language pre-training models pay less attention to fine-grained domain features, while these features are important in distinguishing the specific domain tasks from general tasks. We propose a method for fine-grained fashion vision-language pre-training based on fashion Symbols and Attributes Prompt (FashionSAP) to model fine-grained multi-modalities fashion attributes and characteristics. Firstly, we propose the fashion symbols, a novel abstract fashion concept layer, to represent different fashion items and to generalize various kinds of finegrained fashion features, making modelling fine-grained attributes more effective. Secondly, the attributes prompt method is proposed to make the model learn specific attributes of fashion items explicitly. We design proper prompt templates according to the format of fashion data. Comprehensive experiments are conducted on two public fashion benchmarks, i.e., FashionGen and FashionIQ, and Fash-ionSAP gets SOTA performances for four popular fashion tasks. The ablation study also shows the proposed abstract fashion symbols, and the attribute prompt method enables the model to acquire fine-grained semantics in the fashion domain effectively. The obvious performance gains from FashionSAP provide a new baseline for future fashion task research.

Fashion-Gen: The Generative Fashion Dataset And Challenge

Abstract: We introduce a new dataset of 293,008 high definition (1360 x 1360 pixels) fashion images paired with item descriptions provided by professional stylists. Each item is photographed from a variety of angles. We provide baseline results on 1) high-resolution image generation, and 2) image generation conditioned on the given text descriptions. We invite the community to improve upon these baselines. In this paper, we also outline the details of a challenge that we are launching based upon this dataset.


Most tracking-based networks first do single-frame object detection and use the association for the detection reciprocally #REFR .

While memory networks, such as STDnet-ST #OTHEREFR , achieve high accuracies, their heavy architectures prohibit their deployment on embedded devices.

Optical flow-based networks are faster, but their benefit over single-frame object detectors is limited #OTHEREFR . Figure 2 .

Here, we have that h = 20m and β = 15 • resulting in y = 110m and x = 25m.

This means, that the swimmer is 110m in front and 25m to the right of the UAV as measured on the ground.

Note that we also estimate the horizon line as indicated by the blue line.

In fact, the best two VOD models of the last VisDrone-VID challenge were single-frame methods, entirely ignoring the temporal domain #OTHEREFR .

Furthermore, all common video object detectors on UAVs ignore the underlying 3D geometry of the scene and only operate in image space #OTHEREFR .

While there is much research focusing on geolocation, it is only focused on obtaining world coordinates of objects for downstream tasks, such as following a target #OTHEREFR .

Similarly, occupancy networks and other mapping approaches aim at obtaining a map for mapping or scene understanding #OTHEREFR .

As opposed to these works, we aim to leverage metadata to build a memory map in GPS space that improves the video object detection performance.

Memory Maps For Video Object Detection And Tracking On Uavs

Abstract: This paper introduces a novel approach to video object detection detection and tracking on Unmanned Aerial Vehicles (UAVs). By incorporating metadata, the proposed approach creates a memory map of object locations in actual world coordinates, providing a more robust and interpretable representation of object locations in both, image space and the real world. We use this representation to boost confidences, resulting in improved performance for several temporal computer vision tasks, such as video object detection, short and long-term single and multi-object tracking, and video anomaly detection. These findings confirm the benefits of metadata in enhancing the capabilities of UAVs in the field of temporal computer vision and pave the way for further advancements in this area.

Detect Or Track: Towards Cost-Effective Video Object Detection/Tracking

Abstract: State-of-the-art object detectors and trackers are developing fast. Trackers are in general more efficient than detectors but bear the risk of drifting. A question is hence raised – how to improve the accuracy of video object detection/tracking by utilizing the existing detectors and trackers within a given time budget? A baseline is frame skipping – detecting every N-th frames and tracking for the frames in between. This baseline, however, is suboptimal since the detection frequency should depend on the tracking quality. To this end, we propose a scheduler network, which determines to detect or track at a certain frame, as a generalization of Siamese trackers. Although being light-weight and simple in structure, the scheduler network is more effective than the frame skipping baselines and flow-based approaches, as validated on ImageNet VID dataset in video object detection/tracking.


Therefore, #REFR argues for constraining the learnt policy to take actions in the support of the behaviour policy.

this can be highly restrictive and result in suboptimal behaviour.

However, support constraints generally require approximations to implement #OTHEREFR .

Therefore, subsequent works have shown that other techniques for ensuring that the policy does not leave the data distribution work better in practice #OTHEREFR .

Another approach is to incorporate conservatism into the value function update to prevent overestimation of the value of state-action pairs that are not present in the dataset #OTHEREFR .

For example, the value function update for CQL #OTHEREFR penalises the Q-values of state-action pairs that are not contained in the dataset.

Another line of work uses uncertainty quantification to obtain more robust value estimates #OTHEREFR .

Risk-Sensitive And Robust Model-Based Reinforcement Learning And Planning

Abstract: Goal-Oriented Long-Lived Systems Group (GOALS) Department of Engineering Science I'm deeply grateful to my supervisors Nick and Bruno. Thank you for being fantastic mentors, and for always being generous with your time. This thesis would not have been possible without your guidance, knowledge, enthusiasm, and support. I greatly appreciate the insightful feedback of my examiners, Jakob Foerster and Aviv Tamar, which has helped to significantly improve the final version of this thesis. Many thanks to Benjamin Morrell, KC Wong, Rob Reid, and David Willson, for mentoring me when I first became interested in research. Thank you for having the patience and generosity to assist an enthusiastic undergraduate student. I would like to extend my thanks to the members of the GOALS group that have been here throughout my DPhil: Charlie,

Stabilizing Off-Policy Q-Learning Via Bootstrapping Error Reduction

Abstract: Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q-learning and actor-critic methods are highly sensitive to the data distribution, and can make only limited progress without collecting additional on-policy data. As a step towards more robust off-policy algorithms, we study the setting where the off-policy experience is fixed and there is no further interaction with the environment. We identify bootstrapping error as a key source of instability in current methods. Bootstrapping error is due to bootstrapping from actions that lie outside of the training data distribution, and it accumulates via the Bellman backup operator. We theoretically analyze bootstrapping error, and demonstrate how carefully constraining action selection in the backup can mitigate it. Based on our analysis, we propose a practical algorithm, bootstrapping error accumulation reduction (BEAR). We demonstrate that BEAR is able to learn robustly from different off-policy distributions, including random and suboptimal demonstrations, on a range of continuous control tasks.


Original image-level FPN (i-FPN) #REFR introduces a topdown path to propagate contextual semantic information from higher layers to lower ones.

Actually, the lower-level features contain more fine-grained details than higher-level ones, which are beneficial for high-quality segmentation, especially on boundary regions, but these information is not fully explored in Mask R-CNN #OTHEREFR .

In this work, we propose a region-level FPN (r-FPN) to integrate more detailed information from lower layers of i-FPN into region-wise feature hierarchies.

The information flows from each level of i-FPN to r-FPN are shown in Fig. 2 .

We follow the original i-FPN to define the layers producing feature maps of the same resolution as one network stage corresponding to one feature level.

Dynamask: Dynamic Mask Selection For Instance Segmentation

Abstract: The representative instance segmentation methods mostly segment different object instances with a mask of the fixed resolution, e.g., 28 × 28 grid. However, a lowresolution mask loses rich details, while a high-resolution mask incurs quadratic computation overhead. It is a challenging task to predict the optimal binary mask for each instance. In this paper, we propose to dynamically select suitable masks for different object proposals. First, a dual-level Feature Pyramid Network (FPN) with adaptive feature aggregation is developed to gradually increase the mask grid resolution, ensuring high-quality segmentation of objects. Specifically, an efficient region-level top-down path (r-FPN) is introduced to incorporate complementary contextual and detailed information from different stages of image-level FPN (i-FPN). Then, to alleviate the increase of computation and memory costs caused by using large masks, we develop a Mask Switch Module (MSM) with negligible computational cost to select the most suitable mask resolution for each instance, achieving high efficiency while maintaining high segmentation accuracy. Without bells and whistles, the proposed method, namely DynaMask, brings consistent and noticeable performance improvements over other state-ofthe-arts at a moderate computation overhead. The source code: https://github.com/lslrh/DynaMask.

Feature Pyramid Networks For Object Detection

Abstract: Feature pyramids are a basic component in recognition systems for detecting objects at different scales. But pyramid representations have been avoided in recent object detectors that are based on deep convolutional networks, partially because they are slow to compute and memory intensive. In this paper, we exploit the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed for building high-level semantic feature maps at all scales. This architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using a basic Faster R-CNN system, our method achieves state-of-the-art single-model results on the COCO detection benchmark without bells and whistles, surpassing all existing single-model entries including those from the COCO 2016 challenge winners. In addition, our method can run at 5 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. Code will be made publicly available.


They retrained the BIDAF model #REFR on these generated adversarial examples to test its robustness.

The use of adversarial examples to evaluate and improve the robustness of machine learning models has a long-standing history #OTHEREFR .

In the field of NLP, one of the early works by #OTHEREFR showed that despite existing neural network QA systems proving their success when evaluated on standard metrics, they perform poorly when evaluated on adversarial examples.

In their work, they propose the creation of adversarial examples for SQuAD v1.1 using the AddSent and AddAny algorithms.

In AddSent, a distractor sentence is appended at the end of each context.

For the AddAny algorithm, a random sequence of grammatical and ungrammatical words are appended to each context.

AddSent algorithm swaps namedentities and numbers in the question with the nearest word in GloVe word vector space #OTHEREFR . Our method, EntSwap, slightly differs from AddSent. We replace named-entities, while leaving numbers unchanged.

Unlike AddSent, which replaces entities in the question, appends distracting sentences to the context, and leaves the answers unchanged, EntSwap replaces all detected named-entities for the questions, context, and answers to create an altered SQuAD2.0 dev set.

Although the BIDAF model #OTHEREFR was retrained on adversarial examples, there is no guarantee of its robustness when evaluated on adversarial examples generated differently.

Wang and Bansal #OTHEREFR ) generated slightly different adversarial examples for SQuAD using the AddSentMod algorithm by prepending the distractor sentences to the context instead of appending them and also used a different set of fake answers from ADD-SENT.

The authors show that the pre-trained BiDAF model #OTHEREFR is not robust to this set of adversarial examples as the model's F1-score drops by 30%.

Evaluating The Robustness Of Machine Read-Ing Comprehension Models To Low Resource Entity Renaming

Abstract: Question answering (QA) models have shown compelling results in the task of Machine Reading Comprehension (MRC). Recently these systems have proved to perform better than humans on held-out test sets of datasets e.g. SQuAD, but their robustness is not guaranteed. The QA model's brittleness is exposed when evaluated on adversarial generated examples by a performance drop. In this study, we explore the robustness of MRC models to entity renaming, with entities from low resource regions such as Africa. We propose EntSwap, a method for test-time perturbations, to create a test set whose entities have been renamed. In particular, we rename entities of type: country, person, nationality, location, organization and city, to create AfriSQuAD2. Using the perturbed test set, we evaluate the robustness of three popular MRC models. We find that compared to base models, large models perform well comparatively on novel entities. Furthermore, our analysis indicate that person, as an entity type, highly challenges the model performance.

Bidirectional Attention Flow For Machine Comprehension

Abstract: Machine comprehension (MC), answering a query about a given context paragraph, requires modeling complex interactions between the context and the query. Recently, attention mechanisms have been successfully extended to MC. Typically these methods use attention to focus on a small portion of the context and summarize it with a fixed-size vector, couple attentions temporally, and/or often form a uni-directional attention. In this paper we introduce the Bi-Directional Attention Flow (BIDAF) network, a multi-stage hierarchical process that represents the context at different levels of granularity and uses bi-directional attention flow mechanism to obtain a query-aware context representation without early summarization. Our experimental evaluations show that our model achieves the state-of-the-art results in Stanford Question Answering Dataset (SQuAD) and CNN/DailyMail cloze test.

generated adversarial examples
BIDAF model


Meta-World: Meta-World #REFR is a collection of 50 tasks, each set in its own environment involving a Sawyer arm. Meta-World provides several task splits.

We consider the ML50 split, which consists of 45 training tasks and 5 target tasks: door-lock, door-unlock, hand-insert, bin-picking, and box-close.

These 5 target tasks serve for evaluating various PLEX models.

We use Meta-World's v2 version with image observations, which is described in more detail in Appendix C.

Meta-World comes with high-quality scripted policies for all tasks.

To get video demonstration data (D mtvd ), we use these scripted policies to generate 100 successful video-only demonstrations for each of the 45 pretraining tasks. Therefore, |D mtvd | = 4500 trajectories.

Plex: Making The Most Of The Available Data For Robotic Manipulation Pretraining

Abstract: A rich representation is key to general robotic manipulation, but existing model architectures require a lot of data to learn it. Unfortunately, ideal robotic manipulation training data, which comes in the form of expert visuomotor demonstrations for a variety of annotated tasks, is scarce. In this work we propose PLEX, a transformer-based architecture that learns from task-agnostic visuomotor trajectories accompanied by a much larger amount of taskconditioned object manipulation videos-a type of robotics-relevant data available in quantity. The key insight behind PLEX is that the trajectories with observations and actions help induce a latent feature space and train a robot to execute task-agnostic manipulation routines, while a diverse set of video-only demonstrations can efficiently teach the robot how to plan in this feature space for a wide variety of tasks. In contrast to most works on robotic manipulation pretraining, PLEX learns a generalizable sensorimotor multi-task policy, not just an observational representation. We also show that using relative positional encoding in PLEX's transformers further increases its data efficiency when learning from human-collected demonstrations. Experiments showcase PLEX's generalization on Meta-World benchmark and establish state-of-the-art performance in challenging Robosuite environments.

Meta-World: A Benchmark And Evaluation For Multi-Task And Meta Reinforcement Learning

Abstract: Meta-reinforcement learning algorithms can enable robots to acquire new skills much more quickly, by leveraging prior experience to learn how to learn. However, much of the current research on meta-reinforcement learning focuses on task distributions that are very narrow. For example, a commonly used meta-reinforcement learning benchmark uses different running velocities for a simulated robot as different tasks. When policies are meta-trained on such narrow task distributions, they cannot possibly generalize to more quickly acquire entirely new tasks. Therefore, if the aim of these methods is to enable faster acquisition of entirely new behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new behaviors. In this paper, we propose an open-source simulated benchmark for meta-reinforcement learning and multi-task learning consisting of 50 distinct robotic manipulation tasks. Our aim is to make it possible to develop algorithms that generalize to accelerate the acquisition of entirely new, held-out tasks. We evaluate 6 state-of-the-art meta-reinforcement learning and multi-task learning algorithms on these tasks. Surprisingly, while each task and its variations (e.g., with different object positions) can be learned with reasonable success, these algorithms struggle to learn with multiple tasks at the same time, even with as few as ten distinct training tasks. Our analysis and open-source environments pave the way for future research in multi-task learning and meta-learning that can enable meaningful generalization, thereby unlocking the full potential of these methods.


More recently, #REFR introduces an attention-based mechanism, called Transformers, to replace the former RNN-based models.

In this section, we will review some basic concepts from language modeling.

Many architectures have been proposed to be used as LMs in various applications.

#OTHEREFR introduced Long Short-Term Memory recurrent neural networks to generate real-valued sequences with long-range structures to predict the next token.

#OTHEREFR introduced an encoding-decoding approach using an attention mechanism for machine translation systems.

All the large-scale NLP algorithms are using transformers, like BERT #OTHEREFR , T5 #OTHEREFR , GPT2 #OTHEREFR , GPT3 #OTHEREFR .

We will skip the details of the architectures and refer you to the mentioned works.

On The Risks Of Stealing The Decoding Algorithms Of Language Models

Abstract: A key component of generating text from modern language models (LM) is the selection and tuning of decoding algorithms. These algorithms determine how to generate text from the internal probability distribution generated by the LM. The process of choosing a decoding algorithm and tuning its hyperparameters takes significant time, manual effort, and computation, and it also requires extensive human evaluation. Therefore, the identity and hyperparameters of such decoding algorithms are considered to be extremely valuable to their owners. In this work, we show, for the first time, that an adversary with typical API access to an LM can steal the type and hyperparameters of its decoding algorithms at very low monetary costs. Our attack is effective against popular LMs used in text generation APIs, including GPT-2 and GPT-3. We demonstrate the feasibility of stealing such information with only a few dollars, e.g.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

former RNN-based models
attention-based mechanism


For text data, in addition to BERT-base, we also consider two BERT variants: RoBERTa and Distill-BERT #REFR .

Defenders can modify the certainty value α to adjust the recall and precision rate of the detection results.

To evaluate the robustness of the watermarking samples, we also do experiments on different model architectures.

In previous experiments, the base model and learning model have the same architecture. Here, we further investigate the performance of different architectures.

Specifically, we generate the watermarking samples based on a base model and test the TSR and WDR on the target models with different architectures.

For image datasets besides ResNet, we select two commonly used models: .

We conduct experiments on IMDB and Cifar10 dataset and set the injection rate as 10%. Results are shown in Tab. 5.

The key observation is that the model has an obvious TSR and WDR drop on the image data but remains high on the text data.

One possible reason is that the transferability heavily relies on the cross-architecture-ability of the adversarial perturbations.

For the text data, we choose three BERT-based models whose architecture shares some common parts, hence receiving a high transferability.

Did You Train On My Dataset? Towards Public Dataset Protection With Clean-Label Backdoor Watermarking

Abstract: The huge supporting training data on the Internet has been a key factor in the success of deep learning models. However, this abundance of public-available data also raises concerns about the unauthorized exploitation of datasets for commercial purposes, which is forbidden by dataset licenses. In this paper, we propose a backdoor-based watermarking approach that serves as a general framework for safeguarding publicavailable data. By inserting a small number of watermarking samples into the dataset, our approach enables the learning model to implicitly learn a secret function set by defenders. This hidden function can then be used as a watermark to track down third-party models that use the dataset illegally. Unfortunately, existing backdoor insertion methods often entail adding arbitrary and mislabeled data to the training set, leading to a significant drop in performance and easy detection by anomaly detection algorithms. To overcome this challenge, we introduce a clean-label backdoor watermarking framework that uses imperceptible perturbations to replace mislabeled samples. As a result, the watermarking samples remain consistent with the original labels, making them difficult to detect. Our experiments on text, image, and audio datasets demonstrate that the proposed framework effectively safeguards datasets with minimal impact on original task performance. We also show that adding just 1% of watermarking samples can inject a traceable watermarking function and that our watermarking samples are stealthy and look benign upon visual inspection.

Distilbert, A Distilled Version Of Bert: Smaller, Faster, Cheaper And Lighter

Abstract: As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.


The advantage of this underhanded construction is that now we can translate the empty word as the relative pronoun "who" or "that." This ties in perfectly with the Frobenius algebra approach of #REFR .

The relative pronoun "who" acts as a bridge that "copies" the subject into the relative clause and then transfers it back into the main clause.

We translate the following relative clause. We assign types in a less straightforward way.

We first insert an empty word between the modifier "tōkyō kara untensita" and the head "onna." We assign the following types: "tōkyō" and "onna" are both type n, the ablative particle "kara" has type n r o 7 , "kyō" has type t (temporal adverb), the verb "untensita" has type o r 7 t r so ℓ 1 and the empty word acts like a phantom relative pronoun with type o 1 s r nn ℓ . kyō tōkyō kara untensita

∅ onna t n n r o 7 o r 7 t r s o ℓ 1 o 1 s r n n ℓ n

This construction generates a noun phrase and it can be translated using a straightforward anti-homomorphism.

In this example we modelled our relative clause as what the authors of the reference call a subject relative clause. 4.5. Coordinate sentences.

The simplest way of coordinating sentences is by connecting them with the particle "ga" ("and") to which we assign the type s r ss ℓ . We translate the following sentence where subjects are omitted.

We decorate the pregroup with braces and assign the following type

n • n r o 5 • o r 5 s s r ss ℓ n • n r o 2 • o r 2 s .

Extending the morphism Ψ from Example 7 to monoids with 3-braces, we obtain

An Algebraic Approach To Translating Japanese

Abstract: We use Lambek's pregroups and the framework of compositional distributional models of language ("DisCoCat") to study translations from Japanese to English as pairs of functors. Adding decorations to pregroups we show how to handle word order changes between languages.

The Frobenius Anatomy Of Word Meanings I: Subject And Object Relative Pronouns

Abstract: This paper develops a compositional vector-based semantics of subject and object relative pronouns within a categorical framework. Frobenius algebras are used to formalise the operations required to model the semantics of relative pronouns, including passing information between the relative clause and the modified noun phrase, as well as copying, combining, and discarding parts of the relative clause. We develop two instantiations of the abstract semantics, one based on a truth-theoretic approach and one based on corpus statistics.


Bpc: Train-Time Calibration Loss For Detection

The remaining detections after populating t AC and t AN are false positives (inaccurate). Similar to Eq. #REFR and Eq. 7, we categorize them based on their confidence scores.

In principle, we need accurate detections to be more confident and inaccurate ones to be less confident, so we define the following objective that should be maximized:

In object detection, the obtained predictions are either accurate or inaccurate.

Given the predicted class label, bounding boxes, 1 as an indicator function, and th is the threshold on score, we define the following:

t AC = i 1[IoU(b i , b i ) ≥ ρ]1[ȳ i = y i ] |s i ≥ th (6) t AN = i 1[IoU(b i , b i ) ≥ ρ]1[ȳ i = y i ] |s i < th (7) t IC & t IN :

In our loss formulation, we consider precision since it includes true positives and false positives, for which we have confidence scores.

Whereas false negatives cannot be considered as they do not have confidence scores because of no detections. Since Eq.

(5) is not differentiable owing to the indicator functions for t AC , t AN , t IC and t IN , we formulate its differentiable version to approximate these quantities.

Let t AC , t AN , t IC and t IN be the approximations to t AC , t AN , t IC and t IN , respectively. We express the differentiable formulation as:

t AC = i∈ Ki=1 & si≥th s i tanh(s i ) t AN = i∈ Ki=1 & si<th s i (1 − tanh(s i )) t IC = i∈ Ki=0 & si≥th (1 −s i ) tanh(s i ) t IN = i∈ Ki=0 & si<th (1 −s i ) (1 − tanh(s i ))

Bridging Precision And Confidence: A Train-Time Loss For Calibrating Object Detection

Abstract: Deep neural networks (DNNs) have enabled astounding progress in several vision-based problems. Despite showing high predictive accuracy, recently, several works have revealed that they tend to provide overconfident predictions and thus are poorly calibrated. The majority of the works addressing the miscalibration of DNNs fall under the scope of classification and consider only in-domain predictions. However, there is little to no progress in studying the calibration of DNN-based object detection models, which are central to many vision-based safety-critical applications. In this paper, inspired by the train-time calibration methods, we propose a novel auxiliary loss formulation that explicitly aims to align the class confidence of bounding boxes with the accurateness of predictions (i.e. precision). Since the original formulation of our loss depends on the counts of true positives and false positives in a minibatch, we develop a differentiable proxy of our loss that can be used during training with other application-specific loss functions. We perform extensive experiments on challenging in-domain and out-domain scenarios with six benchmark datasets including MS-COCO, Cityscapes, Sim10k, and BDD100k. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios. Our source code and pre-trained models are available at https:// github.com/akhtarvision/bpc_calibration

Analyzing The Role Of Model Uncertainty For Electronic Health Records

Abstract: In medicine, both ethical and monetary costs of incorrect predictions can be significant, and the complexity of the problems often necessitates increasingly complex models. Recent work has shown that changing just the random seed is enough for otherwise well-tuned deep neural networks to vary in their individual predicted probabilities. In light of this, we investigate the role of model uncertainty methods in the medical domain. Using RNN ensembles and various Bayesian RNNs, we show that population-level metrics, such as AUC-PR, AUC-ROC, log-likelihood, and calibration error, do not capture model uncertainty. Meanwhile, the presence of significant variability in patient-specific predictions and optimal decisions motivates the need for capturing model uncertainty. Understanding the uncertainty for individual patients is an area with clear clinical impact, such as determining when a model decision is likely to be brittle. We further show that RNNs with only Bayesian embeddings can be a more efficient way to capture model uncertainty compared to ensembles, and we analyze how model uncertainty is impacted across individual input features and patient subgroups.


Clauset and Eagle introduced topological overlap as a similarity measure between two networks (the two networks can be consecutive snapshots of the same network) #REFR . Buttner et al.

Many metrics have been used with various temporal networks in analyzing network properties. Zhao et al.

used temporal centrality on a stock market based network as a portfolio selection tool #OTHEREFR .

They emphasized the importance of temporal attributes of the network for real-world applications. Prado et.

al determined the most relevant characters in a story by using eigenvector centrality over a temporal network of literary text #OTHEREFR . Cunha et al.

analyzed contact duration and inter-contact times in vehicular networks to compare original and calibrated mobility traces of GPS-equipped taxis #OTHEREFR .

adapted the metric for edge cases and used it on a real-world animal trade network #OTHEREFR . Navarro et al.

used topological overlap to measure embeddedness of a link to show correlation with tie persistence. #OTHEREFR .

Utilizing the methodical information gained from the aforementioned studies, we used a network of cities connected via linkages of multinational firms.

These linkages allow us to study stable and changing cities from the perspective of economic growth.

We use a temporal metric to examine networks of cities from 2010 to 2019 and use this data to identify the top stable and changing cities over the time period.

World City Networks And Multinational Firms: An Analysis Of Economic Ties Over A Decade

Abstract: One perspective to view the economic development of cities is through the presence of multinational firms; how subsidiaries of various organizations are set up throughout the globe and how cities are connected to each other through these networks of multinational firms. Analysis of these networks can reveal interesting economical and spatial trends, as well as help us understand the importance of cities in national and regional economic development. This paper aims to study networks of cities formed due to the linkages of multinational firms over a decade (from 2010 to 2019). More specifically we are interested in analyzing the growth and stability of various cities in terms of the connections they form with other cities over time. Our results can be summarized into two key findings: First, we ascertain the central position of several cities due to their economically stable connections; Second, we successfully identify cities that have evolved over the past decade as the presence of multinational firms has increased in these cities.

Persistence And Periodicity In A Dynamic Proximity Network

Abstract: The topology of social networks can be understood as being inherently dynamic, with edges having a distinct position in time. Most characterizations of dynamic networks discretize time by converting temporal information into a sequence of network "snapshots" for further analysis. Here we study a highly resolved data set of a dynamic proximity network of 66 individuals. We show that the topology of this network evolves over a very broad distribution of time scales, that its behavior is characterized by strong periodicities driven by external calendar cycles, and that the conversion of inherently continuous-time data into a sequence of snapshots can produce highly biased estimates of network structure. We suggest that dynamic social networks exhibit a natural time scale \Delta_{nat}, and that the best conversion of such dynamic data to a discrete sequence of networks is done at this natural rate.


Large language models (LLM) are the prototype of foundation models, which are trained on a large corpus collected from the internet with unsupervised tasks such as masked language modeling (MLM) #REFR .

Recently, foundation models have gained a lot of popularity in deep learning.

People use these huge pre-trained models with billions or even trillion parameters to solve downstream specific tasks such as machine translation and document classification #OTHEREFR .

Foundation models have been prevalent in natural language processing fields for a while in recent years, but it took years for this trend to deep vision learn-ing and its related fields.

Compared to LLMs, foundation models for vision tasks have two possible pre-training approaches.

The first approach is to reconstruct the input image using masked auto-encoders (MAE) #OTHEREFR directly, and the second is to introduce the pairing text descriptions of images as weak supervising labels.

The first approach has shown its extraordinary imaging understanding capability in many downstream vision tasks #OTHEREFR .

However, based on some recent research, the MAE method may not have strong domain generalization capability compared to the second approach.

Towards General Purpose Medical Ai: Continual Learning Medical Foundation Model

Abstract: * Equal contribution † Corresponding author 1 work in progress. Hippocampus (ADNI) Figure 1. Results on three different paradigms for implementing universal medical foundation models: specialized learning, joint learning, and continual learning. Continual learning may be a practical and efficient paradigm for the medical foundation model.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


E.g., human faces are roughly symmetric, and exploiting this as a prior yields considerable improvements #REFR .

This task is difficult, however, due to various forms of underdetermination, including depth ambiguity and occlusion.

The straighforward resolution to these problems is applying supervised learning with 2D/3D ground truth information, keypoints #OTHEREFR or depth maps #OTHEREFR .

However, it is hard and sometimes even impossible to collect such supervisory signals, as this requires considerable human effort.

Hence, recent techniques #OTHEREFR exploit unsupervised learning to leverage human prior knowledge about the natural objects in the world.

This alleviates the need for annotated data, but the prior has to be chosen wisely for each object class.

Besides reconstructing the 3D structure, one may be also interested in altering its style to one that is different from that of the original input image.

Consider, e.g., supporting -or even replacing -CG artists by automatically changing a 3D human model's gender, haircut, skin color or age.

Interestingly, Unsupervised 2D to Explicit 3D Style Transfer (UE3DST) is a less discussed problem in the literature.

By explicit, we mean that the method extracts a set of features, which allow recovering the 3D shape via using an explicit rendering function #OTHEREFR .

Unfortunately, existing methods are either (i) implicit 3D, or (ii) supervised.

Unsupervised Style-Based Explicit 3D Face Reconstruction From Single Image

Abstract: self style albedo depth shade norm 3D reconstruction Figure 1. Given a single 2D image, we reconstruct the 3D object (top) by predicting the features of the rendering process: albedo, depth, shading and surface normals. Then the method can synthesize images from an arbitrary viewpoint with a new style, potentially including changes in both shape and appearance (bottom).

Unsupervised Learning Of Probably Symmetric Deformable 3D Objects From Images In The Wild

Abstract: We propose a method to learn 3D deformable object categories from raw single-view images, without external supervision. The method is based on an autoencoder that factors each input image into depth, albedo, viewpoint and illumination. In order to disentangle these components without supervision, we use the fact that many object categories have, at least in principle, a symmetric structure. We show that reasoning about illumination allows us to exploit the underlying object symmetry even if the appearance is not symmetric due to shading. Furthermore, we model objects that are probably, but not certainly, symmetric by predicting a symmetry probability map, learned end-to-end with the other components of the model. Our experiments show that this method can recover very accurately the 3D shape of human faces, cat faces and cars from single-view images, without any supervision or a prior shape model. On benchmarks, we demonstrate superior accuracy compared to another method that uses supervision at the level of 2D image correspondences.


Compared to Waymo #REFR , nuScenes includes more object categories, covering different types of vehicles such as cars, trucks, buses, and trailers.

There are two difficulties: levels 1 and 2, where the bounding boxes contain at least five and one LiDAR points, respectively.

We use official metrics, including the mean Average Precision (mAP) and the mean Average Precision weighted by Heading (mAPH).

We benchmark multi-class 3D object detection using only single-frame LiDAR point clouds.

The IoU thresholds for vehicles, pedestrians, and cyclists are 0.7, 0.5, and 0.5.

nuScenes #OTHEREFR is another popular benchmark dataset with a total of 1,000 driving sequences.

We use the mean Average Precision (mAP) and the nuScenes Detection Score (NDS) to compare different baseline approaches.

Lidar-Based 3D Object Detection Via Hybrid 2D Semantic Scene Generation

Abstract: Figure 1: (a) We propose to integrate the 2D semantic scene representation into LiDAR-based 3D object detectors, which can be generated by either an explicit network (left) or an implicit network (middle) using the projection of 3D bounding boxes into the Bird's Eye View (right) as supervision. (b) The proposed 2D semantic scene representation can be easily integrated into most existing detectors (top) as a 2D semantic scene generation module (bottom) with minimal engineering efforts.

Scalability In Perception For Autonomous Driving: Waymo Open Dataset

Abstract: The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the overall viability of the technology. In an effort to help align the research community's contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span 20 seconds, consisting of well synchronized and calibrated high quality LiDAR and camera data captured across a range of urban and suburban geographies. It is 15x more diverse than the largest camera+LiDAR dataset available based on our proposed diversity metric. We exhaustively annotated this data with 2D (camera image) and 3D (LiDAR) bounding boxes, with consistent identifiers across frames. Finally, we provide strong baselines for 2D as well as 3D detection and tracking tasks. We further study the effects of dataset size and generalization across geographies on 3D detection methods. Find data, code and more up-to-date information at http://www.waymo.com/open.


We incorporate the proposed ASG mechanism into existing contrastive learning framework, i.e., SimCLR #REFR , to further enhance the discriminative power of the learned audio representation, as shown in Fig. 3 .

Unlike the previous methods that only generate positive pairs, we divide the data transformation strategies into two groups: (1) Positive sample generation, which modifies the audio stream while retain its semantic.

(2) Adversarial sample generation, which employs positional transformations on spectrogram to generate negative samples.

Then, the NT-Xent Loss proposed in SimCLR is employed to optimize the network.

The detailed descriptions of each part are given in the following sections.

Enhancing Unsupervised Audio Representation Learning Via Adversarial Sample Generation

Abstract: Existing audio analysis methods generally first transform the audio stream to spectrogram, and then feed it into CNN for further analysis. A standard CNN recognizes specific visual patterns over feature map, then pools for high-level representation, which overlooks the positional information of recognized patterns. However, unlike natural image, the semantic of an audio spectrogram is sensitive to positional change, as its vertical and horizontal axes indicate the frequency and temporal information of the audio, instead of naive rectangular coordinates. Thus, the insensitivity of CNN to positional change plays a negative role on audio spectrogram encoding. To address this issue, this paper proposes a new self-supervised learning mechanism, which enhances the audio representation by first generating adversarial samples (i.e., negative samples), then driving CNN to distinguish the embeddings of negative pairs in the latent space. Extensive experiments show that the proposed approach achieves best or competitive results on 9 downstream datasets compared with previous methods, which verifies its effectiveness on audio representation learning.

A Simple Framework For Contrastive Learning Of Visual Representations

Abstract: This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.

learned audio representation
existing contrastive learning


During training, incorporating 10% stochastic depth #REFR lead to a 0.102 dB improvement in PSNR.

Due to the limitation of the GPU resources, we do not further enlarge the window size and training patch size.

This shows that the rectangular training patch and larger local window size indeed can help improve the feature extraction ability across stereo images. Stochastic Depth.

As per the research conducted by #OTHEREFR , a deeper stochastic depth can improve the performance of stereoSR.

Therefore, we employ SwinFSR-L to examine how stochastic depth affects our Swin Transformer based model.

Our results based on the validation set of Flickr1024 #OTHEREFR are presented in Table 6 .

When using 20% stochastic depth, the performance of SwinFSR-L improves slightly by 0.1014 dB.

However, setting the stochastic depth to 30% results in a performance decrease of 0.121 dB, but it still outperforms the baseline.

This suggests that larger models have a tendency to overfit the Flickr1024 training data.

However, incorporating stochastic depth can help enhance the overall performance and generalization ability of the networks.

Swinfsr: Stereo Image Super-Resolution Using Swinir And Frequency Domain Knowledge

Abstract: Stereo Image Super-Resolution (stereoSR) has attracted significant attention in recent years due to the extensive deployment of dual cameras in mobile phones, autonomous vehicles and robots. In this work, we propose a new StereoSR method, named SwinFSR, based on an extension of SwinIR, originally designed for single image restoration, and the frequency domain knowledge obtained by the Fast Fourier Convolution (FFC). Specifically, to effectively gather global information, we modify the Residual Swin Transformer blocks (RSTBs) in SwinIR by explicitly incorporating the frequency domain knowledge using the FFC and employing the resulting residual Swin Fourier Transformer blocks (RSFTBs) for feature extraction. Besides, for the efficient and accurate fusion of stereo views, we propose a new cross-attention module referred to as RCAM, which achieves highly competitive performance while requiring less computational cost than the state-of-the-art cross-attention modules. Extensive experimental results and ablation studies demonstrate the effectiveness and efficiency of our proposed SwinFSR.

Deep Networks With Stochastic Depth

Abstract: Very deep convolutional networks with hundreds of layers have led to significant reductions in error on competitive benchmarks. Although the unmatched expressiveness of the many layers can be highly desirable at test time, training very deep networks comes with its own set of challenges. The gradients can vanish, the forward flow often diminishes, and the training time can be painfully slow. To address these problems, we propose stochastic depth, a training procedure that enables the seemingly contradictory setup to train short networks and use deep networks at test time. We start with very deep networks but during training, for each mini-batch, randomly drop a subset of layers and bypass them with the identity function. This simple approach complements the recent success of residual networks. It reduces training time substantially and improves the test error significantly on almost all data sets that we used for evaluation. With stochastic depth we can increase the depth of residual networks even beyond 1200 layers and still yield meaningful improvements in test error (4.91 % on CIFAR-10).