DL-based UDA approaches can broadly be split into three categories #REFR : discrepancybased, reconstruction-based, and adversarial.

Unsupervised Domain Adaptation (UDA): UDA aims to exploit the knowledge learned from a source dataset to help to create a discriminative model for a related but unlabelled target dataset #OTHEREFR .

Discrepancy based approaches aim to minimise a divergence criterion, which measures the distance between the source and target data distributions encoded in a learned feature space #OTHEREFR .

Reconstruction-based approaches, instead, use reconstruction as a proxy task to enable the learning of a shared representation for both image domains #OTHEREFR .

Finally, adversarial approaches deploy a discriminator that aims to identify the source of the data; the model is trained both to do the task and to trick the discriminator, creating domain invariant features #OTHEREFR .

These methods all assume simultaneous access to the source and target data, which poses data privacy challenges.

Federated Learning and Domain Adaptation: Federated learning (FL) has been proposed as a method to train models on distributed data #OTHEREFR .

Sfharmony: Source Free Domain Adaptation For Distributed Neuroimaging Analysis

Abstract: To represent the biological variability of clinical neuroimaging populations, it is vital to be able to combine data across scanners and studies. However, different MRI scanners produce images with different characteristics, resulting in a domain shift known as the 'harmonisation problem'. Additionally, neuroimaging data is inherently personal in nature, leading to data privacy concerns when sharing the data. To overcome these barriers, we propose an Unsupervised Source-Free Domain Adaptation (SFDA) method, SFHarmony. Through modelling the imaging features as a Gaussian Mixture Model and minimising an adapted Bhattacharyya distance between the source and target features, we can create a model that performs well for the target data whilst having a shared feature representation across the data domains, without needing access to the source data for adaptation or target labels. We demonstrate the performance of our method on simulated and real domain shifts, showing that the approach is applicable to classification, segmentation and regression tasks, requiring no changes to the algorithm. Our method outperforms existing SFDA approaches across a range of realistic data scenarios, demonstrating the potential utility of our approach for MRI harmonisation and general SFDA problems. Our code is available at https://github.com/ nkdinsdale/SFHarmony.

Do We Really Need To Access The Source Data? Source Hypothesis Transfer For Unsupervised Domain Adaptation

Abstract: Unsupervised domain adaptation (UDA) aims to leverage the knowledge learned from a labeled source dataset to solve similar tasks in a new unlabeled domain. Prior UDA methods typically require to access the source data when learning to adapt the model, making them risky and inefficient for decentralized private data. In this work we tackle a novel setting where only a trained source model is available and investigate how we can effectively utilize such a model without source data to solve UDA problems. To this end, we propose a simple yet generic representation learning framework, named \emph{Source HypOthesis Transfer} (SHOT). Specifically, SHOT freezes the classifier module (hypothesis) of the source model and learns the target-specific feature extraction module by exploiting both information maximization and self-supervised pseudo-labeling to implicitly align representations from the target domains to the source hypothesis. In this way, the learned target model can directly predict the labels of target data. We further investigate several techniques to refine the network architecture to parameterize the source model for better transfer performance. To verify its versatility, we evaluate SHOT in a variety of adaptation cases including closed-set, partial-set, and open-set domain adaptation. Experiments indicate that SHOT yields state-of-the-art results among multiple domain adaptation benchmarks.


The sequential algorithm for constructing PMFGs #REFR first sorts all of the edges by their edge weights.

We briefly describe the sequential construction of PMFGs and TMFGs, and introduce the notation and primitives that we use.

A planar graph can be drawn on the plane such that no edges cross each other.

Both PMFGs and TMFGs are maximal planar subgraphs of a complete undirected, edge-weighted graph ("maximal" means that no additional edge can be added to the vertex set without violating planarity).

They give an approximation to the NP-hard Weighted Maximum Planar Graph problem, which requires the sum of the edge weights kept to be maximized #OTHEREFR .

The complete edge-weighted graph input can also be viewed as a similarity matrix S, where S[i, j] is the weight of edge (i, j) in the graph. Planar Maximally Filtered Graph (PMFG).

It then starts with an empty graph G and considers each edge in the sorted order, from highest to lowest weight.

An edge is added to the graph if and only if it does not violate the planarity of G, which is checked by running a planarity testing algorithm.

This process is computationally expensive due to the need to run a planarity testing algorithm Θ(n 2 ) times. Triangulated Maximally Filtered Graph (TMFG).

TM-FGs #OTHEREFR have been proposed to improve the efficiency of PMFG construction.

Instead of considering edges one by one, the original sequential TMFG algorithm adds one vertex x to the vertex set of the subgraph under construction, as well as three edges (each from x to one of the three vertices of a triangular face in a planar embedding of the subgraph) on every iteration, and eliminates the vertex from consideration in future iterations.

Parallel Filtered Graphs For Hierarchical Clustering

Abstract: Given all pairwise weights (distances) among a set of objects, filtered graphs provide a sparse representation by only keeping an important subset of weights. Such graphs can be passed to graph clustering algorithms to generate hierarchical clusters. In particular, the directed bubble hierarchical tree (DBHT) algorithm on filtered graphs has been shown to produce good hierarchical clusters for time series data. We propose a new parallel algorithm for constructing triangulated maximally filtered graphs (TMFG), which produces valid inputs for DBHT, and a scalable parallel algorithm for generating DBHTs that is optimized for TMFG inputs. In addition to parallelizing the original TMFG construction, which has limited parallelism, we also design a new algorithm that inserts multiple vertices on each round to enable more parallelism. We show that the graphs generated by our new algorithm have similar quality compared to the original TMFGs, while being much faster to generate. Our new parallel algorithms for TMFGs and DBHTs are 136-2483x faster than state-of-the-art implementations, while achieving up to 41.56x self-relative speedup on 48 cores with hyper-threading, and achieve better clustering results compared to the standard average-linkage and complete-linkage hierarchical clustering algorithms. We show that on a data set, our algorithms produce clusters that align well with human experts' classification.

A Tool For Filtering Information In Complex Systems

Abstract: We introduce a technique to filter out complex data sets by extracting a subgraph of representative links. Such a filtering can be tuned up to any desired level by controlling the genus of the resulting graph. We show that this technique is especially suitable for correlation-based graphs, giving filtered graphs that preserve the hierarchical organization of the minimum spanning tree but containing a larger amount of information in their internal structure. In particular in the case of planar filtered graphs (genus equal to 0), triangular loops and four-element cliques are formed. The application of this filtering procedure to 100 stocks in the U.S. equity markets shows that such loops and cliques have important and significant relationships with the market structure and properties.


Our previous experiences on continual learning make us confident that this target is feasible: in particular, latent replay techniques #REFR seem to be very appropriate in this context because they allow to update the model by storing only a small subset of latent representation of past data.

In this work we introduced a region prediction technique, where map clustering is used to partition nodes into regions and a deep model is trained to predict the region that the robot is currently navigating.

This approach enables an efficient implementation of place recognition for realtime loop closure detection.

Our experiments validate the effectiveness of the idea but leave some issues open.

In fact, while the proposed technique could be already useful in some practical scenario where the predictor is retrained off-line on the whole accumulated map knowledge, its natural evolution is toward an on-line continual learning implementation as required in lifelong SLAM.

To this purpose, new regions and new instances (e.g., day/night variations) of already known regions need to be incrementally learned by the model without forgetting the old knowledge (a scenario known as NIC in the continual learning terminology #OTHEREFR ).

Region Prediction For Efficient Robot Localization On Large Maps

Abstract: Recognizing already explored places (a.k.a. place recognition) is a fundamental task in Simultaneous Localization and Mapping (SLAM) to enable robot relocalization and loop closure detection. In topological SLAM the recognition takes place by comparing a signature (or feature vector) associated to the current node with the signatures of the nodes in the known map. However, as the number of nodes increases, matching the current node signature against all the existing ones becomes inefficient and thwarts real-time navigation. In this paper we propose a novel approach to pre-select a subset of map nodes for place recognition. The map nodes are clustered during exploration and each cluster is associated with a region. The region labels become the prediction targets of a deep neural network and, during navigation, only the nodes associated with the regions predicted with high probability are considered for matching. While the proposed technique can be integrated in different SLAM approaches, in this work we describe an effective integration with RTAB-Map (a popular framework for real-time topological SLAM) which allowed us to design and run several experiments to demonstrate its effectiveness.

Latent Replay For Real-Time Continual Learning

Abstract: Training deep networks on light computational devices is nowadays very challenging. Continual learning techniques, where complex models are incrementally trained on small batches of new data, can make the learning problem tractable even for CPU-only edge devices. However, a number of practical problems need to be solved: catastrophic forgetting before anything else. In this paper we introduce an original technique named ``Latent Replay'' where, instead of storing a portion of past data in the input space, we store activations volumes at some intermediate layer. This can significantly reduce the computation and storage required by native rehearsal. To keep the representation stable and the stored activations valid we propose to slow-down learning at all the layers below the latent replay one, leaving the layers above free to learn at full pace. In our experiments we show that Latent Replay, combined with existing continual learning techniques, achieves state-of-the-art accuracy on a difficult benchmark such as CORe50 NICv2 with nearly 400 small and highly non-i.i.d. batches. Finally, we demonstrate the feasibility of nearly real-time continual learning on the edge through the porting of the proposed technique on a smartphone device.

continual learning
latent replay techniques


The first branch of the downsampling path in the proposed architecture consists of a U-Net #REFR encoder as shown in Figure 2 .

This branch contains 5 steps, each of them consisting of a 3 × 3 convolution operation, followed by batch normalization.

For incorporating non-linearity into the model, each output of the batch normalization is passed through a Rectified Linear Unit (ReLU) activation followed by a 2 × 2 MaxPooling operation.

This downsampling path progressively extracts image features and increases these feature dimensions in every layer.

Thus, ultimately a high-dimensional feature representation with high semantic information is obtained at the final layer.

Skip connections are introduced, enabling the fine-grained information from the encoder to be recovered in the upsampling half of the network.

Ut-Net: Combining U-Net And Transformer For Joint Optic Disc And Cup Segmentation And Glaucoma Detection

Abstract: Glaucoma is a chronic visual disease that may cause permanent irreversible blindness. Measurement of the cup-to-disc ratio (CDR) plays a pivotal role in the detection of glaucoma in its early stage, preventing visual disparities. Therefore, accurate and automatic segmentation of optic disc (OD) and optic cup (OC) from retinal fundus images is a fundamental requirement. Existing CNNbased segmentation frameworks resort to building deep encoders with aggressive downsampling layers, which suffer from a general limitation on modeling explicit long-range dependency. To this end, in this paper, we propose a new segmentation pipeline, called UT-Net, availing the advantages of UNet and transformer both in its encoding layer, followed by an attention-gated bilinear fusion scheme. In addition to this, we incorporate Multi-Head Contextual attention to enhance the regular self-attention used in traditional vision transformers. Thus low-level features along with global dependencies are captured in a shallow manner. Besides, we extract context information at multiple encoding layers for better exploration of receptive fields, and to aid the model to learn deep hierarchical representations. Finally, an enhanced mixing loss is proposed to tightly supervise the overall learning process. The proposed model has been implemented for joint OD and OC segmentation on three publicly available datasets: DRISHTI-GS, RIM-ONE R3, and REFUGE. Additionally, to validate our proposal, we have performed exhaustive experimentation on Glaucoma detection from all three datasets by measuring the Cup to Disc Ratio (CDR) value. Experimental results demonstrate the superiority of UT-Net as compared to the state-of-the-art methods.

U-Net: Convolutional Networks For Biomedical Image Segmentation

Abstract: There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .


To explore how the TCM block aggregates local and non-local information, we present effective receptive fields (ERF) #REFR of our model.

Based on the proposed TCM Block, the main path (g a and g s ) is designed as Fig. 2 shows.

The Residual Block with stride (RBS), Residual Block Upsampling (RBU) and subpixel conv3x3 are proposed by #OTHEREFR .

The detailed architectures of these three modules are reported in Supplementary.

In our framework, except for the last layer of g a and g s , we attach a TCM block after each RBS/RBU to obtain non-local and local information.

Also, we add TCM blocks into the hyper-prior path by re-designing the hyper-prior encoder h a and hyper-prior decoder h s as Fig. 4 shows.

Besides, the effective receptive fields of the transformer-based model #OTHEREFR and the CNNbased model #OTHEREFR are used to make comparisons.

ERF is defined as absolute gradients of a pixel in the output (i.e., | dxp dx |).

Here, we calculate gradients of the analysis point p = (70, 700) of kodim04 in Kodak dataset.

In order to estimate the importance of information at different distances, we clip gradients with two thresholds t (i.e., 0.01, 0.0001) to visualize them.

Note that the clip operation means we reduce the gradient values larger than the threshold to the threshold, and the gradient values smaller than the threshold remain unchanged. This can help our visualization has higher visibility precision. The visualization is shown in Fig. 3 .

Learned Image Compression With Mixed Transformer-Cnn Architectures

Abstract: Learned image compression (LIC) methods have exhibited promising progress and superior rate-distortion performance compared with classical image compression standards. Most existing LIC methods are Convolutional Neural Networks-based (CNN-based) or Transformer-based, which have different advantages. Exploiting both advantages is a point worth exploring, which has two challenges: 1) how to effectively fuse the two methods? 2) how to achieve higher performance with a suitable complexity? In this paper, we propose an efficient parallel Transformer-CNN Mixture (TCM) block with a controllable complexity to incorporate the local modeling ability of CNN and the non-local modeling ability of transformers to improve the overall architecture of image compression models. Besides, inspired by the recent progress of entropy estimation models and attention modules, we propose a channel-wise entropy model with parameter-efficient swin-transformer-based attention (SWAtten) modules by using channel squeezing. Experimental results demonstrate our proposed method achieves state-of-the-art rate-distortion performances on three different resolution datasets (i.e., Kodak, Tecnick, CLIC Professional Validation) compared to existing LIC methods. The code is at https://github.com/jmliu206/ LIC_TCM .

Understanding The Effective Receptive Field In Deep Convolutional Neural Networks

Abstract: We study characteristics of receptive fields of units in deep convolutional networks. The receptive field size is a crucial issue in many visual tasks, as the output must respond to large enough areas in the image to capture information about large objects. We introduce the notion of an effective receptive field size, and show that it both has a Gaussian distribution and only occupies a fraction of the full theoretical receptive field size. We analyze the effective receptive field in several architecture designs, and the effect of sub-sampling, skip connections, dropout and nonlinear activations on it. This leads to suggestions for ways to address its tendency to be too small.


However, as discussed in #REFR , the information in spatial and temporal domains is different in natural video data.

However, searching for a single 3D architecture requires days on multiple GPUs or TPUs, as training and evaluation of the accuracy indicator are required in the process, making the automatic 3D CNN design process time-consuming and/or hardware-dependent.

To tackle the above issues, we study how to automatically generate (or design) efficient and expressive 3D CNNs with limited computations.

Recently, training-free technologies have been introduced by some approaches #OTHEREFR , in which kernel spectrum analysis or forward inference are adopted to measure the expressiveness of spatial 2D CNNs.

Inspired by the training-free concept and information theory, we suggest that a deep network can be regarded as an information system, and measuring the expressiveness of the network can be considered equivalent to analyzing how much information it can capture.

Therefore, based on the Maximum Entropy Principle #OTHEREFR , the probability distribution of the system that best represents the current state of knowledge is the one with the highest entropy.

The spatial dimension is usually limited to some local properties, like connectivity #OTHEREFR , while the temporal dimension usually contains more drastic variations with more complex information.

To address the spatio-temporal discrepancy in video data, we conduct a kernel selection experiment and observe that different 3D kernel selections in different stages have different effects on performance, and the focus of 3D CNNs changes from spatial information to spatio-temporal information, as the network depth increases.

We thus consider that the design of 3D CNN architecture should focus on spatial-temporal aggregation depth-wisely.

The above analysis has motivated us to propose a training-free NAS approach to obtain optimal architectures, i.e., entropy-based 3D CNNs (E3D family).

Concretely, we first formulate a 3D CNN-based architecture as an information system whose expressiveness can be measured by the value of its differential entropy.

Maximizing Spatio-Temporal Entropy Of Deep 3D Cnns For Efficient Video Recognition

Abstract: 3D convolution neural networks (CNNs) have been the prevailing option for video recognition. To capture the temporal information, 3D convolutions are computed along the sequences, leading to cubically growing and expensive computations. To reduce the computational cost, previous methods resort to manually designed 3D/2D CNN structures with approximations or automatic search, which sacrifice the modeling ability or make training time-consuming. In this work, we propose to automatically design efficient 3D CNN architectures via a novel training-free neural architecture search approach tailored for 3D CNNs considering the model complexity. To measure the expressiveness of 3D CNNs efficiently, we formulate a 3D CNN as an information system and derive an analytic entropy score, based on the Maximum Entropy Principle. Specifically, we propose a spatio-temporal entropy score (STEntr-Score) with a refinement factor to handle the discrepancy of visual information in spatial and temporal dimensions, through dynamically leveraging the correlation between the feature map size and kernel size depth-wisely. Highly efficient and expressive 3D CNN architectures, i.e., entropy-based 3D CNNs (E3D family), can then be efficiently searched by maximizing the STEntr-Score under a given computational budget, via an evolutionary algorithm without training the network parameters. Extensive experiments on Something-Something V1&V2 and Kinetics400 demonstrate that the E3D family achieves state-of-the-art performance with higher computational efficiency. Code is available at https://github.com/alibaba/ lightweight-neural-architecture-search.

Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-Offs In Video Classification

Abstract: Despite the steady progress in video analysis led by the adoption of convolutional neural networks (CNNs), the relative improvement has been less drastic as that in 2D static image classification. Three main challenges exist including spatial (image) feature representation, temporal information representation, and model/computation complexity. It was recently shown by Carreira and Zisserman that 3D CNNs, inflated from 2D networks and pretrained on ImageNet, could be a promising way for spatial and temporal representation learning. However, as for model/computation complexity, 3D CNNs are much more expensive than 2D CNNs and prone to overfit. We seek a balance between speed and accuracy by building an effective and efficient video classification system through systematic exploration of critical network design choices. In particular, we show that it is possible to replace many of the 3D convolutions by low-cost 2D convolutions. Rather surprisingly, best result (in both speed and accuracy) is achieved when replacing the 3D convolutions at the bottom of the network, suggesting that temporal representation learning on high-level “semantic” features is more useful. Our conclusion generalizes to datasets with very different properties. When combined with several other cost-effective designs including separable spatial/temporal convolution and feature gating, our system results in an effective video classification system that that produces very competitive results on several action classification benchmarks (Kinetics, Something-something, UCF101 and HMDB), as well as two action detection (localization) benchmarks (JHMDB and UCF101-24).


The trend started with the introduction of ViT (Vision Transformers), which globally models non-overlapping spatial relationships in image patches using the standard Transformers encoder #REFR .

These outputs are combined to create beat-to-beat predictions of the EF and to predict the presence of heart failure.

This study uses echocardiography video dataset from EchoNet-Dynamic #OTHEREFR .

In other research #OTHEREFR which can also perform the task of predicting EF values by utilizing the Transformers based architecture that capable to process the videos of arbitrary duration.

The method uses a Transformers architecture based on the Residual Auto-Encoder (ResAE) Network and a BERT model adapted for token classification.

Based on Reynaud et al #OTHEREFR , it can be concluded that the backbone architecture modeling in computer vision (CV) has begun to shift to the use of the Transformers architecture.

For this research, we use Video Swin Transformers [16] , which completely follows the original Swin Transformers hierarchical structure #OTHEREFR .

However, we extend the local attention computation scope from the spatial domain to the spatio-temporal domain.

The adaptation process was carried out in the 3D patch partition section and replaced the local window self-attention module into a 3D shifted window based on multi-head self-attention (MSA) and shifted window multi-head self-attention (SW-MSA) in the Transformers Block section.

Video Swin Transformers can do the video-recognition tasks that contains an inductive bias towards spatio-temporal locality.

Hierarchical Vision Transformers For Cardiac Ejection Fraction Estimation

Abstract: The left ventricular of ejection fraction is one of the most important metric of cardiac function. It is used by cardiologist to identify patients who are eligible for lifeprolonging therapies. However, the assessment of ejection fraction suffers from inter-observer variability. To overcome this challenge, we propose a deep learning approach, based on hierarchical vision Transformers, to estimate the ejection fraction from echocardiogram videos. The proposed method can estimate ejection fraction without the need for left ventrice segmentation first, make it more efficient than other methods. We evaluated our method on EchoNet-Dynamic dataset resulting 5.59, 7.59 and 0.59 for MAE, RMSE and R 2 respectivelly. This results are better compared to the state-of-the-art method, Ultrasound Video Transformer (UVT).

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

Vision Transformers
standard Transformers encoder


Table 5 and 7 show the coarse-grained predicates and fine-grained predicates with the number of training instances for each predicate in the Visual Genome dataset #REFR .

In this work, we propose an automatic boosting framework CaCao that exploits linguistic knowledge from pretrained language models to enrich existing datasets in a lowresource way.

We tackle the long-tail issue of SGG with the help of abundant informative predicates from CaCao and generalize to open-world predicate learning with the entangled cross-modal prompt design based on VL models.

Our extensive experiments on three datasets illustrate the significant improvement of our CaCao on fine-grained scene graph generation and open-world generalization capability.

Table 6 and 8 show the coarse-grained predicates and finegrained predicates with the number of training instances for each predicate after cross-modal boosting by our CaCao.

We can observe that CaCao increases dataset scale, especially the tail predicates, which significantly alleviates the long-tail distribution problem in SGG.

For large-scale benchmark SGG, GQA #OTHEREFR contains 113K images and over 3.8M relation annotations.

In order to ensure the quality and representativeness of the dataset, we perform a manual cleaning process to remove annotations that had poor quality or ambiguous meanings following prior works #OTHEREFR .

We finally select the top 200 object classes and top 100 predicate classes as the GQA-200 split like VG-50 to explore the generalization ability of CaCao in large-scale SGG.

Visually-Prompted Language Model For Fine-Grained Scene Graph Generation In An Open World

Abstract: Scene Graph Generation (SGG) aims to extract <subject, predicate, object> relationships in images for vision understanding. Although recent works have made steady progress on SGG, they still suffer long-tail distribution issues that tail-predicates are more costly to train and hard to distinguish due to a small amount of annotated data compared to frequent predicates. Existing re-balancing strategies try to haddle it via prior rules but are still confined to pre-defined conditions, which are not scalable for various models and datasets. In this paper, we propose a Crossmodal prediCate boosting (CaCao) framework, where a visually-prompted language model is learned to generate diverse fine-grained predicates in a low-resource way. The proposed CaCao can be applied in a plug-and-play fashion and automatically strengthen existing SGG to tackle the long-tailed problem. Based on that, we further introduce a novel Entangled cross-modal prompt approach for open-world predicate scene graph generation (Epic), where models can generalize to unseen predicates in a zeroshot manner. Comprehensive experiments on three benchmark datasets show that CaCao consistently boosts the performance of multiple scene graph generation models in a model-agnostic way. Moreover, our Epic achieves competitive performance on open-world predicate prediction.

Visual Genome: Connecting Language And Vision Using Crowdsourced Dense Image Annotations

Abstract: Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in an image. When asked "What vehicle is the person riding?", computers will need to identify the objects in an image as well as the relationships riding(man, carriage) and pulling(horse, carriage) to answer correctly that "the person is riding a horse-drawn carriage." In this paper, we present the Visual Genome dataset to enable the modeling of such relationships. We collect dense annotations of objects, attributes, and relationships within each image to learn these models. Specifically, our dataset contains over 108K images where each image has an average of $$35$$35 objects, $$26$$26 attributes, and $$21$$21 pairwise relationships between objects. We canonicalize the objects, attributes, relationships, and noun phrases in region descriptions and questions answer pairs to WordNet synsets. Together, these annotations represent the densest and largest dataset of image descriptions, objects, attributes, relationships, and question answer pairs.


B.1 ; it could be due to a strong obfuscated gradient effect #REFR that cannot be detected by a naive straight-through estimator.

We also evaluate the nonquantized version of the quantizer (i.e., omitting the vector quantization process, but using the extracted feature by the encoder directly to the ViT input), but we empirically observe that attacking the non-quantized version cannot drop the performance at all.

Instead, we use the STE, also used during the training as well as the previous extensive robustness study #OTHEREFR .

We compare the attacked accuracies of DeiT and SeiT by varying ε (a control parameter for the attack intensity) from 0 to 8 in Fig. B.1 .

We observe that SeiT shows almost neglectable performance drops even under the strongest attack (showing 73.95 for ε = 8 where 73.98 for ε = 0), where DeiT shows 2.8% top-1 accuracy.

However, we should be careful to interpret Fig.

Moreover, our method could be vulnerable to the codebook attack by changing the token indices directly, not by perturbing the pixels.

However, as an efficient and natural ad- versarial attack on discrete domains is still an open problem #OTHEREFR (e.g., altering indices as imperceptible to humans but sensitive to machines -only a small index change can make a huge semantic gap, such as replacing "huge" in the previous sentence to "neglectable"), we leave the investigation of advanced adversarial attack methods for SeiT beyond straight-through estimator as future work.

Seit: Storage-Efficient Vision Training With Tokens Using 1% Of Pixel Storage

Abstract: We need billion-scale images to achieve more generalizable and groundbreaking vision models, as well as massive dataset storage to ship the images (e.g., the LAION-4B dataset needs 240TB storage space). However, it has become challenging to deal with unlimited dataset storage with limited storage infrastructure. A number of storageefficient training methods have been proposed to tackle the problem, but they are rarely scalable or suffer from severe damage to performance. In this paper, we propose a storage-efficient training strategy for vision classifiers for large-scale datasets (e.g., ImageNet) that only uses 1024 tokens per instance without using the raw level pixels; our token storage only needs <1% of the original JPEG-compressed raw pixels. We also propose token augmentations and a Stem-adaptor module to make our approach able to use the same architecture as pixel-based approaches with only minimal modifications on the stem layer and the carefully tuned optimization settings. Our experimental results on ImageNet-1k show that our method significantly outperforms other storage-efficient training methods with a large gap. We further show the effectiveness of our method in other practical scenarios, storage-efficient pre-training, and continual learning. Code is available at https://github.com/naver-ai/seit Adjusting image resolution Adjusting JPEG quality C-score based sampling Uniform random sampling Original DeiT SeiT SeiT (pixel fine-tuned) Storage (GB, log scale) Top 1 Accuracy (%)

Obfuscated Gradients Give A False Sense Of Security: Circumventing Defenses To Adversarial Examples

Abstract: We identify obfuscated gradients as a phenomenon that leads to a false sense of security in defenses against adversarial examples. While defenses that cause obfuscated gradients appear to defeat optimization-based attacks, we find defenses relying on this effect can be circumvented. ::: For each of the three types of obfuscated gradients we discover, we describe indicators of defenses exhibiting this effect and develop attack techniques to overcome it. In a case study, examining all defenses accepted to ICLR 2018, we find obfuscated gradients are a common occurrence, with 7 of 8 defenses relying on obfuscated gradients. Using our new attack techniques, we successfully circumvent all 7 of them.


As the most popular implicit representation in neural rendering, Neural Radiance Fields (NeRF) #REFR has exhibited stunning results in various tasks since its first introduction.

Recent years have witnessed the rapid development of neural implicit representations #OTHEREFR in 3D modeling and image synthesis.

Compared with classic discrete counterparts such as meshes, point clouds, and voxels, neural implicit representations model the scene with neural networks, which are spatially continuous and indicate a higher fidelity and flexibility.

The original NeRF overfits on one static scene by design, therefore it cannot model time-varying contents.

Many efforts have been made to adapt NeRF to dynamic scenes.

Some works condition the NeRF with local #OTHEREFR or global scene representations #OTHEREFR ] to implicitly provide generalizability for it.

As the pioneers, D-NeRF #OTHEREFR and Nerfies #OTHEREFR use an explicit deformation field to bend straight rays passing through varying targets into a common canonical scene, where a conventional NeRF is optimized.

Such a pipeline is adopted by many follow-up works #OTHEREFR .

Handnerf: Neural Radiance Fields For Animatable Interacting Hands

Abstract: Multi-view images Unified modeling and rendering for photo-realistic single / interacting hands under novel views / poses

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.


The top 11 of them produced PSNR scores higher than the baseline method LF-InterNet #REFR .

Among the 148 registered participants, 12 teams have successfully participated the final test phase and submitted their results, codes, and factsheets.

Table 1 reports the PSNR and SSIM scores achieved by these methods on both test and validation sets, together with their major details.

We briefly describe these solutions in Section 4, and introduce the corresponding team members in Appendix 6.

It can be observed from Table 1 that all these methods surpass the state-of-the-art method DistgSSR #OTHEREFR , and 9 of them surpass the recent top-performing method EPIT #OTHEREFR .

Note that, the winner solution proposed by the OpenMeow achieves around 1 dB improvement in PSNR over Dist-gSSR #OTHEREFR on both test and validation sets, which significantly push the state-of-the-art of LF image SR to a new height.

Moreover, the accuracy of the top 2 methods are very close with a minor PSNR difference of 0.02 dB on the test set.

Ntire 2023 Challenge On Light Field Image Super-Resolution: Dataset, Methods And Results

Abstract: In this report, we summarize the first NTIRE challenge on light field (LF) image super-resolution (SR), which aims at super-resolving LF images under the standard bicubic degradation with a magnification factor of 4. This challenge develops a new LF dataset called NTIRE-2023 for validation and test, and provides a toolbox called BasicLFSR to facilitate model development. Compared with single image SR, the major challenge of LF image SR lies in how to exploit complementary angular information from plenty of views with varying disparities. In total, 148 participants have registered the challenge, and 11 teams have successfully submitted results with PSNR scores higher than the baseline method LF-InterNet [1]. These newly developed methods have set new state-of-the-art in LF image SR, e.g., the winning method achieves around 1 dB PSNR improvement over the existing state-of-the-art method DistgSSR [2]. We report the solutions proposed by the participants, and summarize their common trends and useful tricks. We hope this challenge can stimulate future research and inspire new ideas in LF image SR.

Spatial-Angular Interaction For Light Field Image Super-Resolution

Abstract: Light field (LF) cameras record both intensity and directions of light rays, and capture scenes from a number of viewpoints. Both information within each perspective (i.e., spatial information) and among different perspectives (i.e., angular information) is beneficial to image super-resolution (SR). In this paper, we propose a spatial-angular interactive network (namely, LF-InterNet) for LF image SR. In our method, spatial and angular features are separately extracted from the input LF using two specifically designed convolutions. These extracted features are then repetitively interacted to incorporate both spatial and angular information. Finally, the interacted spatial and angular features are fused to super-resolve each sub-aperture image. Experiments on 6 public LF datasets have demonstrated the superiority of our method. As compared to existing LF and single image SR methods, our method can recover much more details, and achieves significant improvements over the state-of-the-arts in terms of PSNR and SSIM.


We discuss the application of the information bottleneck principle #REFR , connections between information theory and generalization, and recent information-theoretic learning algorithms.

In this paper, we aim to achieve two objectives.

First, we propose a unified framework that synthesizes existing research on self-supervised and semi-supervised learning from an information-theoretic standpoint.

This framework allows us to present and compare current methods, analyze their assumptions and difficulties, and discuss the optimal representation for neural networks in general and self-supervised networks in particular.

Second, we explore different methods and estimators for optimizing information-theoretic quantities in deep neural networks and investigate how recent models optimize various theoretical-information terms.

By reviewing the literature on various aspects of information-theoretic learning, we provide a comprehensive understanding of the interplay between information theory, self-supervised learning, and deep neural networks.

Furthermore, we examine how the information-theoretic perspective can offer insights into the design of better self-supervised learning algorithms and the potential benefits of using information theory in SSL across a wide range of applications.

In addition to the main structure of the paper, we dedicate a section to the challenges and opportunities in extending the information-theoretic perspective to other learning paradigms, such as energy-based models.

We highlight the potential advantages of incorporating these extensions into self-supervised learning algorithms and discuss the technical and conceptual challenges that need to be addressed.

The structure of the paper is as follows.

Section 2 introduces the key concepts in supervised, semi-supervised, and self-supervised learning, information theory, and representation learning.

To Compress Or Not To Compress -Self-Supervised Learning And Information Theory: A Review

Abstract: Deep neural networks have demonstrated remarkable performance in supervised learning tasks but require large amounts of labeled data. Self-supervised learning offers an alternative paradigm, enabling the model to learn from data without explicit labels. Information theory has been instrumental in understanding and optimizing deep neural networks. Specifically, the information bottleneck principle has been applied to optimize the trade-off between compression and relevant information preservation in supervised settings. However, the optimal information objective in self-supervised learning remains unclear. In this paper, we review various approaches to self-supervised learning from an information-theoretic standpoint and present a unified framework that formalizes the self-supervised informationtheoretic learning problem. We integrate existing research into a coherent framework, examine recent self-supervised methods, and identify research opportunities and challenges. Moreover, we discuss empirical measurement of information-theoretic quantities and their estimators. This paper offers a comprehensive review of the intersection between information theory, self-supervised learning, and deep neural networks.

The Information Bottleneck Method

Abstract: We define the relevant information in a signal $x\in X$ as being the information that this signal provides about another signal $y\in \Y$. Examples include the information that face images provide about the names of the people portrayed, or the information that speech sounds provide about the words spoken. Understanding the signal $x$ requires more than just predicting $y$, it also requires specifying which features of $\X$ play a role in the prediction. We formalize this problem as that of finding a short code for $\X$ that preserves the maximum information about $\Y$. That is, we squeeze the information that $\X$ provides about $\Y$ through a `bottleneck' formed by a limited set of codewords $\tX$. This constrained optimization problem can be seen as a generalization of rate distortion theory in which the distortion measure $d(x,\x)$ emerges from the joint statistics of $\X$ and $\Y$. This approach yields an exact set of self consistent equations for the coding rules $X \to \tX$ and $\tX \to \Y$. Solutions to these equations can be found by a convergent re-estimation method that generalizes the Blahut-Arimoto algorithm. Our variational principle provides a surprisingly rich framework for discussing a variety of problems in signal processing and learning, as will be described in detail elsewhere.


Carlini and Wagner #REFR demonstrated that defensive distillation is no more robust to adversity than unprotected neural networks.

This method, while showing good prospects in some areas of research, is subject to the problem of blind spots #OTHEREFR , which means that if adversarial examples of a certain attack family are not included in the learning phase, the classifier may fail to be robust to these adversarial examples.

Consequently, this implies that adversarial training could not be as effective against zero-day attacks, which are one of the key issues facing the intrusion detection sector.

Moreover, the IDS may perform worse in its original classification job as a result of being retrained with adversarial instances #OTHEREFR .

Defensive distillation, which was originally used to lower the dimensionality of DNNs, is another adversarial defense presented in the literature #OTHEREFR . Papernot et al.

#OTHEREFR suggest a distillation-based defense with the goal of smoothing the model's decision surface.

As such, this defense is broken using the C&W attack.

To defend against adversarial assaults, one might discard any features that could be modified by the attacker without affecting the logic of network communication #OTHEREFR . However, Apruzzese et al.

#OTHEREFR demonstrated that doing so has a negative influence on detector performance in non-adversarial conditions, for example, by generating more false alarms, which is extremely undesired for current cyber security platforms.

This decrease in performance is due to the deleted features having a major influence on the underlying processes of the baseline detectors.

Anomaly detection is an interesting alternative defensive mechanism against evasion attacks.

Adv-Bot: Realistic Adversarial Botnet Attacks Against Network Intrusion Detection Systems

Abstract: Due to the numerous advantages of machine learning (ML) algorithms, many applications now incorporate them. However, many studies in the field of image classification have shown that MLs can be fooled by a variety of adversarial attacks. These attacks take advantage of ML algorithms' inherent vulnerability. This raises many questions in the cybersecurity field, where a growing number of researchers are recently investigating the feasibility of such attacks against machine learning-based security systems, such as intrusion detection systems. The majority of this research demonstrates that it is possible to fool a model using features extracted from a raw data source, but it does not take into account the real implementation of such attacks, i.e., the reverse transformation from theory to practice. The real implementation of these adversarial attacks would be influenced by various constraints that would make their execution more difficult. As a result, the purpose of this study was to investigate the actual feasibility of adversarial attacks, specifically evasion attacks, against network-based intrusion detection systems (NIDS), demonstrating that it is entirely possible to fool these ML-based IDSs using our proposed adversarial algorithm while assuming as many constraints as possible in a black-box setting. In addition, since it is critical to design defense mechanisms to protect ML-based IDSs against such attacks, a defensive scheme is presented. Realistic botnet traffic traces are used to assess this work. Our goal is to create adversarial botnet traffic that can avoid detection while still performing all of its intended malicious functionality.

Defensive Distillation Is Not Robust To Adversarial Examples

Abstract: We show that defensive distillation is not secure: it is no more resistant to targeted misclassification attacks than unprotected neural networks.

unprotected neural networks
defensive distillation


The next result, which directly follows from #REFR , shows that the cumulative error due to the estimation of the transition probabilities grows sublinearly during time. Lemma 2.

The confidence set P i is updated at every epoch i as, ∀ (x, a) ∈ X × A:

and k(x) denotes the index of the layer to which x belongs and δ ∈ (0, 1) is the given confidence.

When the confidence set P is updated as in Equation 10, with probability at least 1 − 2δ, it holds:

T t=1 ||q t − q t || 1 ≤ E q δ , where E q δ ≤Õ( √ T ).

UC-O-GDPS.INIT procedure (Line 1 of Algorithm 2) initializes the epoch index as i = 1 and confidence set P 1 as the set of all possible transition functions.

For all k ∈ [0..L − 1] and all (x, a, x ′ ) ∈ X k × A× X k+1 , the counters are initialized as

A Best-Of-Both-Worlds Algorithm For Constrained Mdps With Long-Term Constraints

Abstract: We study online learning in episodic constrained Markov decision processes (CMDPs), where the goal of the learner is to collect as much reward as possible over the episodes, while guaranteeing that some long-term constraints are satisfied during the learning process. Rewards and constraints can be selected either stochastically or adversarially, and the transition function is not known to the learner. While online learning in classical (unconstrained) MDPs has received considerable attention over the last years, the setting of CMDPs is still largely unexplored. This is surprising, since in real-world applications, such as, e.g., autonomous driving, automated bidding, and recommender systems, there are usually additional constraints and specifications that an agent has to obey during the learning process. In this paper, we provide the first best-of-both-worlds algorithm for CMDPs with longterm constraints. Our algorithm is capable of handling settings in which rewards and constraints are selected either stochastically or adversarially, without requiring any knowledge of the underling process. Moreover, our algorithm matches state-of-the-art regret and constraint violation bounds for settings in which constraints are selected stochastically, while it is the first to provide guarantees in the case in which they are chosen adversarially.

Online Convex Optimization In Adversarial Markov Decision Processes

Abstract: We consider online learning in episodic loop-free Markov decision processes (MDPs), where the loss function can change arbitrarily between episodes, and the transition function is not known to the learner. We show $\tilde{O}(L|X|\sqrt{|A|T})$ regret bound, where $T$ is the number of episodes, $X$ is the state space, $A$ is the action space, and $L$ is the length of each episode. Our online algorithm is implemented using entropic regularization methodology, which allows to extend the original adversarial MDP model to handle convex performance criteria (different ways to aggregate the losses of a single episode) , as well as improve previous regret bounds.


Specifically, in this paper, we adopt the traditional teacher-student network #REFR to compose our model.

Update the teacher network's parameters from student network's parameters by the exponential moving average (EMA) strategy. their temporal context.

We find that simply applying consistency would cause the model to over-focus on the most discriminative parts of the video and affect the completeness of localization.

Hence we boost the semantic consistency constraint in a bidirectional manner to suppress co-scene actions while maintaining the stability and integrity of positive actions prediction.

Furthermore, considering that different transformations of the same action video will produce different predictions, combining multiple transforms of the same video to generate a comprehensive T-CAM always covers a more comprehensive part of actions than a single one.

Hence we also utilize the comprehensive background suppressed T-CAM to ensuring the integrity of positive actions.

The model proposed in this paper contains an original branch and an augmentation branch, both of which contain a teacher network and a student network.

Specifically, in this paper, the original branch includes a teacher-student network.

The student network of the original branch takes an original video feature X as input while the inputs of the teacher network of the original branch are multiple video features transformed by Intra-TCA in III-C1c.

In the teacher network of the original branch, K different inputs including I 1 (X), ..., I K (X) are fed into the same modules to get multiple background-suppressed T-CAMS T 1 , ...,S T K , where I() means the proposed intra-video temporal context augmentation strategy.

Then the comprehensive T-CAM generation module would generate comprehensive background suppressed T-CAM,

> Ieee Transactions On Neural Networks And Learning Systems <

Abstract: Weakly Supervised Temporal Action Localization (WTAL) aims to classify and localize temporal boundaries of actions for the video, given only video-level category labels in the training datasets. Due to the lack of boundary information during training, existing approaches formulate WTAL as a classification problem, i.e., generating the temporal class activation map (T-CAM) for localization. However, with only classification loss, the model would be sub-optimized, i.e., the action-related scenes are enough to distinguish different class labels. Regarding other actions in the action-related scene (i.e., the scene same as positive actions) as co-scene actions, this sub-optimized model would misclassify the co-scene actions as positive actions. To address this misclassification, we propose a simple yet efficient method, named bidirectional semantic consistency constraint (Bi-SCC), to discriminate the positive actions from co-scene actions. The proposed Bi-SCC firstly adopts a temporal context augmentation to generate an augmented video that breaks the correlation between positive actions and their co-scene actions in the intervideo; Then, a semantic consistency constraint (SCC) is used to enforce the predictions of the original video and augmented video to be consistent, hence suppressing the co-scene actions. However, we find that this augmented video would destroy the original temporal context. Simply applying the consistency constraint would affect the completeness of localized positive actions. Hence, we boost the SCC in a bidirectional way to suppress co-scene actions while ensuring the integrity of positive actions, by cross-supervising the original and augmented videos. Finally, our proposed Bi-SCC can be applied to current WTAL approaches, and improve their performance. Experimental results show that our approach outperforms the state-of-the-art methods on THUMOS14 and ActivityNet. The code is available at https://github.com/lgzlIlIlI/BiSCC.

Weight-Averaged Consistency Targets Improve Semi-Supervised Deep Learning Results

Abstract: The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change only once per epoch, Temporal Ensembling becomes unwieldy when learning large datasets. To overcome this problem, we propose Mean Teacher, a method that averages model weights instead of label predictions. As an additional benefit, Mean Teacher improves test accuracy and enables training with fewer labels than Temporal Ensembling. Without changing the network architecture, Mean Teacher achieves an error rate of 4.35% on SVHN with 250 labels, outperforming Temporal Ensembling trained with 1000 labels. We also show that a good network architecture is crucial to performance. Combining Mean Teacher and Residual Networks, we improve the state of the art on CIFAR-10 with 4000 labels from 10.55% to 6.28%, and on ImageNet 2012 with 10% of the labels from 35.24% to 9.11%.

model
traditional teacher-student network


Benchmark For Semi-Supervised Multi-Modal Segmentation

We compare our proposed M3L semi-supervised framework against supervised-only and mean teacher #REFR to prove its efficacy.

We create three different semi-supervised configurations using 0.1% (49), 0.2% (98), and 1% (491) of the data as labeled and the rest as unlabeled (49159, 49101, 48708) .

SUN RGBD is a challenging dataset with 37 classes and has 5049 test samples.

It originally had 5285 training samples which we split into 90/10 train/val (4757 / 528) samples.

For semi-supervised training, we create three configurations and treat 6.25% (297), 12.5% (594), and 25% (1189) of the data as labeled and the rest as unlabeled (4460, 4163, 3568) .

We also compare the performance of Linear Fusion with supervised-only transformer-based fusion approaches presented in the literature #OTHEREFR .

• URN #OTHEREFR : We choose a straightforward fusion mechanism that closely resembles the approach presented in Unified Representation Network #OTHEREFR , but we use Segformer as the base architecture.

Different encoders are used for the two modalities and the encoded representations are fused (averaged) before passing them to a single decoder.

Since two encoders are trained, URN has ∼ 2× the number of parameters than other methods described below.

• TF #OTHEREFR : The state-of-the-art Token Fusion #OTHEREFR framework.

Missing Modality Robustness In Semi-Supervised Multi-Modal Semantic Segmentation

Abstract: Using multiple spatial modalities has been proven helpful in improving semantic segmentation performance. However, there are several real-world challenges that have yet to be addressed: (a) improving label efficiency and (b) enhancing robustness in realistic scenarios where modalities are missing at the test time. To address these challenges, we first propose a simple yet efficient multi-modal fusion mechanism Linear Fusion, that performs better than the stateof-the-art multi-modal models even with limited supervision. Second, we propose M3L: Multi-modal Teacher for Masked Modality Learning, a semi-supervised framework that not only improves the multi-modal performance but also makes the model robust to the realistic missing modality scenario using unlabeled data. We create the first benchmark for semi-supervised multi-modal semantic segmentation and also report the robustness to missing modalities. Our proposal shows an absolute improvement of up to 10% on robust mIoU above the most competitive baselines. Our code is available at https:

M3L semi-supervised framework
supervised-only


These learning algorithms are adapted from the noise-free learning algorithm in #REFR , and work for both tree hierarchies and general concept hierarchies.

Section 6 contains algorithms for robust recognition in networks with feedback, for both tree hierarchies and general hierarchies.

For tree hierarchies in networks with feedback, we show that recognition requires only enough time for two passes: an upward pass to recognize whatever can be recognized based on lower-level information, and a downward pass to recognize concepts based on a combination of higher-level and lower-level information.

On the other hand, for general concept hierarchies in networks with feedback, we may need much more time-enough for many passes, both upward and downward.

We again describe how the recognition results extend to settings in which weights are approximate, and in which weights and thresholds are scaled.

In Section 7, we describe noise-free learning algorithms in feed-forward networks, which produce edge weights for the upward edges that suffice to support robust recognition.

The extension to general hierarchies requires us to reconsider the use of the Winner-Take-All (WTA) module in the algorithm of #OTHEREFR , since the previous version does not work with overlap; we present a new version of the WTA mechanism.

As before, the weight adjustments are based on Oja's rule #OTHEREFR .

We show that our new learning algorithms can be viewed as producing approximate, scaled weights as described in Section 5, which can be used to decompose the correctness proof for the learning algorithms. Finally, we briefly discuss extensions to noisy learning.

In Section 8, we extend the learning algorithms for feed-forward networks to accommodate feedback.

Here we simply separate the learning mechanisms for the weights of the upward and downward edges, using a different rule for learning the weights of the downward edges.

Learning Hierarchically-Structured Concepts Ii: Overlapping Concepts, And Networks With Feedback

Abstract: We continue our study from [5], of how concepts that have hierarchical structure might be represented in brain-like neural networks, how these representations might be used to recognize the concepts, and how these representations might be learned. In [5], we considered simple tree-structured concepts and feed-forward layered networks. Here we extend the model in two ways: we allow limited overlap between children of different concepts, and we allow networks to include feedback edges. For these more general cases, we describe and analyze algorithms for recognition and algorithms for learning. We also use the abbreviations supp r,f (B, * , t) for S(, t), supp r,f (B, , *) for t S(, t), and supp r,f (B, * , *) for ,t S(, t), Notice that each of these three unions must converge to a finite set since all the sets S(, t) are subsets of the single finite set C of concepts. Now we have two monotonicity results, for r and f : Lemma 2.5. Let C be any concept hierarchy satisfying the limited-overlap property, and let B ⊆ D 0. Consider r, r , where 0 ≤ r ≤ r ≤ 1, and arbitrary f. Then supp r ,f (B) ⊆ supp r,f (B). Lemma 2.6. Let C be any concept hierarchy satisfying the limited-overlap property, and let B ⊆ D 0. Consider f, f , where 0 ≤ f ≤ f , and arbitrary r ∈ [0, 1]. Then supp r,f (B) ⊆ supp r,f (B).

Learning Hierarchically Structured Concepts

Abstract: We study the question of how concepts that have structure get represented in the brain. Specifically, we introduce a model for hierarchically structured concepts and we show how a biologically plausible neural network can recognize these concepts, and how it can learn them in the first place. Our main goal is to introduce a general framework for these tasks and prove formally how both (recognition and learning) can be achieved. ::: We show that both tasks can be accomplished even in presence of noise. For learning, we analyze Oja's rule formally, a well-known biologically-plausible rule for adjusting the weights of synapses. We complement the learning results with lower bounds asserting that, in order to recognize concepts of a certain hierarchical depth, neural networks must have a corresponding number of layers.

learning algorithms
general concept hierarchies


To create language-specific embeddings, we utilize Deep-Walk #REFR , and follow a uniform random sampling approach: the next entity to visit in the random walk is chosen uniformly from all neighbors of the current entity.

To leverage the information of entities and the structure of the language-specific knowledge graph efficiently, we propose a language-specific embedding technique that learns continuous vector representation of entities representing their relations in a language .

This technique maps the entities to low-dimensional vectors, which are similar in cases where two entities appear close to each other in the language-specific context of .

After creating a set of random walks following the Deep-Walk approach, we train a Word2Vec model.

The resulting language-specific embeddings are utilized for (i) creating a candidate set of events relevant to the query entity and (ii) for measuring the language-specific relevance between the query entity and the event.

Laser: Language-Specific Event Recommendation

Abstract: While societal events often impact people worldwide, a significant fraction of events has a local focus that primarily affects specific language communities. Examples include national elections, the development of the Coronavirus pandemic in different countries, and local film festivals such as the César Awards in France and the Moscow International Film Festival in Russia. However, existing entity recommendation approaches do not sufficiently address the language context of recommendation. This article introduces the novel task of language-specific event recommendation, which aims to recommend events relevant to the user query in the language-specific context. This task can support essential information retrieval activities, including web navigation and exploratory search, considering the language context of user information needs. We propose LaSER, a novel approach toward language-specific event recommendation. LaSER blends the language-specific latent representations (embeddings) of entities and events and spatio-temporal event features in a learning to rank model. This model is trained on publicly available Wikipedia Clickstream data. The results of our user study demonstrate that LaSER outperforms state-of-the-art recommendation baselines by up to 33 percentage points in MAP@5 concerning the language-specific relevance of recommended events.

Deepwalk: Online Learning Of Social Representations

Abstract: We present DeepWalk, a novel approach for learning latent representations of vertices in a network. These latent representations encode social relations in a continuous vector space, which is easily exploited by statistical models. DeepWalk generalizes recent advancements in language modeling and unsupervised feature learning (or deep learning) from sequences of words to graphs. DeepWalk uses local information obtained from truncated random walks to learn latent representations by treating walks as the equivalent of sentences. We demonstrate DeepWalk's latent representations on several multi-label network classification tasks for social networks such as BlogCatalog, Flickr, and YouTube. Our results show that DeepWalk outperforms challenging baselines which are allowed a global view of the network, especially in the presence of missing information. DeepWalk's representations can provide $F_1$ scores up to 10% higher than competing methods when labeled data is sparse. In some experiments, DeepWalk's representations are able to outperform all baseline methods while using 60% less training data. DeepWalk is also scalable. It is an online learning algorithm which builds useful incremental results, and is trivially parallelizable. These qualities make it suitable for a broad class of real world applications such as network classification, and anomaly detection.


The CoLa task, which is based on The Corpus of Linguistic Acceptability #REFR , is a linguistic acceptability task.

The Aggression and AggressionPer tasks, which are based on the Wikipedia Talk Labels dataset #OTHEREFR , are offensive language detection tasks.

These tasks are presented in the form of binary classification, with the task being to classify text as either aggressive or non-aggressive.

Specifically, AggressionPer uses the personalized variant by providing user annotations with prompts.

This task is presented in the form of binary classification, with the task being to classify text as either grammatically correct or not.

The ColBERT task, which is based on the ColBERT dataset #OTHEREFR , is a humor recognition task.

This task is presented in the form of binary classification, with the task being to classify text as either humor or not humor.

• Sarcasm The Sarcasm task, which is based on the Sarcasmania dataset #OTHEREFR , is sarcasm classification task.

Chatlog: Recording And Analyzing Chatgpt Across Time

Abstract: While there are abundant researches about evaluating ChatGPT on natural language understanding and generation tasks, few studies have investigated how ChatGPT's behavior changes over time. In this paper, we collect a coarse-to-fine temporal dataset called ChatLog, consisting of two parts that update monthly and daily: ChatLog-Monthly is a dataset of 38,730 question-answer pairs collected every month including questions from both the reasoning and classification tasks. ChatLog-Daily, on the other hand, consists of ChatGPT's responses to 1000 identical questions for long-form generation every day. We conduct comprehensive automatic and human evaluation to provide the evidence for the existence of ChatGPT evolving patterns. We further analyze the unchanged characteristics of ChatGPT over time by extracting its knowledge and linguistic features. We find some stable features to improve the robustness of a RoBERTa-based detector on new versions of ChatGPT. We will continuously maintain our project at https://github.com/ THU-KEG/ChatLog.

Neural Network Acceptability Judgments

Abstract: This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus...


There is a standard and very flexible technique for incorporating equality, and other "enrichments" of logics #REFR .

We introduce the categorical structure abstracting logical equivalence with respect to the studied logical fragment, establish the corresponding categorical FVM theorem and illustrate this with concrete examples.

The examples in Section IV include a new refinement of a result by Dawar, Severini, and Zapata #OTHEREFR , showing that equivalence in 3variable counting logic without equality and with restricted conjunctions implies cospectrality.

Section VI develops FVM theorems for products of structures, uniformly in the logic of interest, greatly generalising Mostowski's original result.

Unlike our methods, which are parametric in the choice of game comonad, a typical game argument cannot be used to prove such a result, as the game is tied to a specific choice of logic.

In fact, the game comonads E k and P k naturally deal with first-order logic without equality.

This is introduced and related to FVM theorems in Section VII, and illustrated with further examples, including the handling of weak-bisimulation.

A Categorical Account Of Composition Methods In Logic

Abstract: We present a categorical theory of the composition methods in finite model theory-a key technique enabling modular reasoning about complex structures by building them out of simpler components. The crucial results required by the composition methods are Feferman-Vaught-Mostowski (FVM) type theorems, which characterize how logical equivalence behaves under composition and transformation of models. Our results are developed by extending the recently introduced game comonad semantics for model comparison games. This level of abstraction allow us to give conditions yielding FVM type results in a uniform way. Our theorems are parametric in the classes of models, logics and operations involved. Furthermore, they naturally account for the positive existential fragment, and extensions with counting quantifiers of these logics. We also reveal surprising connections between FVM type theorems, and classical concepts in the theory of monads. We illustrate our methods by recovering many classical theorems of practical interest, including a refinement of a previous result by Dawar, Severini, and Zapata concerning the 3-variable counting logic and cospectrality. To highlight the importance of our techniques being parametric in the logic of interest, we prove a family of FVM theorems for products of structures, uniformly in the logic in question, which cannot be done using specific game arguments.

Relating Structure And Power: Comonadic Semantics For Computational Resources

Abstract: Combinatorial games are widely used in finite model theory, constraint satisfaction, modal logic and concurrency theory to characterize logical equivalences between structures. In particular, Ehrenfeucht-Fraisse games, pebble games, and bisimulation games play a central role. We show how each of these types of games can be described in terms of an indexed family of comonads on the category of relational structures and homomorphisms. The index k is a resource parameter which bounds the degree of access to the underlying structure. The coKleisli categories for these comonads can be used to give syntax-free characterizations of a wide range of important logical equivalences. Moreover, the coalgebras for these indexed comonads can be used to characterize key combinatorial parameters: tree-depth for the Ehrenfeucht-Fraisse comonad, tree-width for the pebbling comonad, and synchronization-tree depth for the modal unfolding comonad. These results pave the way for systematic connections between two major branches of the field of logic in computer science which hitherto have been almost disjoint: categorical semantics, and finite and algorithmic model theory.


In this case, the reconstructions are symmetric and do not require more computation time than the Taylor series-based first-order reconstructions in equation #REFR .

λ i→ (x) = λ(x) − 1 2 ∇ i λ, x − x i (25b)

yields first-order accurate reconstruction functions.

Empirically, these lead to worse results than the Pythagorean means, possibly due to the asymmetry of the reconstructions λ ij = λ ji , while requiring additional computational steps. A higher-order scheme follows from a one-dimensional Hermite interpolation

with the directional vector d ij = x j − x i .

Higher-Order Gfdm For Linear Elliptic Operators *

Abstract: We present a novel approach of discretizing diffusion operators of the form ∇ • (λ∇u) in the context of meshfree generalized finite difference methods. Our ansatz uses properties of derived operators and combines the discrete Laplace operator with reconstruction functions approximating the diffusion coefficient λ. Provided that the reconstructions are of a sufficiently high order, we prove that the order of accuracy of the discrete Laplace operator transfers to the derived diffusion operator. We show that the new discrete diffusion operator inherits the diagonal dominance property of the discrete Laplace operator and fulfills enrichment properties. Our numerical results for elliptic and parabolic partial differential equations show that even loworder reconstructions preserve the order of the underlying discrete Laplace operator for sufficiently smooth diffusion coefficients. In experiments, we demonstrate the applicability of the new discrete diffusion operator to interface problems with point clouds not aligning to the interface and numerically prove first-order convergence.

The Overlapped Radial Basis Function-Finite Difference (Rbf-Fd) Method For The Numerical Solution Of Pdes

Abstract: We present a generalization of the RBF-FD method that allows full control of the overlap between RBF-FD stencils. We accomplish this by introducing a continuous overlap parameter $\delta \in [0,1]$ such that $\delta=1$ recovers the standard RBF-FD method and $\delta=0$ results in a full decoupling of the RBF-FD stencils. We show with a simple example that global interpolation with both RBFs and RBFs augmented with polynomials is superior to polynomial least squares in confining Runge oscillations to the boundary, thereby justifying retaining more RBF-FD weights from each RBF-FD stencil than just the center; this also provides greater insight into the success of the recent polyharmonic spline RBF-FD method with high-degree polynomials appended. We demonstrate the efficiency of the \emph{overlapped RBF-FD method} by solving parabolic differential equations in 2D and 3D. Our results show that the new method can achieve as high as a 20x speedup over the standard RBF-FD method in the task of forming differentiation matrices. Further, the overlapped RBF-FD method allows for stencil sizes up to 6x larger than those usable in the standard RBF-FD method for a lower total computational cost.


Similar to traditional PLMs like BERT, these recent LLMs also follow the Transformer architecture #REFR , but they differ from each other in training objectives, data sources, optimization details, etc.

In #OTHEREFR , a largescale corpus of agricultural literature with more than 300 million tokens is used to fine-tune the generic BERT into AgriBERT.

Such domain-specific language models could be potentially used to assist many agricultural applications, such as agricultural marketing, agricultural social media analytics, etc.

LLMs usually refer to PLMs with more than billions of model parameters, such as the GPT (Generative Pre-trained Transformer) models (GPT-1 #OTHEREFR , GPT-2 #OTHEREFR , GPT-3 #OTHEREFR , Instruct-GPT #OTHEREFR , ChatGPT, and GPT-4 [125] ), LLaMA (Large Language Model Meta AI) #OTHEREFR , and BLOOM #OTHEREFR .

For instance, GPT-3 and BLOOM have 175 billion parameters, and LLaMA has 65 billion parameters.

For example, BERT only adopts the encoder of Transformer as it focuses on learning language representations, while GPT models adopt the decoder of Transformer, as their major objective is to generate realistic text.

Owing to their huge model sizes and effective incorporation of human feedback (e.g., prompts), the latest LLMs like ChatGPT and GPT-4 have not only achieved amazing performance in traditional NLP tasks like language understanding, language generation, and question answering, but also demonstrated remarkable capabilities in zero-shot learning and reasoning #OTHEREFR .

For instance, ChatGPT has shown a very promising capability in inductive reasoning and deductive reasoning #OTHEREFR .

GPT-4 is a large multimodal model that can deal with both language and visual inputs.

In addition, LLMs are also effective tools for data augmentation #OTHEREFR , which is very helpful for certain real-world scenarios with limited data.

Agi For Agriculture

Abstract: Artificial General Intelligence (AGI) is poised to revolutionize a variety of sectors, including healthcare, finance, transportation, and education. Within healthcare, AGI is being utilized to analyze clinical medical notes, recognize patterns in patient data, and aid in patient management. Agriculture is another critical sector that impacts the lives of individuals worldwide. It serves as a foundation for providing food, fiber, and fuel, yet faces several challenges, such as climate change, soil degradation, water scarcity, and food security. AGI has the potential to tackle these issues by enhancing crop yields, reducing waste, and promoting sustainable farming practices. It can also help farmers make informed decisions by leveraging real-time data, leading to more efficient and effective farm management. This paper delves into the potential future applications of AGI in agriculture, such as agriculture image processing, natural language processing (NLP), robotics, knowledge graphs, and infrastructure, and their impact on precision livestock and precision crops. By leveraging the power of AGI, these emerging technologies can provide farmers with actionable insights, allowing for optimized decisionmaking and increased productivity. The transformative potential of AGI in agriculture is vast, and this paper aims to highlight its potential to revolutionize the industry.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


Furthermore, we adopt the ground truth sampling strategy of #REFR which randomly takes boxes and their points from a database and places them in the input point cloud.

The point cloud of a non keyframe collected at timestamp t − k (k = 1, . . .

, 9) are mapped from the LiDAR frame at this time step to the world frame W using ego vehicle poses and LiDAR calibration.

The ground truth of rectification transformation in (2) requires knowing objects' poses in the world frame W at the keyframe timestamp W T O(t) and non keyframe timestamp W T O(t−k) .

Since NuScenes only provides objects' poses in keyframes, we obtain their poses in non keyframes by linearly interpolating annotations of two keyframes that are respectively prior and successor to each non keyframe.

To improve the generalization of our model, the following geometric transformations are applied to point clouds and ground truth: random flip along the x-and y-axis of the world frame W, global scaling with a factor sampled from U [0.95,1.05] , and global rotation by an angle sampled from U [−π/8,π/8] around the z-axis of the world frame W. Here, U denotes the uniform distribution.

Notably, we introduce a modification to this sampling strategy by complementing each sampled ground truth with points in its trajectory spanning from the current time step to 10 steps in the past.

This modification is similar to the "Sequence GtAug" of #OTHEREFR .

Its motivation is to increase the number of moving objects in each point cloud, thus increasing the amount of supervision on the Object Motion Estimation.

A comparison between the regular ground truth sampling and our modified version is shown in Fig. 3 .

In our experiments, input point clouds are limited to the range of [−51.2, 51.2] × [−51.2, 51.2] × [−5.0, 3.0] meters along X-, Y-, and Z-axis of the world frame W and the voxel size is set to (0.1, 0.1, 0.2). We use OpenPCDet #OTHEREFR for our implementation.

Aligning Bird-Eye View Representation Of Point Cloud Sequences Using Scene Flow

Abstract: Low-resolution point clouds are challenging for object detection methods due to their sparsity. Densifying the present point cloud by concatenating it with its predecessors is a popular solution to this challenge. Such concatenation is possible thanks to the removal of ego vehicle motion using its odometry. This method is called Ego Motion Compensation (EMC). Thanks to the added points, EMC significantly improves the performance of single-frame detectors. However, it suffers from the shadow effect that manifests in dynamic objects' points scattering along their trajectories. This effect results in a misalignment between feature maps and objects' locations, thus limiting performance improvement to stationary and slow-moving objects only. Scene flow allows aligning point clouds in 3D space, thus naturally resolving the misalignment in feature spaces. By observing that scene flow computation shares several components with 3D object detection pipelines, we develop a plug-in module that enables single-frame detectors to compute scene flow to rectify their Bird-Eye View representation. Experiments on the NuScenes dataset show that our module leads to a significant increase (up to 16%) in the Average Precision of large vehicles, which interestingly demonstrates the most severe shadow effect. The code is published at https://github.com/quan-dao/pc-corrector. Index Terms-object detection, deep learning, scene flow, perception, LiDAR • Scene flow pipelines require points' features which can be computed by first converting input point clouds to BEV representation, then interpolating using points' projection

Class-Balanced Grouping And Sampling For Point Cloud 3D Object Detection

Abstract: This report presents our method which wins the nuScenes3D Detection Challenge [17] held in Workshop on Autonomous Driving(WAD, CVPR 2019). Generally, we utilize sparse 3D convolution to extract rich semantic features, which are then fed into a class-balanced multi-head network to perform 3D object detection. To handle the severe class imbalance problem inherent in the autonomous driving scenarios, we design a class-balanced sampling and augmentation strategy to generate a more balanced data distribution. Furthermore, we propose a balanced group-ing head to boost the performance for the categories withsimilar shapes. Based on the Challenge results, our methodoutperforms the PointPillars [14] baseline by a large mar-gin across all metrics, achieving state-of-the-art detection performance on the nuScenes dataset. Code will be released at CBGS.


Recently, federated learning (FL) #REFR has been widely studied as a privacypreserving distributed training paradigm, where clients cooperatively train a machine learning (ML) model under the coordination of a central server. FL training consists of multiple communication rounds.

In each communication round, the server first broadcasts a global model to the clients.

Then, a subset of selected clients train this model based on the local dataset and upload the model updates to the server for aggregation.

Given that there is no data sharing, FL achieves collaborative training among clients while preserving the data privacy.

However, recent studies #OTHEREFR demonstrate that FL is vulnerable to model attacks due to the data and device heterogeneity of clients.

Backdoor attack #OTHEREFR ) misleads an ML model to misclassify the data with specific triggers into a certain label.

Dabs: Data-Agnostic Backdoor Attack At The Server In Federated Learning

Abstract: Federated learning (FL) attempts to train a global model by aggregating local models from distributed devices under the coordination of a central server. However, the existence of a large number of heterogeneous devices makes FL vulnerable to various attacks, especially the stealthy backdoor attack. Backdoor attack aims to trick a neural network to misclassify data to a target label by injecting specific triggers while keeping correct predictions on original training data. Existing works focus on client-side attacks which try to poison the global model by modifying the local datasets. In this work, we propose a new attack model for FL, namely Data-Agnostic Backdoor attack at the Server (DABS), where the server directly modifies the global model to backdoor an FL system. Extensive simulation results show that this attack scheme achieves a higher attack success rate compared with baseline methods while maintaining normal accuracy on the clean data.

Communication-Efficient Learning Of Deep Networks From Decentralized Data

Abstract: Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning. We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.

federated learning
privacypreserving distributed training


The HAWQ algorithm #REFR utilizes the Hessian eigenvalues as the sensitivity measurement matrix for hierarchical features in the network and adopts mixed-precision quantization for network layers with divergent sensitivities.

where (r j , r j + 1] represents the range of real values depending on the quantization bits k, i.e., j = 0, ..., 2 k − 1.

The quantization process map this range (r j , r j + 1] to the value q j .

However, directly using a low-bit precision to quantize a model will decrease the model accuracy.

A promising approach to achieve a trade-off between model size and accuracy is to mix different precisions to quantize models.

This motivates us to consider sensitivity when modifying network features.

In other words, changing parts of the model at different levels will have different effects on how the model works.

To measure network layers, we adopt the average Hessian trace H proposed by the HAWQ-V2 algorithm as the sensitivity matrix. We have the following Lemma: Lemma 1 #OTHEREFR .

Suppose that the model is twice differentiable and converges to a local minimum when we quantize two layers with the same perturbation, we have:

Diversifying The High-Level Features For Better Adversarial Transferability

Abstract: Given the great threat of adversarial attacks against Deep Neural Networks (DNNs), numerous works have been proposed to boost transferability to attack real-world applications. However, existing attacks often utilize advanced gradient calculation or input transformation but ignore the white-box model. Inspired by the fact that DNNs are overparameterized for superior performance, we propose diversifying the high-level features (DHF) for more transferable adversarial examples. In particular, DHF perturbs the highlevel features by randomly transforming the high-level features and mixing them with the feature of benign samples when calculating the gradient at each iteration. Due to the redundancy of parameters, such transformation does not affect the classification performance but helps identify the invariant features across different models, leading to much better transferability. Empirical evaluations on ImageNet dataset show that DHF could effectively improve the transferability of existing momentum-based attacks. Incorporated into the input transformation-based attacks, DHF generates more transferable adversarial examples and outperforms the baselines with a clear margin when attacking several defense models, showing its generalization to various attacks and high effectiveness for boosting transferability. Recent studies have shown that Deep Neural Networks (DNNs) are vulnerable to adversarial examples [46], i.e., the inputs with indistinguishable perturbation can mislead the DNNs. Such vulnerability brings a significant threat to the security of widely deployed DNNs in the physical world, e.g.

Hawq: Hessian Aware Quantization Of Neural Networks With Mixed-Precision

Abstract: Model size and inference speed/power have become a major challenge in the deployment of Neural Networks for many applications. A promising approach to address these problems is quantization. However, uniformly quantizing a model to ultra low precision leads to significant accuracy degradation. A novel solution for this is to use mixed-precision quantization, as some parts of the network may allow lower precision as compared to other layers. However, there is no systematic way to determine the precision of different layers. A brute force approach is not feasible for deep networks, as the search space for mixed-precision is exponential in the number of layers. Another challenge is a similar factorial complexity for determining block-wise fine-tuning order when quantizing the model to a target precision. Here, we introduce Hessian AWare Quantization (HAWQ), a novel second-order quantization method to address these problems. HAWQ allows for the automatic selection of the relative quantization precision of each layer, based on the layer's Hessian spectrum. Moreover, HAWQ provides a deterministic fine-tuning order for quantizing layers, based on second-order information. We show the results of our method on Cifar-10 using ResNet20, and on ImageNet using Inception-V3, ResNet50 and SqueezeNext models. Comparing HAWQ with state-of-the-art shows that we can achieve similar/better accuracy with $8\times$ activation compression ratio on ResNet20, as compared to DNAS~\cite{wu2018mixed}, and up to $1\%$ higher accuracy with up to $14\%$ smaller models on ResNet50 and Inception-V3, compared to recently proposed methods of RVQuant~\cite{park2018value} and HAQ~\cite{wang2018haq}. Furthermore, we show that we can quantize SqueezeNext to just 1MB model size while achieving above $68\%$ top1 accuracy on ImageNet.