DRQN #REFR is a DRL method focusing on solving discrete control and POMDPs using historical information.

Other quantities, including average trip speed and delay, are also important to measure the control effects of the control method.

The average trip speed is computed as the mean of average speeds for all trips, and the delay is calculated as the ratio of the gaps between free-flow TTT and current TTT over the free-flow TTT.

Furthermore, we compared the proposed model with the no control, demonstrators (ALINEA or Gating), and other baseline methods, such as DAgger, DQN and DRQN. DAgger #OTHEREFR

2011) is an imitation learning algorithm that iteratively learns deterministic policies from a prepared generated dataset. DQN #OTHEREFR

2015 ) is a classic DRL method for discrete control and fully observable problem.

Demonstration-Guided Deep Reinforcement Learning For Coordinated Ramp Metering And Perimeter Control In Large Scale Networks

Abstract: Effective traffic control methods have great potential in alleviating network congestion. Existing literature generally focuses on a single control approach, while few studies have explored the effectiveness of integrated and coordinated control approaches. This study considers two representative control approaches: ramp metering for freeways and perimeter control for homogeneous urban roads, and we aim to develop a deep reinforcement learning (DRL)-based coordinated control framework for large-scale networks. The main challenges are 1) there is a lack of efficient dynamic models for both freeways and urban roads; 2) the standard DRL method becomes ineffective due to the complex and non-stationary network dynamics. In view of this, we propose a novel meso-macro dynamic network model and first time develop a demonstration-guided DRL method to achieve large-scale coordinated ramp metering and perimeter control. The dynamic network model hybridizes the link and generalized bathtub models to depict the traffic dynamics of freeways and urban roads, respectively. For the DRL method, we incorporate demonstration to guide the DRL method for better convergence by introducing the concept of "teacher" and "student" models. The teacher models are traditional controllers (e.g., ALINEA, Gating), which provide control demonstrations. The student models are DRL methods, which learn from the teacher and aim to surpass the teacher's performance. To validate the proposed framework, we conduct two case studies in a small-scale network and a real-world large-scale traffic network in Hong Kong. Numerical results show that the proposed DRL method outperforms demonstrators as well as the state-of-the-art DRL methods, and the coordinated control is more effective than just controlling ramps or perimeters by 8.7% and 22.6%, respectively. The research outcome reveals the great potential of combining traditional controllers with DRL for coordinated control in large-scale networks.

Deep Recurrent Q-Learning For Partially Observable Mdps

Abstract: Deep Reinforcement Learning has yielded proficient controllers for complex tasks. However, these controllers have limited memory and rely on being able to perceive the complete game screen at each decision point. To address these shortcomings, this article investigates the effects of adding recurrency to a Deep Q-Network (DQN) by replacing the first post-convolutional fully-connected layer with a recurrent LSTM. The resulting \textit{Deep Recurrent Q-Network} (DRQN), although capable of seeing only a single frame at each timestep, successfully integrates information through time and replicates DQN's performance on standard Atari games and partially observed equivalents featuring flickering game screens. Additionally, when trained with partial observations and evaluated with incrementally more complete observations, DRQN's performance scales as a function of observability. Conversely, when trained with full observations and evaluated with partial observations, DRQN's performance degrades less than DQN's. Thus, given the same length of history, recurrency is a viable alternative to stacking a history of frames in the DQN's input layer and while recurrency confers no systematic advantage when learning to play the game, the recurrent net can better adapt at evaluation time if the quality of observations changes.


Following the methods introduced in an ODE context in #REFR , we assume we are given an sˆs real or complex matrix D and a real or complex column vector Θ " pθ 1 ,¨¨¨, θ s q t , and we consider some h ą 0 as a time step.

A b L "¨a 11 L¨¨¨a 1s L . . . . . . a s1 L¨¨¨a ss L‹ ‚.

The notation I will be used for the identity operator in general.

Since the problem (1) is autonomous, we always consider t " 0 as the starting time without loss of generality.

We take u 0 P H σ (classical Sobolev space of order σ, to be defined in the next section), and we denote by t Þ Ñ uptq the exact solution of (1) with initial datum up0q " u 0 .

Of course, for fixed t ě 0, the function uptq is itself a function of x P Ω. We will often omit this dependency in the notations.

We initialize the linearly implicit method with u 0 P H σ , close to u 0 .

We initialize the additional variables pγ´1 ,i q 1ďiďs with approximations of pN puppc i´1 qhqqq 1ďiďs and we set Γ´1 " pγ´1 ,i q t 1ďiďs P pH σ q s .

High Order Linearly Implicit Methods For Semilinear Evolution Pdes

Abstract: This paper considers the numerical integration of semilinear evolution PDEs using the high order linearly implicit methods developped in [21] in the ODE setting. These methods use a collocation Runge-Kutta method as a basis, and additional variables that are updated explicitly and make the implicit part of the collocation Runge-Kutta method only linearly implicit. In this paper, we introduce several notions of stability for the underlying Runge-Kutta methods as well as for the explicit step on the additional variables necessary to fit the context of evolution PDE. We prove a main theorem about the high order of convergence of these linearly implicit methods in this PDE setting, using the stability hypotheses introduced before. We use nonlinear Schrödinger equations and heat equations as main examples but our results extend beyond these two classes of evolution PDEs. We illustrate our main result numerically in dimensions 1 and 2, and we compare the efficiency of the linearly implicit methods with other methods from the litterature. We also illustrate numerically the necessity of the stability conditions of our main result.

High Order Linearly Implicit Methods For Evolution Equations: How To Solve An Ode By Inverting Only Linear Systems

Abstract: This paper introduces a new class of numerical methods for the time integration of evolution equations set as Cauchy problems of ODEs or PDEs. The systematic design of these methods mixes the Runge-Kutta collocation formalism with collocation techniques, in such a way that the methods are linearly implicit and have high order. The fact that these methods are implicit allows to avoid CFL conditions when the large systems to integrate come from the space discretization of evolution PDEs. Moreover, these methods are expected to be efficient since they only require to solve one linear system of equations at each time step, and efficient techniques from the literature can be used to do so. After the introduction of the methods, we set suitable definitions of consistency and stability for these methods. This allows for a proof that arbitrarily high order linearly implicit methods exist and converge when applied to ODEs. Eventually, we perform numerical experiments on ODEs and PDEs that illustrate our theoretical results for ODEs, and compare our methods with standard methods for several evolution PDEs.


Numerical Optimization For Max-Loss Form

AutoAttack also includes non-PGD methods, e.g., Square Attack #REFR , which is based on gradientfree random search.

x − x p ≤ ε} with p = 1, ∞, P ∆(x) takes simple forms.

For p = 2, sequential projection-first onto the box and then the norm ball, at least finds a feasible solution (see our clarification of these projections in Appendix A.1). Therefore, PGD is viable for these cases.

For other general p metrics and nonp metrics, analytical projectors mostly cannot be derived and existing PGD methods do not apply.

Previous work has shown that the solution quality of PGD is sensitive to hyperparameter tuning, e.g., step-size schedule and iteration budget #OTHEREFR .

PGD variants, such as AGPD methods where 'A' stands for Auto(matic) #OTHEREFR , try to make the tuning process automatic by combining a heuristic adaptive step-size schedule and momentum acceleration under a fixed iteration budget and are built into the popular AutoAttack package #OTHEREFR .

Square Attack does not have as effective attack performance as APGDs (see Table 2 in later sections) but is included as #OTHEREFR states that the diversity in the evaluation algorithms is important to achieve reliable numerical RE.

Optimization And Optimizers For Adversarial Robustness

Abstract: Empirical robustness evaluation (RE) of deep learning models against adversarial perturbations entails solving nontrivial constrained optimization problems. Existing numerical algorithms that are commonly used to solve them in practice predominantly rely on projected gradient, and mostly handle perturbations modeled by the 1, 2 and ∞ distances. In this paper, we introduce a novel algorithmic framework that blends a general-purpose constrained-optimization solver PyGRANSO with Constraint-Folding (PWCF), which can add more reliability and generality to the state-of-the-art RE packages, e.g., AutoAttack. Regarding reliability, PWCF provides solutions with stationarity measures and feasibility tests to assess the solution quality. For generality, PWCF can handle perturbation models that are typically inaccessible to the existing projected gradient methods; the main requirement is the distance metric to be almost everywhere differentiable. Taking advantage of PWCF and other existing numerical algorithms, we further explore the distinct patterns in the solutions found for solving these optimization problems using various combinations of losses, perturbation models, and optimization algorithms. We then discuss the implications of these patterns on the current robustness evaluation and adversarial training.

Square Attack: A Query-Efficient Black-Box Adversarial Attack Via Random Search

Abstract: We propose the Square Attack, a new score-based black-box $l_2$ and $l_\infty$ adversarial attack that does not rely on local gradient information and thus is not affected by gradient masking. The Square Attack is based on a randomized search scheme where we select localized square-shaped updates at random positions so that the $l_\infty$- or $l_2$-norm of the perturbation is approximately equal to the maximal budget at each step. Our method is algorithmically transparent, robust to the choice of hyperparameters, and is significantly more query efficient compared to the more complex state-of-the-art methods. In particular, on ImageNet we improve the average query efficiency for various deep networks by a factor of at least $2$ and up to $7$ compared to the recent state-of-the-art $l_\infty$-attack of Meunier et al. while having a higher success rate. The Square Attack can even be competitive to gradient-based white-box attacks in terms of success rate. Moreover, we show its utility by breaking a recently proposed defense based on randomization. The code of our attack is available at this https URL


Therefore, we fine-tuned the model with the multi-domain public task-oriented dialogue dataset MultiWOZ 2.2 #REFR .

With the advance of the pre-trained langue model, leveraging these models for embedding dialogue has exhibited promising results #OTHEREFR .

Following their success, we utilize the pre-trained SBERT #OTHEREFR ) as a backbone model; SBERT has a siamese network structure and performs well in classification #OTHEREFR , summarization (Zhong et al., 2020) , and intent clustering #OTHEREFR ).

The SBERT model is pre-trained on a written form text dataset that has a different linguistic pattern with the dialogue utterances.

This difference could hurt the accuracy on dialogue related tasks #OTHEREFR .

This dataset has nine intents types, and we finetuned the model to learn the embedding of utterances according to intent type.

In this data, intents are spanned multiple turns, and some utterances contain multiple intents in one utterance.

To clarify the match between the utterance and the intent label, we used only the first utterance of the spanned intent dialogue and excluded utterances that contained multiple intents.

We analyze the processed fine-tuning dataset in Table 1 .

As we aim for unsupervised intent induction, domain robust fine-tuning is crucial to identify the intents across the domain.

Doric : Domain Robust Fine-Tuning For Open Intent Clustering Through Dependency Parsing

Abstract: We present our work on Track 2 in the Dialog System Technology Challenges 11 (DSTC11). DSTC11-Track2 aims to provide a benchmark for zero-shot, cross-domain, intent-set induction. In the absence of in-domain training dataset, robust utterance representation that can be used across domains is necessary to induce users' intentions. To achieve this, we leveraged a multi-domain dialogue dataset to fine-tune the language model and proposed extracting Verb-Object pairs to remove the artifacts of unnecessary information. Furthermore, we devised the method that generates each cluster's name for the explainability of clustered results. Our approach achieved 3rd place in the precision score and showed superior accuracy and normalized mutual information (NMI) score than the baseline model on various domain datasets.

Multiwoz 2.1: A Consolidated Multi-Domain Dialogue Dataset With State Corrections And State Tracking Baselines

Abstract: MultiWOZ 2.0 (Budzianowski et al., 2018) is a recently released multi-domain dialogue dataset spanning 7 distinct domains and containing over 10,000 dialogues. Though immensely useful and one of the largest resources of its kind to-date, MultiWOZ 2.0 has a few shortcomings. Firstly, there is substantial noise in the dialogue state annotations and dialogue utterances which negatively impact the performance of state-tracking models. Secondly, follow-up work (Lee et al., 2019) has augmented the original dataset with user dialogue acts. This leads to multiple co-existent versions of the same dataset with minor modifications. In this work we tackle the aforementioned issues by introducing MultiWOZ 2.1. To fix the noisy state annotations, we use crowdsourced workers to re-annotate state and utterances based on the original utterances in the dataset. This correction process results in changes to over 32% of state annotations across 40% of the dialogue turns. In addition, we fix 146 dialogue utterances by canonicalizing slot values in the utterances to the values in the dataset ontology. To address the second problem, we combined the contributions of the follow-up works into MultiWOZ 2.1. Hence, our dataset also includes user dialogue acts as well as multiple slot descriptions per dialogue state slot. We then benchmark a number of state-of-the-art dialogue state tracking models on the MultiWOZ 2.1 dataset and show the joint state tracking performance on the corrected state annotations. We are publicly releasing MultiWOZ 2.1 to the community, hoping that this dataset resource will allow for more effective models across various dialogue subproblems to be built in the future.


ImageNet is a well-known dataset that consists of 14 million images from 20,000 different classes #REFR .

In various transfer learning applications, it is proposed to utilize a pre-trained neural network with its weights and modify some parts of the model to train for fine-tuning.

Pre-trained networks decrease the required time and resources for the training since they are already trained with intense resources by deep learning specialists #OTHEREFR .

Pre-trained CNNs are networks that are previously trained on massive datasets by deep learning specialists.

The VGG-19, EfficientNet-B0, ResNet50, VGG16, DenseNet121, MobileNet, Inception-v3 and InceptionResNet models are used as base models for the feature extraction during the study.

Mentioned models are the state-of-the-art pre-trained CNNs that are trained on the ImageNet dataset.

Researchers have explored the use of pre-trained feature extractors from diverse domains to extract valuable insights from data. Pre-trained feature extractors deliver noteworthy results. As an example, Narin et al.

built a feedforward neural network to classify COVID-19 patients based on chest X-Ray pictures and utilized pre-trained CNNs to extract features #OTHEREFR .

In addition, pre-trained CNNs have been applied to extract features from scalograms (combined with SVM classifiers) to detect schizophrenia #OTHEREFR .

Furthermore, pre-trained CNNs have been utilized to extract features from EEG images to identify neonatal seizures #OTHEREFR .

Recently, various events recorded by Φ-OTDR sensing system were classified using the pre-trained AlexNet architecture followed by an SVM classifier #OTHEREFR .

Deep Reproductive Feature Generation Framework For The Diagnosis Of Covid-19 And Viral Pneumonia Using Chest X-Ray Images

Abstract: The rapid and accurate detection of COVID-19 cases is critical for timely treatment and preventing the spread of the disease. In this study, a two-stage feature extraction framework using eight state-ofthe-art pre-trained deep Convolutional Neural Networks (CNNs) and an autoencoder is proposed to determine the health conditions of patients (COVID-19, Normal, Viral Pneumonia) based on chest X-rays. The X-ray scans are divided into four equally sized sections and analyzed by deep pre-trained CNNs. Subsequently, an autoencoder with three hidden layers is trained to extract reproductive features from the concatenated ouput of CNNs. To evaluate the performance of the proposed framework, three different classifiers, which are single-layer perceptron (SLP), multi-layer perceptron (MLP), and support vector machine (SVM) are used. Furthermore, the deep CNN architectures are used to create benchmark models and trained on the same dataset for comparision. The proposed framework outperforms other frameworks wih pre-trained feature extractors in binary classification and shows competitive results in three-class classification. The proposed methodology is taskindependent and suitable for addressing various problems. The results show that the discriminative features are a subset of the reproductive features, suggesting that extracting task-independent features is superior to the extraction only task-based features. The flexibility and task-independence of the reproductive features make the conceptive information approach more favorable. The proposed methodology is novel and shows promising results for analyzing medical image data.

Imagenet Large Scale Visual Recognition Challenge

Abstract: The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the five years of the challenge, and propose future directions and improvements.


We tokenize and truecase all text via Mose and use BPE #REFR to learn a vocabulary on each translation direction. The vocabulary size is set to 8000.

We conduct experiments on MuST-C dataset #OTHEREFR to verify the effectiveness of DNKD.

It contains translation from English (En) to 8 languages: German (De), French (Fr), Russian (Ru), Spanish (Es), Italian (It), Romanian (Ro), Portuguese (Pt), and Dutch (Nl).

We use dev set for validation and tst-COMMON set for test.

For speech input, we use the raw 16-bit 16kHz monochannel audio wave.

Decoupled Non-Parametric Knowledge Distillation For End-To-End Speech Translation

Abstract: Existing techniques often attempt to make knowledge transfer from a powerful machine translation (MT) to speech translation (ST) model with some elaborate techniques, which often requires transcription as extra input during training. However, transcriptions are not always available, and how to improve the ST model performance without transcription, i.e., data efficiency, has rarely been studied in the literature. In this paper, we propose Decoupled Non-parametric Knowledge Distillation (DNKD) from data perspective to improve the data efficiency. Our method follows the knowledge distillation paradigm. However, instead of obtaining the teacher distribution from a sophisticated MT model, we construct it from a non-parametric datastore via k-Nearest-Neighbor (kNN) retrieval, which removes the dependence on transcription and MT model. Then we decouple the classic knowledge distillation loss into target and non-target distillation to enhance the effect of the knowledge among non-target logits, which is the prominent "dark knowledge". Experiments on MuST-C corpus show that, the proposed method can achieve consistent improvement over the strong baseline without requiring any transcription.

Neural Machine Translation Of Rare Words With Subword Units

Abstract: Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character ngram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English!German and English!Russian by up to 1.1 and 1.3 BLEU, respectively.


The graphs of bounded functionality have at most factorial speed of growth #REFR . However, whether they admit an implicit representation is wide-open.

More generally, bounded degeneracy in a class provides us with an implicit representation, where the degeneracy of a graph G is the minimum k such that every induced subgraph of G contains a vertex of degree at most k.

Spinrad showed in #OTHEREFR that bounded clique-width also yields an implicit representation.

The recently introduced parameter twin-width generalizes clique-width in the sense that bounded clique-width implies bounded twin-width, but not vice versa.

It was shown in #OTHEREFR that bounded twin-width also implies the existence of an implicit representation.

The notion of graph functionality, introduced in [2], generalizes both degeneracy and twin-width in the sense that bounded degeneracy or bounded twin-width implies bounded functionality, but not vice versa.

To approach this question, in Section 3.1 we analyse a parameter intermediate between twin-width and functionality.

Then in Sections 3.2 and 3.3, we introduce more parameters and report some results concerning them.

Graph Parameters, Implicit Representations And Factorial Properties *

Abstract: How to efficiently represent a graph in computer memory is a fundamental data structuring question. In the present paper, we address this question from a combinatorial point of view. A representation of an n-vertex graph G is called implicit if it assigns to each vertex of G a binary code of length O(log n) so that the adjacency of two vertices is a function of their codes. A necessary condition for a hereditary class X of graphs to admit an implicit representation is that X has at most factorial speed of growth. This condition, however, is not sufficient, as was recently shown in [Hatami & Hatami, FOCS 2022]. Several sufficient conditions for the existence of implicit representations deal with boundedness of some parameters, such as degeneracy or clique-width. In the present paper, we analyze more graph parameters and prove a number of new results related to implicit representation and factorial properties.

Implicit Representations And Factorial Properties Of Graphs

Abstract: The idea of implicit representation of graphs was introduced in [S. Kannan, M. Naor, S. Rudich, Implicit representation of graphs, SIAM J. Discrete Mathematics, 5 (1992) 596--603] and can be defined as follows. A representation of an $n$-vertex graph $G$ is said to be implicit if it assigns to each vertex of $G$ a binary code of length $O(\log n)$ so that the adjacency of two vertices is a function of their codes. Since an implicit representation of an $n$-vertex graph uses $O(n\log n)$ bits, any class of graphs admitting such a representation contains $2^{O(n\log n)}$ labelled graphs with $n$ vertices. In the terminology of [J. Balogh, B. Bollob\'{a}s, D. Weinreich, The speed of hereditary properties of graphs, J. Combin. Theory B 79 (2000) 131--156] such classes have at most factorial speed of growth. In this terminology, the implicit graph conjecture can be stated as follows: every class with at most factorial speed of growth which is hereditary admits an implicit representation. The question of deciding whether a given hereditary class has at most factorial speed of growth is far from being trivial. In the present paper, we introduce a number of tools simplifying this question. Some of them can be used to obtain a stronger conclusion on the existence of an implicit representation. We apply our tools to reveal new hereditary classes with the factorial speed of growth. For many of them we show the existence of an implicit representation.


Properties Of Benamou-Brenier Energy And The Proposed Model

Then by #REFR and Proposition 3, we obtain (t → ρ n t ) ∈ C w ([0, 1]; M + (Ω)).

Therefore, by Proposition 2, it follows that ρ n ≥ 0,

By (13), we also have (ρ n , m n ) ∈ D and in particular ρ n 0 = µ 0 .

Consequently, it follows from Proposition 1 that ρ n = dt⊗ρ n t for some ρ n t ∈ M + (Ω), t ∈ [0, 1].

Moreover, by Proposition 1, we have that ρ n t (Ω) is constant with respect to time t, i.e., ρ n t (Ω) = ρ n 0 (Ω), for t ∈ (0, 1].

Since µ 0 ∈ M + (Ω), there exists a constant C 2 > 0 such that

It is then easy to see that there exists a subsequence (not relabeled) ρ n * ρ for some ρ ∈ M(X). Also, by Lemma 2, we have

|m n |(X) ≤ B(ρ n , m n )ρ n (X), It follows that sup n |m n |(X) ≤ sup n B(ρ n , m n )ρ n (X) < C 1 C 2 ,

An Optimal Transport Regularized Model To Image Reconstruction Problems

Abstract: Optimal transport problem has gained much attention in image processing field, such as computer vision, image interpolation and medical image registration. In this paper, we incorporate optimal transport into linear inverse problems as a regularization technique. We establish a new variational model based on Benamou-Brenier energy to regularize the evolution path from a template to latent image dynamically. The initial state of the continuity equation can be regarded as a template, which can provide priors for the reconstructed images. Also, we analyze the existence of solutions of such variational problem in Radon measure space. Moreover, the first-order primal-dual algorithm is constructed for solving this general imaging problem in a special grid strategy. Finally, numerical experiments for undersampled MRI reconstruction are presented which show that our proposed model can recover images well with high quality and structure preservation.

Indirect Image Registration With Large Diffeomorphic Deformations

Abstract: The paper adapts the large deformation diffeomorphic metric mapping framework for image registration to the indirect setting where a template is registered against a target that is given through indirect noisy observations. The registration uses diffeomorphisms that transform the template through a (group) action. These diffeomorphisms are generated by solving a flow equation that is defined by a velocity field with certain regularity. The theoretical analysis includes a proof that indirect image registration has solutions (existence) that are stable and that converge as the data error tends so zero, so it becomes a well-defined regularization method. The paper concludes with examples of indirect image registration in 2D tomography with very sparse and/or highly noisy data.


This automatic detection included discovering cyberbullying signals, such as the various forms of cyberbullying, and identifying posts from bullies, victims, and bystanders #REFR .

The results of their experiments showed 97% accuracy and 96% for F1-Score.

According to the researchers, the results of their studies performed very well in detecting cyberbullying in both languages.

In this way, their model can therefore detect multiple languages in the cyberbullying space #OTHEREFR , #OTHEREFR .

Automatic detection of cyberbullying in social media text by Van Hee et al.

#OTHEREFR aimed at exploring cyberbullying automatic detection on social media.

This study leveraged a binary classification to detect cyberbullying in English and Dutch.

In addition, the paper highlighted the issue of few publicly available datasets, making it difficult for researchers to conduct constructive research addressing cyberbullying.

To conclude, the classification revealed that the binary approach is promising to detect cyberbullying automatically in the future.

However, a dive into using deep learning techniques to detect cyberbullying automatically would improve the ability of the classifier to predict cyberbullying content. In #OTHEREFR , Lu et al.

proposed a Character-level Convolutional Neural Network with shortcuts (Char-CNNS) for the detection of cyberbullying in a social media text.

Cyberbullying In Text Content Detection: An Analytical Review

Abstract: Technological advancements have resulted in an exponential increase in the use of online social networks (OSNs) worldwide. While online social networks provide a great communication medium, they also increase the user's exposure to life-threatening situations such as suicide, eating disorder, cybercrime, compulsive behavior, anxiety, and depression. To tackle the issue of cyberbullying, most existing literature focuses on developing approaches to identifying factors and understanding the textual factors associated with cyberbullying. While most of these approaches have brought great success in cyberbullying research, data availability needed to develop model detection remains a challenge in the research space. This paper conducts a comprehensive literature review to provide an understanding of cyberbullying detection.

Automatic Detection Of Cyberbullying In Social Media Text

Abstract: While social media offer great communication opportunities, they also increase the vulnerability of young people to threatening situations online. Recent studies report that cyberbullying constitutes a growing problem among youngsters. Successful prevention depends on the adequate detection of potentially harmful messages and the information overload on the Web requires intelligent systems to identify potential risks automatically. The focus of this paper is on automatic cyberbullying detection in social media text by modelling posts written by bullies, victims, and bystanders of online bullying. We describe the collection and fine-grained annotation of a cyberbullying corpus for English and Dutch and perform a series of binary classification experiments to determine the feasibility of automatic cyberbullying detection. We make use of linear support vector machines exploiting a rich feature set and investigate which information sources contribute the most for the task. Experiments on a hold-out test set reveal promising results for the detection of cyberbullying-related posts. After optimisation of the hyperparameters, the classifier yields an F1 score of 64% and 61% for English and Dutch respectively, and considerably outperforms baseline systems.


GIFT #REFR constructs groupequivariant features by rotating or rescaling the images, and then collapses the group dimension using bilinear pooling to obtain invariant local descriptors.

Several studies #OTHEREFR suggest extracting local descriptors by invariant mapping of the order-based gradient histogram of a patch.

However, these classical methods for shallow gradient-based feature maps cannot be applied to deep feature maps from standard neural networks, in which rotation induces unpredictable feature variations.

Therefore, we propose a deep end-to-end pipeline to obtain orientation-normalized local descriptors by utilizing rotation-equivariant CNNs #OTHEREFR with additional losses.

Learning-based invariant local descriptors.

A branch of learning-based methods learns to obtain invariant local descriptors in an explicit manner.

However, their groups are limited to non-cyclic discrete rotations ranging from −90 • to 90

Furthermore, their reliance on data augmentation implies a lower sampling efficiency compared to groupequivariant networks.

LISRD #OTHEREFR jointly learns meta descriptors with different levels of regional variations and selects the most appropriate level of invariance given the context.

Another branch of learning methods aims to learn the invariance implicitly using descriptor similarity losses from the image pair using camera pose or homography supervision.

Learning Rotation-Equivariant Features For Visual Correspondence

Abstract: Extracting discriminative local features that are invariant to imaging variations is an integral part of establishing correspondences between images. In this work, we introduce a self-supervised learning framework to extract discriminative rotation-invariant descriptors using groupequivariant CNNs. Thanks to employing group-equivariant CNNs, our method effectively learns to obtain rotationequivariant features and their orientations explicitly, without having to perform sophisticated data augmentations. The resultant features and their orientations are further processed by group aligning, a novel invariant mapping technique that shifts the group-equivariant features by their orientations along the group dimension. Our group aligning technique achieves rotation-invariance without any collapse of the group dimension and thus eschews loss of discriminability. The proposed method is trained end-to-end in a self-supervised manner, where we use an orientation alignment loss for the orientation estimation and a contrastive descriptor loss for robust local descriptors to geometric/photometric variations. Our method demonstrates state-of-the-art matching accuracy among existing rotationinvariant descriptors under varying rotation and also shows competitive results when transferred to the task of keypoint matching and camera pose estimation.

Gift: Learning Transformation-Invariant Dense Visual Descriptors Via Group Cnns

Abstract: Finding local correspondences between images with different viewpoints requires local descriptors that are robust against geometric transformations. An approach for transformation invariance is to integrate out the transformations by pooling the features extracted from transformed versions of an image. However, the feature pooling may sacrifice the distinctiveness of the resulting descriptors. In this paper, we introduce a novel visual descriptor named Group Invariant Feature Transform (GIFT), which is both discriminative and robust to geometric transformations. The key idea is that the features extracted from the transformed versions of an image can be viewed as a function defined on the group of the transformations. Instead of feature pooling, we use group convolutions to exploit underlying structures of the extracted features on the group, resulting in descriptors that are both discriminative and provably invariant to the group of transformations. Extensive experiments show that GIFT outperforms state-of-the-art methods on several benchmark datasets and practically improves the performance of relative pose estimation.


The fundamental shift in operation grants event cameras higher dynamic range and higher temporal resolution than most conventional shutter cameras #REFR .

Neuromorphic sensing describes a class of technologies designed to mimic biological sensory and perceptual functions.

One such device in the field of neuromorphic vision is an event camera.

Instead of using a conventional shutter-based approach to capture an image, an event camera asynchronously streams 'events' whenever a pixel detects a relative change in brightness.

Each event has 4 parameters: A timestamp, an x and y coordinate corresponding to the location of the pixel that reported the event, and a polarity to denote whether an increase or decrease in brightness triggered the event.

Since events are only produced by motion or lighting changes in an otherwise still scene, the output data contains minimal redundant information of components that haven't changed over time.

Events are logged with microsecond precision and can reach equivalent framerates exceeding 10,000 frames per second.

These features make event cameras extremely practical sensors for the analysis of motion.

Drowsy driving is one of the leading causes of motor accidents globally, increasing a driver's risk of an accident by a factor of 5 or more compared to when they are alert #OTHEREFR .

Moreover, in the context of autonomous driving, drowsiness detection for the driver is important as it provides a key indicator that the DMS should present warnings and reduce the dependence of the vehicular control systems on the driver.

Many existing drowsiness detection systems tend to focus on analysis of eye-regions and the blinking behaviour of the subject #OTHEREFR .

Neuromorphic Sensing For Yawn Detection In Driver Drowsiness

Abstract: Driver monitoring systems (DMS) are a key component of vehicular safety and essential for the transition from semiautonomous to fully autonomous driving. A key task for DMS is to ascertain the cognitive state of a driver and to determine their level of tiredness. Neuromorphic vision systems, based on event camera technology, provide advanced sensing of facial characteristics, in particular the behavior of a driver's eyes. This research explores the potential to extend neuromorphic sensing techniques to analyze the entire facial region, detecting yawning behaviors that give a complimentary indicator of tiredness. A neuromorphic dataset is constructed from 952 video clips (481 yawns, 471 not-yawns) captured with an RGB colour camera, with 37 subjects. A total of 95,200 neuromorphic image frames are generated from this video data using a video-to-event converter. From these data 21 subjects were selected to provide a training dataset, 8 subjects were used for validation data, and the remaining 8 subjects were reserved for an 'unseen' test dataset. An additional 12,300 frames were generated from event simulations of a public dataset to test against other methods. A CNN with self-attention and a recurrent head was designed, trained, and tested with these data. Respective precision and recall scores of 95.9% and 94.7% were achieved on our test set, and 89.9% and 91% on the simulated public test set, demonstrating the feasibility to add yawn detection as a sensing component of a neuromorphic DMS.

Event-Based Vision: A Survey

Abstract: Event cameras are bio-inspired sensors that work radically different from traditional cameras. Instead of capturing images at a fixed rate, they measure per-pixel brightness changes asynchronously. This results in a stream of events, which encode the time, location and sign of the brightness changes. Event cameras posses outstanding properties compared to traditional cameras: very high dynamic range (140 dB vs. 60 dB), high temporal resolution (in the order of microseconds), low power consumption, and do not suffer from motion blur. Hence, event cameras have a large potential for robotics and computer vision in challenging scenarios for traditional cameras, such as high speed and high dynamic range. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential. This paper provides a comprehensive overview of the emerging field of event-based vision, with a focus on the applications and the algorithms developed to unlock the outstanding properties of event cameras. We present event cameras from their working principle, the actual sensors that are available and the tasks that they have been used for, from low-level vision (feature detection and tracking, optic flow, etc.) to high-level vision (reconstruction, segmentation, recognition). We also discuss the techniques developed to process events, including learning-based techniques, as well as specialized processors for these novel sensors, such as spiking neural networks. Additionally, we highlight the challenges that remain to be tackled and the opportunities that lie ahead in the search for a more efficient, bio-inspired way for machines to perceive and interact with the world.

conventional shutter cameras
higher temporal resolution


Next, we estimate the ego-motion by fitting and decomposing essential matrices from the background flow map using a neural-guided RANSAC #REFR .

Since most background objects in KITTI are rigid (e.g.

ground, buildings, etc), we can refine the background scene flow using a rigidity refinement step.

Specifically, we employ DDRNet-Slim #OTHEREFR , a lightweight 2D semantic segmentation network, to determine the rigid background.

DDRNet-Slim is pretrained on Cityscapes #OTHEREFR and then fine-tuned on KITTI.

Finally, the background scene flow is refined using the ego motion and the disparity of the first frame.

Comparison with State-of-the-art Methods.

We submit our approach to the website of the KITTI Scene Flow benchmark and report the leaderboard in Tab. 3. A visualized comparison is shown in Fig. 10 .

Notably, without leveraging any rigid body assumption, CamLiRAFT performs on par with the previous state-of-the-art method RigidMask #OTHEREFR (SF-all: 4.97% vs. 4.89%).

In contrast, RigidMask leverages more strict rigid-body assumptions by assigning rigid motion to all objects and employs 10 × more parameters.

Learning Optical Flow And Scene Flow With Bidirectional Camera-Lidar Fusion

Abstract: In this paper, we study the problem of jointly estimating the optical flow and scene flow from synchronized 2D and 3D data. Previous methods either employ a complex pipeline that splits the joint task into independent stages, or fuse 2D and 3D information in an "early-fusion" or "late-fusion" manner. Such one-size-fits-all approaches suffer from a dilemma of failing to fully utilize the characteristic of each modality or to maximize the inter-modality complementarity. To address the problem, we propose a novel end-to-end framework, which consists of 2D and 3D branches with multiple bidirectional fusion connections between them in specific layers. Different from previous work, we apply a point-based 3D branch to extract the LiDAR features, as it preserves the geometric structure of point clouds. To fuse dense image features and sparse point features, we propose a learnable operator named bidirectional camera-LiDAR fusion module (Bi-CLFM). We instantiate two types of the bidirectional fusion pipeline, one based on the pyramidal coarse-to-fine architecture (dubbed CamLiPWC), and the other one based on the recurrent all-pairs field transforms (dubbed CamLiRAFT). On FlyingThings3D, both CamLiPWC and CamLiRAFT surpass all existing methods and achieve up to a 47.9% reduction in 3D end-point-error from the best published result. Our best-performing model, CamLiRAFT, achieves an error of 4.26% on the KITTI Scene Flow benchmark, ranking 1st among all submissions with much fewer parameters. Besides, our methods have strong generalization performance and the ability to handle non-rigid motion. Code is available at https://github.com/MCG-NJU/CamLiFlow.

Neural-Guided Ransac: Learning Where To Sample Model Hypotheses

Abstract: We present Neural-Guided RANSAC (NG-RANSAC), an extension to the classic RANSAC algorithm from robust optimization. NG-RANSAC uses prior information to improve model hypothesis search, increasing the chance of finding outlier-free minimal sets. Previous works use heuristic side-information like hand-crafted descriptor distance to guide hypothesis search. In contrast, we learn hypothesis search in a principled fashion that lets us optimize an arbitrary task loss during training, leading to large improvements on classic computer vision tasks. We present two further extensions to NG-RANSAC. Firstly, using the inlier count itself as training signal allows us to train neural guidance in a self-supervised fashion. Secondly, we combine neural guidance with differentiable RANSAC to build neural networks which focus on certain parts of the input data and make the output predictions as good as possible. We evaluate NG-RANSAC on a wide array of computer vision tasks, namely estimation of epipolar geometry, horizon line estimation and camera re-localization. We achieve superior or competitive results compared to state-of-the-art robust estimators, including very recent, learned ones.

background flow map
neural-guided RANSAC


This performance surprisingly differs from the performance for OneMax since, as shown in #REFR , the var-ctrl shows competitive results for OneMax.

For reproducibility, we provide the codes and data in GitHub [20] .

Table 1 lists the average running time to find the full Pareto set for the GSEMO variants, which use multi-objective metrics to guide self-adaptation.

Compared to the GSEMO using static mutation rate, we observe that the two-rate GSEMO presents better results for OneMinMax.

However, the lognormal and var-ctrl use a significantly large number of function evaluations to converge to the full Pareto front.

Therefore, we plot in Figure 1 the average function evaluations to find each Pareto solution.

Because all solutions to OneMinMax are located at the Pareto front, this figure can represent the complete convergence procedure.

We observe that the self-adaptation methods outperform the static setting in the premature optimization stage.

However, the performance of log-normal and var-ctrl deteriorates later in the convergence process.

What Performance Indicators To Use For Self-Adaptation In Multi-Objective Evolutionary Algorithms

Abstract: Parameter control has succeeded in accelerating the convergence process of evolutionary algorithms. Empirical and theoretical studies for classic pseudo-Boolean problems, such as OneMax, Leadin-gOnes, etc., have explained the impact of parameters and helped us understand the behavior of algorithms for single-objective optimization. In this work, by transmitting the techniques of single-objective optimization, we perform an extensive experimental investigation into the behavior of the self-adaptive GSEMO variants. We test three self-adaptive mutation techniques designed for single-objective optimization for the OneMinMax, COCZ, LOTZ, and OneJumpZeroJump problems. While adopting these techniques for the GSEMO algorithm, we consider different performance metrics based on the current non-dominated solution set. These metrics are used to guide the self-adaption process. Our results indicate the benefits of self-adaptation for the tested benchmark problems. We reveal that the choice of metrics significantly affects the performance of the self-adaptive algorithms. The self-adaptation methods based on the progress in one objective can perform better than the methods using multi-objective metrics such as hypervolume, inverted generational distance, and the number of the obtained Pareto solutions. Moreover, we find that the self-adaptive methods benefit from the large population size for OneMinMax and COCZ.

Interpolating Local And Global Search By Controlling The Variance Of Standard Bit Mutation

Abstract: A key property underlying the success of evolutionary algorithms (EAs) is their global search behavior, which allows the algorithms to `jump' from a current state to other parts of the search space, thereby avoiding to get stuck in local optima. This property is obtained through a random choice of the radius at which offspring are sampled from previously evaluated solutions. It is well known that, thanks to this global search behavior, the probability that an EA using standard bit mutation finds a global optimum of an arbitrary function $f:\{0,1\}^n \to \mathbb{R}$ tends to one as the number of function evaluations grows. This advantage over heuristics using a fixed search radius, however, comes at the cost of using non-optimal step sizes also in those regimes in which the optimal rate is stable for a long time. This downside results in significant performance losses for many standard benchmark problems. ::: We introduce in this work a simple way to interpolate between the random global search of EAs and their deterministic counterparts which sample from a fixed radius only. To this end, we introduce \emph{normalized standard bit mutation}, in which the binomial choice of the search radius is replaced by a normal distribution. Normalized standard bit mutation allows a straightforward way to control its variance, and hence the degree of randomness involved. We experiment with a self-adjusting choice of this variance, and demonstrate its effectiveness for the two classic benchmark problems LeadingOnes and OneMax. Our work thereby also touches a largely ignored question in discrete evolutionary computation: multi-dimensional parameter control.


Lifted structure loss #REFR make full use of all the samples in mini-batch and mine the features of all pairs.

It consists of samples of the same class into positive pairs, and different classes of samples into negative pairs.

It requires positive pairs to be close to each other and negative pairs to be far away from each other.

Triplet loss #OTHEREFR takes each sample as a query sample, and selects samples from the same class and different classes to form a triplet.

It requires the distance between negative pairs in the triplet to be greater than that between positive pairs.

Many subsequent studies are carried out on the basis of triplet, such as N-pair loss #OTHEREFR , Beyond Triplet loss #OTHEREFR , etc.

It does not distinguish between query samples or positive samples.

forcing the distance between each sample in the pair and all other negative samples to be greater than the given threshold.

Multi-similarity loss #OTHEREFR can fully utilize both positive and negative pairs by a more generalized weighting strategy.

Concretely, Multi-similarity loss mines informative pairs by the triplet criterion for both positive and negative pairs, then respectively assigns different weights for them.

Circle loss #OTHEREFR use a unified form, which is compatible with both class-level labels and pair-wise labels.

Robust Calibrate Proxy Loss For Deep Metric Learning

Abstract: The mainstream researche in deep metric learning can be divided into two genres: proxy-based and pair-based methods. Proxy-based methods have attracted extensive attention due to the lower training complexity and fast network convergence. However, these methods have limitations as the poxy optimization is done by network, which makes it challenging for the proxy to accurately represent the feature distrubtion of the real class of data. In this paper, we propose a Calibrate Proxy (CP) structure, which uses the real sample information to improve the similarity calculation in proxy-based loss and introduces a calibration loss to constraint the proxy optimization towards the center of the class features. At the same time, we set a small number of proxies for each class to alleviate the impact of intra-class differences on retrieval performance. The effectiveness of our method is evaluated by extensive experiments on three public datasets and multiple synthetic label-noise datasets. The results show that our approach can effectively improve the performance of commonly used proxy-based losses on both regular and noisy datasets.

Deep Metric Learning Via Lifted Structured Feature Embedding

Abstract: Learning the distance metric between pairs of examples is of great importance for learning and visual recognition. With the remarkable success from the state of the art convolutional neural networks, recent works have shown promising results on discriminatively training the networks to learn semantic feature embeddings where similar examples are mapped close to each other and dissimilar examples are mapped farther apart. In this paper, we describe an algorithm for taking full advantage of the training batches in the neural network training by lifting the vector of pairwise distances within the batch to the matrix of pairwise distances. This step enables the algorithm to learn the state of the art feature embedding by optimizing a novel structured prediction objective on the lifted problem. Additionally, we collected Online Products dataset: 120k images of 23k classes of online products for metric learning. Our experiments on the CUB-200-2011, CARS196, and Online Products datasets demonstrate significant improvement over existing deep feature embedding methods on all experimented embedding sizes with the GoogLeNet network.


Oymak and Recht #REFR present a similar result, stated in terms of the Gaussian width 3 of the signal set instead of the box-counting dimension.

This result extends Theorem 2 in #OTHEREFR , which holds for k-sparse sets only, to general low-dimensional sets and is included in the Appendix.

For example, if X is the intersection of L (s + 1)-dimensional subspaces with the unit sphere, Theorem 1 holds with constant 0 = (3 s L) − 1 k−s and k > s.

This theorem tells us that we can recover sparse signals from binary measurements up to an error of

which is sharp, up to the logarithmic factor #OTHEREFR .

Learning To Reconstruct Signals From Binary Measurements

Abstract: Recent advances in unsupervised learning have highlighted the possibility of learning to reconstruct signals from noisy and incomplete linear measurements alone. These methods play a key role in medical and scientific imaging and sensing, where ground truth data is often scarce or difficult to obtain. However, in practice measurements are not only noisy and incomplete but also quantized. Here we explore the extreme case of learning from binary observations and provide necessary and sufficient conditions on the number of measurements required for identifying a set of signals from incomplete binary data. Our results are complementary to existing bounds on signal recovery from binary measurements. Furthermore, we introduce a novel self-supervised learning approach, which we name SSBM, that only requires binary data for training. We demonstrate in a series of experiments with real datasets that SSBM performs on par with supervised learning and outperforms sparse reconstruction methods with a fixed wavelet basis by a large margin.

Near-Optimal Bounds For Binary Embeddings Of Arbitrary Sets

Abstract: We study embedding a subset $K$ of the unit sphere to the Hamming cube $\{-1,+1\}^m$. We characterize the tradeoff between distortion and sample complexity $m$ in terms of the Gaussian width $\omega(K)$ of the set. For subspaces and several structured sets we show that Gaussian maps provide the optimal tradeoff $m\sim \delta^{-2}\omega^2(K)$, in particular for $\delta$ distortion one needs $m\approx\delta^{-2}{d}$ where $d$ is the subspace dimension. For general sets, we provide sharp characterizations which reduces to $m\approx{\delta^{-4}}{\omega^2(K)}$ after simplification. We provide improved results for local embedding of points that are in close proximity of each other which is related to locality sensitive hashing. We also discuss faster binary embedding where one takes advantage of an initial sketching procedure based on Fast Johnson-Lindenstauss Transform. Finally, we list several numerical observations and discuss open problems.


Domain adversarial neural network (DANN) #REFR employs a domain classifier to distinguish whether the input is from the source or target domains.

On the one hand, domain discrepancy is directly computed by metrics like maximum mean discrepancy (MMD) #OTHEREFR .

An adaptation layer with MMD in #OTHEREFR is used to learn domaininvariant deep features.

#OTHEREFR extends this work by proposing the multi-kernel variant of MMD which selects optimal kernels to improve the transferability of features.

In #OTHEREFR , weighted MMD introduces class-specific weights for source domain to address the issue of class bias across domains.

On the other hand, some other works focus on learning domain-invariant features adversarially by using a domain classifier #OTHEREFR .

The feature extractor is connected to the domain classifier by a gradient reversal layer which reserves the gradients during back-propagation.

In #OTHEREFR , the training is divided into two stages, in the first stage, a source model is firstly learnt by only using the source data, then a discriminator along with the pre-trained source model is used for adversarial adaptation to learn the target model.

Weighted-based adversarial methods are also explored in #OTHEREFR .

Re-weighted adversarial adaptation network (RAAN) #OTHEREFR proposes to re-weight source domain label distribution to match the target label distribution by minimizing Earth Mover distance to adapt classifier with adversarial learning.

In #OTHEREFR , an extra domain classifier is used to calculate the weights for source samples.

Cross-Domain Food Image-To-Recipe Retrieval By Weighted Adversarial Learning

Abstract: Food image-to-recipe aims to learn an embedded space linking the rich semantics in recipes with the visual content in food image for cross-modal retrieval. The existing research works carry out the learning of such space by assuming that all the image-recipe training example pairs belong to the same cuisine. As a result, despite the excellent performance reported in the literature, such space is not transferable for retrieving recipes of different cuisine. In this paper, we aim to address this issue by cross-domain food imageto-recipe retrieval, such that by leveraging abundant imagerecipe pairs in source domain (one cuisine), the embedding space is generalizable to a target domain (the other cuisine) that does not have images to pair with recipes for training. With the intuition that the importance of different source samples should vary, this paper proposes two novel mechanisms for cross-domain food image-to-recipe retrieval, i.e., source data selector and weighted cross-modal adversarial learning. The former aims to select source samples similar to the target data and filter out distinctive ones for training. The latter is capable to assign higher weights to the source samples more similar to the target data and lower weights to suppress the distinctive ones for both cross-modal and adversarial learning. The weights are computed from the recipe features extracted from a pre-trained source model. Experiments on three different cuisines (Chuan, Yue and Washoku) demonstrate that the proposed method manages to achieve state-of-the-art performances in all the transfers.

Domain-Adversarial Training Of Neural Networks

Abstract: We introduce a new representation learning approach for domain adaptation, in which data at training and test time come from similar but different distributions. Our approach is directly inspired by the theory on domain adaptation suggesting that, for effective domain transfer to be achieved, predictions must be made based on features that cannot discriminate between the training (source) and test (target) domains. ::: ::: The approach implements this idea in the context of neural network architectures that are trained on labeled data from the source domain and unlabeled data from the target domain (no labeled target-domain data is necessary). As the training progresses, the approach promotes the emergence of features that are (i) discriminative for the main learning task on the source domain and (ii) indiscriminate with respect to the shift between the domains. We show that this adaptation behaviour can be achieved in almost any feed-forward model by augmenting it with few standard layers and a new gradient reversal layer. The resulting augmented architecture can be trained using standard backpropagation and stochastic gradient descent, and can thus be implemented with little effort using any of the deep learning packages. ::: ::: We demonstrate the success of our approach for two distinct classification problems (document sentiment analysis and image classification), where state-of-the-art domain adaptation performance on standard benchmarks is achieved. We also validate the approach for descriptor learning task in the context of person re-identification application.


We also continue the work on invariant generation for a sequential circuit we started in #REFR .

The benefits of property generation by PQE are as follows.

First, by using PQE one can make property generation efficient.

Second, PQE facilitates generation of properties covering different parts of the design, which increases the probability of discovering a bug.

Third, every property generated by PQE specifies a large set of high-quality tests.

We show how PQE can be used to generate properties for a combinational circuit.

One can use invariant generation to identify a bug that makes a state of a sequential circuit unreachable (whereas in a correct design this state should be reachable). Such bugs can be easily missed.

The main body of this paper is structured as follows.

(A full version with all appendices will be published as a technical report.) In Section 2, we give basic definitions. Section 3 presents property generation for a combinational circuit.

In Section 4, we describe invariant generation for a sequential circuit.

Sections 5 and 6 present EG-PQE and EG-PQE + respectively.

Partial Quantifier Elimination And Property Generation

Abstract: We study partial quantifier elimination (PQE) for propositional CNF formulas with existential quantifiers. PQE is a generalization of quantifier elimination where one can limit the set of clauses taken out of the scope of quantifiers to a small subset of clauses. The appeal of PQE is that many verification problems (e.g. equivalence checking and model checking) can be solved in terms of PQE and the latter can be dramatically simpler than full quantifier elimination. We show that PQE can be used for property generation that can be viewed as a generalization of testing. The objective here is to produce an unwanted property of a design implementation thus exposing a bug. We introduce two PQE solvers called EG-PQE and EG-PQE +. EG-PQE is a very simple SAT-based algorithm. EG-PQE + is more sophisticated and robust than EG-PQE. We use these PQE solvers to find an unwanted property (namely, an unwanted invariant) of a buggy FIFO buffer. We also apply them to invariant generation for sequential circuits from a HWMCC benchmark set. Finally, we use these solvers to generate properties of a combinational circuit that mimic symbolic simulation. 2 Basic Definitions In this section, when we say "formula" without mentioning quantifiers, we mean "a quantifier-free formula". Definition 1. We assume that formulas have only Boolean variables. A literal of a variable v is either v or its negation. A clause is a disjunction of literals. A formula F is in conjunctive normal form (CNF) if F = C 1 ∧ • • • ∧ C k where C 1 ,. .. , C k are clauses. We will also view F as a set of clauses {C 1 ,. .. , C k }. We assume that every formula is in CNF unless otherwise stated. Definition 2. Let F be a formula. Then Vars(F) denotes the set of variables of F and Vars(∃X[F ]) denotes Vars(F)\X. Definition 3. Let V be a set of variables. An assignment #» q to V is a mapping V ′ → {0, 1} where V ′ ⊆ V. We will denote the set of variables assigned in #» q as Vars(q). We will refer to #» q as a full assignment to V if Vars(q) = V. We will denote as #» q ⊆ #» r the fact that a) Vars(q) ⊆ Vars(r) and b) every variable of Vars(q) has the same value in #» q and #» r. Definition 4. A literal and a clause are said to be satisfied (respectively falsified) by an assignment #» q if they evaluate to 1 (respectively 0) under #» q. Definition 5. Let C be a clause. Let H be a formula that may have quantifiers, and #» q be an assignment to Vars(H). If C is satisfied by #» q , then C q ≡ 1. Otherwise, C q is the clause obtained from C by removing all literals falsified by #» q. Denote by H q the formula obtained from H by removing the clauses satisfied by #» q and replacing every clause C unsatisfied by #» q with C q. Definition 6. Given a formula ∃X[F (X, Y)], a clause C of F is called a quantified clause if Vars(C) ∩ X = ∅. If Vars(C) ∩ X = ∅, the clause C depends only on free i.e. unquantified variables of F and is called a free clause. Definition 7. Let G, H be formulas that may have existential quantifiers. We say that G, H are equivalent, written G ≡ H, if G q = H q for all full assignments #» q to Vars(G) ∪ Vars(H).

Partial Quantifier Elimination By Certificate Clauses

Abstract: We study a modification of the Quantifier Elimination (QE) problem called Partial QE (PQE) for propositional CNF formulas. In PQE, only a small subset of target clauses is taken out of the scope of quantifiers. The appeal of PQE is that many verification problems, e.g. equivalence checking and model checking, reduce to PQE and, intuitively, the latter should be much easier than QE. One can perform PQE by adding a set of clauses depending only on free variables that make the target clauses redundant. Proving redundancy of a target clause is done by derivation of a"certificate"clause implying the former. This idea is implemented in our PQE algorithm called START. It bears some similarity to a SAT-solver with conflict driven learning. A major difference here is that START backtracks when a target clause is proved redundant in the current subspace (a conflict being just one of backtracking conditions). We experimentally evaluate START on a practical problem. We use this problem to show that PQE can be much easier than QE.


Inspired by AGP #REFR , L 2 filter pruning is used in our iterative pruning method for model training.

Finally, we iteratively pruned the reparameterized student network S: where Φ is the pruning operation, r is the pruning rate, ϕ means the finetuning operation, S i p means the network after the i-th pruning.

We stop pruning until the network cannot make effective predictions, and use the network obtained from the last effective pruning as the final network.

Dipnet: Efficiency Distillation And Iterative Pruning For Image Super-Resolution

Abstract: Efficient deep learning-based approaches have achieved remarkable performance in single image super-resolution. However, recent studies on efficient super-resolution have mainly focused on reducing the number of parameters and floating-point operations through various network designs. Although these methods can decrease the number of parameters and floating-point operations, they may not necessarily reduce actual running time. To address this issue, we propose a novel multi-stage lightweight network boosting method, which can enable lightweight networks to achieve outstanding performance. Specifically, we leverage enhanced high-resolution output as additional supervision to improve the learning ability of lightweight student networks. Upon convergence of the student network, we further simplify our network structure to a more lightweight level using reparameterization techniques and iterative network pruning. Meanwhile, we adopt an effective lightweight network training strategy that combines multi-anchor distillation and progressive learning, enabling the lightweight network to achieve outstanding performance. Ultimately, our proposed method achieves the fastest inference time among all participants in the NTIRE 2023 efficient super-resolution challenge while maintaining competitive super-resolution performance. Additionally, extensive experiments are conducted to demonstrate the effectiveness of the proposed components. The results show that our approach achieves comparable performance in representative dataset DIV2K, both qualitatively and quantitatively, with faster inference and fewer number of network parameters.

To Prune, Or Not To Prune: Exploring The Efficacy Of Pruning For Model Compression

Abstract: Model pruning seeks to induce sparsity in a deep neural network's various connection matrices, thereby reducing the number of nonzero-valued parameters in the model. Recent reports (Han et al., 2015; Narang et al., 2017) prune deep networks at the cost of only a marginal loss in accuracy and achieve a sizable reduction in model size. This hints at the possibility that the baseline models in these experiments are perhaps severely over-parameterized at the outset and a viable alternative for model compression might be to simply reduce the number of hidden units while maintaining the model's dense connection structure, exposing a similar trade-off in model size and accuracy. We investigate these two distinct paths for model compression within the context of energy-efficient inference in resource-constrained environments and propose a new gradual pruning technique that is simple and straightforward to apply across a variety of models/datasets with minimal tuning and can be seamlessly incorporated within the training process. We compare the accuracy of large, but pruned models (large-sparse) and their smaller, but dense (small-dense) counterparts with identical memory footprint. Across a broad range of neural network architectures (deep CNNs, stacked LSTM, and seq2seq LSTM models), we find large-sparse models to consistently outperform small-dense models and achieve up to 10x reduction in number of non-zero parameters with minimal loss in accuracy.


In the pseudocode, we also take MAN #REFR as the backbone example for a better explanation.

Our method serves as a plug-and-play solution for MDL and can be applied to different SP models.

We provide the pseudocode for our MDAL in Algorithm 1.

Multi-Domain Learning From Insufficient Annotations

Abstract: Multi-domain learning (MDL) refers to simultaneously constructing a model or a set of models on datasets collected from different domains. Conventional approaches emphasize domain-shared information extraction and domain-private information preservation, following the shared-private framework (SP models), which offers significant advantages over single-domain learning. However, the limited availability of annotated data in each domain considerably hinders the effectiveness of conventional supervised MDL approaches in real-world applications. In this paper, we introduce a novel method called multi-domain contrastive learning (MDCL) to alleviate the impact of insufficient annotations by capturing both semantic and structural information from both labeled and unlabeled data. Specifically, MDCL comprises two modules: inter-domain semantic alignment and intra-domain contrast. The former aims to align annotated instances of the same semantic category from distinct domains within a shared hidden space, while the latter focuses on learning a cluster structure of unlabeled instances in a private hidden space for each domain. MDCL is readily compatible with many SP models, requiring no additional model parameters and allowing for end-to-end training. Experimental results across five textual and image multi-domain datasets demonstrate that MDCL brings noticeable improvement over various SP models. Furthermore, MDCL can further be employed in multi-domain active learning (MDAL) to achieve a superior initialization, eventually leading to better overall performance.

Multinomial Adversarial Networks For Multi-Domain Text Classification

Abstract: Many text classification tasks are known to be highly domain-dependent. Unfortunately, the availability of training data can vary drastically across domains. Worse still, for some domains there may not be any annotated data at all. In this work, we propose a multinomial adversarial network (MAN) to tackle the text classification problem in this real-world multidomain setting (MDTC). We provide theoretical justifications for the MAN framework, proving that different instances of MANs are essentially minimizers of various f-divergence metrics (Ali and Silvey, 1966) among multiple probability distributions. MANs are thus a theoretically sound generalization of traditional adversarial networks that discriminate over two distributions. More specifically, for the MDTC task, MAN learns features that are invariant across multiple domains by resorting to its ability to reduce the divergence among the feature distributions of each domain. We present experimental results showing that MANs significantly outperform the prior art on the MDTC task. We also show that MANs achieve state-of-the-art performance for domains with no labeled data.


Theorem 1 states that by using prioritized sampling, the loss function is equivalent to an L2 loss for the non-zero loss portion, while the loss function is an L1 loss when using uniform sampling, as in #REFR . Theorem 2.

Proof.
EQUATION
Then, setting
EQUATION
completes the proof.

Define the absolute prediction error (APE) for the i-th data point in the replay buffer under L1 loss with learning rate η ∈ R + at time t as

and under L2 loss with learning rate η at time t as

Data-Efficient Control Barrier Function Refinement

Abstract: Control barrier functions (CBFs) have been widely used for synthesizing controllers in safety-critical applications. When used as a safety filter, it provides a simple and computationally efficient way to obtain safe controls from a possibly unsafe performance controller. Despite its conceptual simplicity, constructing a valid CBF is well known to be challenging, especially for high-relative degree systems under nonconvex constraints. Recently, work has been done to learn a valid CBF from data based on a handcrafted CBF (HCBF). Even though the HCBF gives a good initialization point, it still requires a large amount of data to train the CBF network. In this work, we propose a new method to learn more efficiently from the collected data through a novel prioritized data sampling strategy. A priority score is computed from the loss value of each data point. Then, a probability distribution based on the priority score of the data points is used to sample data and update the learned CBF. Using our proposed approach, we can learn a valid CBF that recovers a larger portion of the true safe set using a smaller amount of data. The effectiveness of our method is demonstrated in simulation on a unicycle and a two-link arm.

Learning Control Barrier Functions From Expert Demonstrations

Abstract: Inspired by the success of imitation and inverse reinforcement learning in replicating expert behavior through optimal control, we propose a learning based approach to safe controller synthesis based on control barrier functions (CBFs). We consider the setting of a known nonlinear control affine dynamical system and assume that we have access to safe trajectories generated by an expert - a practical example of such a setting would be a kinematic model of a self-driving vehicle with safe trajectories (e.g. trajectories that avoid collisions with obstacles in the environment) generated by a human driver. We then propose and analyze an optimization-based approach to learning a CBF that enjoys provable safety guarantees under suitable Lipschitz smoothness assumptions on the underlying dynamical system. A strength of our approach is that it is agnostic to the parameterization used to represent the CBF, assuming only that the Lipschitz constant of such functions can be efficiently bounded. Furthermore, if the CBF parameterization is convex, then under mild assumptions, so is our learning process. We end with extensive numerical evaluations of our results on both planar and realistic examples, using both random feature and deep neural network parameterizations of the CBF. To the best of our knowledge, these are the first results that learn provably safe control barrier functions from data.


Algorithm 1 Consistent Prediction-Based Rejection

We note that prior works such as #REFR and Tramèr (2022) have also proposed the idea of using the consistency check for rejecting inputs to a classifier (equivalent to rejecting inputs based on their distance to the decision boundary).

The details of our CPR defense are given in Algorithm 1.

Note that the PGD method in the algorithm is deterministic.

Thus, the mapping T (x) = x m is deterministic and we can summarize the defense as: if y(T (x)) = y(x), then reject x ; otherwise accept x and output the predicted class y(x).

The robust base model for our defense h can be trained using methods such as standard adversarial training (AT) #OTHEREFR , TRADES (Zhang et al., 2019) , and MART (Wang et al., 2020) (we will focus on the first two methods).

An advantage of the proposed defense is that it is agnostic of the rejection loss function used for evaluation, and therefore can generalize to multiple choices of the rejection loss function (see Section 5).

However, we are the first to systematically implement and evaluate such a defense, and propose strong adaptive attacks for evaluating it.

Stratified Adversarial Robustness With Rejection

Abstract: Recently, there is an emerging interest in adversarially training a classifier with a rejection option (also known as a selective classifier) for boosting adversarial robustness. While rejection can incur a cost in many applications, existing studies typically associate zero cost with rejecting perturbed inputs, which can result in the rejection of numerous slightly-perturbed inputs that could be correctly classified. In this work, we study adversarially-robust classification with rejection in the stratified rejection setting, where the rejection cost is modeled by rejection loss functions monotonically non-increasing in the perturbation magnitude. We theoretically analyze the stratified rejection setting and propose a novel defense method-Adversarial Training with Consistent Prediction-based Rejection (CPR)-for building a robust selective classifier. Experiments on image datasets demonstrate that the proposed method significantly outperforms existing methods under strong adaptive attacks. For instance, on CIFAR-10, CPR reduces the total robust loss (for different rejection losses) by at least 7.3% under both seen and unseen attacks.

Playing It Safe: Adversarial Robustness With An Abstain Option

Abstract: We explore adversarial robustness in the setting in which it is acceptable for a classifier to abstain---that is, output no class---on adversarial examples. Adversarial examples are small perturbations of normal inputs to a classifier that cause the classifier to give incorrect output; they present security and safety challenges for machine learning systems. In many safety-critical applications, it is less costly for a classifier to abstain on adversarial examples than to give incorrect output for them. We first introduce a novel objective function for adversarial robustness with an abstain option which characterizes an explicit tradeoff between robustness and accuracy. We then present a simple baseline in which an adversarially-trained classifier abstains on all inputs within a certain distance of the decision boundary, which we theoretically and experimentally evaluate. Finally, we propose Combined Abstention Robustness Learning (CARL), a method for jointly learning a classifier and the region of the input space on which it should abstain. We explore different variations of the PGD and DeepFool adversarial attacks on CARL in the abstain setting. Evaluating against these attacks, we demonstrate that training with CARL results in a more accurate, robust, and efficient classifier than the baseline.


MAVEN #REFR uses a hierarchical architecture to generate a shared latent vector for each agent to explore the space cooperatively.

The Distributional Predict Error (DPE) algorithm #OTHEREFR utilizes Random Network Distillation #OTHEREFR to generate two identical architectures with randomly initialized parameters and use their Wasserstein distance to measure an action's novelty given a state.

Although these methods are effective in the single-agent domain, naïvely applying them separately and independently to each agent in the MARL setting is bound to result in suboptimal performance.

This is because the agents mutually influence one another, creating additional uncertainty that has to be taken into account.

One way of accounting for such uncertainty is to require cooperative behavior between agents.

Recently, there have been some works on ensuring cooperation between the agents.

CMAE #OTHEREFR creates an exploration compartment for each agent that is not shared with other agents, drastically reducing the searching space via cooperative behavior.

However, none of these algorithms account for inherent uncertainty and are applied to other MARL algorithms.

Toward Risk-Based Optimistic Exploration For Cooperative Multi-Agent Reinforcement Learning

Abstract: The multi-agent setting is intricate and unpredictable since the behaviors of multiple agents influence one another. To address this environmental uncertainty, distributional reinforcement learning algorithms that incorporate uncertainty via distributional output have been integrated with multi-agent reinforcement learning (MARL) methods, achieving state-of-the-art performance. However, distributional MARL algorithms still rely on the traditional-greedy, which does not take cooperative strategy into account. In this paper, we present a risk-based exploration that leads to collaboratively optimistic behavior by shifting the sampling region of distribution. Initially, we take expectations from the upper quantiles of state-action values for exploration, which are optimistic actions, and gradually shift the sampling region of quantiles to the full distribution for exploitation. By ensuring that each agent is exposed to the same level of risk, we can force them to take cooperatively optimistic actions. Our method shows remarkable performance in multi-agent settings requiring cooperative exploration based on quantile regression appropriately controlling the level of risk.

Maven: Multi-Agent Variational Exploration

Abstract: Centralised training with decentralised execution is an important setting for cooperative deep multi-agent reinforcement learning due to communication constraints during execution and computational tractability in training. In this paper, we analyse value-based methods that are known to have superior performance in complex environments. We specifically focus on QMIX, the current state-of-the-art in this domain. We show that the representation constraints on the joint action-values introduced by QMIX and similar methods lead to provably poor exploration and suboptimality. Furthermore, we propose a novel approach called MAVEN that hybridises value and policy-based methods by introducing a latent space for hierarchical control. The value-based agents condition their behaviour on the shared latent variable controlled by a hierarchical policy. This allows MAVEN to achieve committed, temporally extended exploration, which is key to solving complex multi-agent tasks. Our experimental results show that MAVEN achieves significant performance improvements on the challenging SMAC domain.

shared latent vector
hierarchical architecture


For example, KL-PN #REFR proposes the Prior Networks (PN) trained with two KL divergence terms.

Dirichlet-based uncertainty models (DBU) predict the parameters of the Dirichlet distribution, which allows the computation of closed-form classical uncertainty metrics such as differential entropy, mutual information, etc.

These metrics can be used to distinguish among data, model, and distributional uncertainty.

Different DBU models differ in the parameterization and training strategy of the Dirichlet distribution.

The first term is used to learn sharp Dirichlet parameters for ID data, while the other learns flat Dirichlet parameters for OOD data.

Since the forward KL divergence is zero-avoiding, RKL-PN #OTHEREFR introduces the reverse KL divergence to avoid undesired multimodal target distributions.

Posterior Network (PostN) #OTHEREFR uses Normalizing Flows to predict the posterior distribution of any input sample without training with OOD data.

Evidential Deep Learning (EDL) #OTHEREFR treats the network's outputs as belief masses based on the Dempster-Shafer Theory of Evidence (DST) #OTHEREFR and derives the loss function using subjective logic #OTHEREFR .

Moreover, deep evidential regression #OTHEREFR introduces evidential priors over the original Gaussian likelihood function to model the uncertainty of regression networks. #OTHEREFR

Uncertainty Estimation By Fisher Information-Based Evidential Deep Learning

Abstract: Uncertainty estimation is a key factor that makes deep learning reliable in practical applications. Recently proposed evidential neural networks explicitly account for different uncertainties by treating the network's outputs as evidence to parameterize the Dirichlet distribution, and achieve impressive performance in uncertainty estimation. However, for high data uncertainty samples but annotated with the one-hot label, the evidencelearning process for those mislabeled classes is over-penalized and remains hindered. To address this problem, we propose a novel method, Fisher Information-based Evidential Deep Learning (I-EDL). In particular, we introduce Fisher Information Matrix (FIM) to measure the informativeness of evidence carried by each sample, according to which we can dynamically reweight the objective loss terms to make the network more focus on the representation learning of uncertain classes. The generalization ability of our network is further improved by optimizing the PAC-Bayesian bound. As demonstrated empirically, our proposed method consistently outperforms traditional EDL-related algorithms in multiple uncertainty estimation tasks, especially in the more challenging few-shot classification settings.

Predictive Uncertainty Estimation Via Prior Networks

Abstract: Estimating uncertainty is important to improving the safety of AI systems. Recently baseline tasks and metrics have been defined and several practical methods for estimating uncertainty developed. However, these approaches attempt to model distributional uncertainty either implicitly through model uncertainty or as data uncertainty. This work proposes a new framework for modeling predictive uncertainty called Prior Networks (PNs) which explicitly models distributional uncertainty. PNs do this by parameterizing a prior distribution over predictive distributions. This work focuses on uncertainty for classification and evaluates PNs on the tasks of identifying out-of-distribution (OOD) samples and detecting misclassification on the MNIST dataset, where they are found to outperform previous methods. Experiments on synthetic and MNIST data show that unlike previous methods PNs are able to distinguish between data and distributional uncertainty.


Following #REFR , we define a class centroid c j ∈ R d in the embedding space for each class j in the support set S.

The centroids are learnable variables but initialized by standard class prototypes #OTHEREFR .

That is, the centroid c j of class j is initialized by the mean

of support embeddings assigned to class j.

Adaptive Manifold For Imbalanced Transductive Few-Shot Learning

Abstract: Transductive few-shot learning algorithms have showed substantially superior performance over their inductive counterparts by leveraging the unlabeled queries. However, the vast majority of such methods are evaluated on perfectly class-balanced benchmarks. It has been shown that they undergo remarkable drop in performance under a more realistic, imbalanced setting. To this end, we propose a novel algorithm to address imbalanced transductive few-shot learning, named Adaptive Manifold. Our method exploits the underlying manifold of the labeled support examples and unlabeled queries by using manifold similarity to predict the class probability distribution per query. It is parameterized by one centroid per class as well as a set of graph-specific parameters that determine the manifold. All parameters are optimized through a loss function that can be tuned towards class-balanced or imbalanced distributions. The manifold similarity shows substantial improvement over Euclidean distance, especially in the 1-shot setting. Our algorithm outperforms or is on par with other state of the art methods in three benchmark datasets, namely miniImageNet, tieredImageNet and CUB, and three different backbones, namely ResNet-18, WideResNet-28-10 and DenseNet-121. In certain cases, our algorithm outperforms the previous state of the art by as much as 4.2%.

Prototypical Networks For Few-Shot Learning

Abstract: A recent approach to few-shot classification called matching networks has demonstrated the benefits of coupling metric learning with a training procedure that mimics test. This approach relies on a complicated fine-tuning procedure and an attention scheme that forms a distribution over all points in the support set, scaling poorly with its size. We propose a more streamlined approach, prototypical networks, that learns a metric space in which few-shot classification can be performed by computing Euclidean distances to prototype representations of each class, rather than individual points. Our method is competitive with state-of-the-art one-shot classification approaches while being much simpler and more scalable with the size of the support set. We empirically demonstrate the performance of our approach on the Omniglot and mini-ImageNet datasets. We further demonstrate that a similar idea can be used for zero-shot learning, where each class is described by a set of attributes, and achieve state-of-the-art results on the Caltech UCSD bird dataset.


These artifacts can include a human setting initial goals, providing training data, or providing an initial configurations for components #REFR .

We consider computational notebooks #OTHEREFR to be artifacts possessing such characteristics, as they are intended to be be shared with other technical data workers or simply with the data scientist themselves.

Artifacts with distribution (c4.4.2) characteristics are those that aim to have a wide audience.

These can include presentations or reports, many of which rely on the compilation and analysis of other AutoML artifacts.

• Category 4.5: Steering Artifacts can act to orient or intervene within an AutoML process.

Artifacts with directing (4.5.1) act to initialize or orient the AutoML processes in a specific trajectory.

They can also include algorithmic processes that define the final configuration of AutoML components.

In contrast, artifacts with refining (c4.5.2) characteristics are subtler than directing as they largely build off of the existing model structure.

Fine-tuning operations, example Transfer Learning #OTHEREFR 117] , produce artifacts for refining an existing neural architecture #OTHEREFR , rather than conducting an expensive neural architecture search #OTHEREFR .

We consider the former to be a refinement, while the latter uses an initial design space configuration to direct the overall neural architecture search process.

Tracing And Visualizing Human-Ml/Ai Collaborative Processes Through Artifacts Of Data Work

Abstract: Artifact Identification and Classification Capture Artifacts Artifact Properties Dependencies Artifact Lineage Model Customization Interaction Version 3 Source files changed Version 2 Source files unchanged Version 1 of artifact Figure 1: We developed an artifact taxonomy that captures both human and ML/AI processes in automated data work. We assess our taxonomy's utility through a collaboration with an enterprise software development team creating an AutoML system. .

Taking Human Out Of Learning Applications: A Survey On Automated Machine Learning

Abstract: Machine learning techniques have deeply rooted in our everyday life. However, since it is knowledge- and labor-intensive to pursue good learning performance, human experts are heavily involved in every aspect of machine learning. In order to make machine learning techniques easier to apply and reduce the demand for experienced human experts, automated machine learning (AutoML) has emerged as a hot topic with both industrial and academic interest. In this paper, we provide an up to date survey on AutoML. First, we introduce and define the AutoML problem, with inspiration from both realms of automation and machine learning. Then, we propose a general AutoML framework that not only covers most existing approaches to date but also can guide the design for new methods. Subsequently, we categorize and review the existing works from two aspects, i.e., the problem setup and the employed techniques. Finally, we provide a detailed analysis of AutoML approaches and explain the reasons underneath their successful applications. We hope this survey can serve as not only an insightful guideline for AutoML beginners but also an inspiration for future research.


To improve the characterization capabilities of the teacher branch further, we adopt a structure similar to CSRNet #REFR by incorporating dilated convolutional layers in the back-end.

It is important to note that the shared module also serves as the initial layers of the student branch, so it should not have too many parameters.

In other words, we should avoid introducing too many shallow layers in the teacher branch when designing the shared module.

2) Teacher branch: In addition to using the first two layers of VGG-16 in the shared shallow module, we employ the remaining layers of the first 10 layers of VGG-16 as the frontend of the teacher branch.

This allows us to utilize the pretrained model, which has strong transfer learning capabilities and a flexible design.

The teacher branch is intended to act as an experienced mentor who can quickly learn new information and guide the student branch.

This type of convolutional layers can directly enhance the network receptive field without increasing its parameters or computational complexity.

Specifically, we use a light version of CSRNet consisting of 3 dilated convolutional layers.

3) Student branch: The channel capacity of a network is a critical factor that affects both inference speed and memory consumption.

In our approach, we limit the number of channels in the teacher branch to approximately one-fourth of its original size to reduce the computational burden.

However, following the design principle of VGG networks, reducing the number of channels directly to one-fourth may limit the network's ability to express shallow features.

Crowd Counting With Online Knowledge Learning

Abstract: Efficient crowd counting models are urgently required for the applications in scenarios with limited computing resources, such as edge computing and mobile devices. A straightforward method to achieve this is knowledge distillation (KD), which involves using a trained teacher network to guide the training of a student network. However, this traditional twophase training method can be time-consuming, particularly for large datasets, and it is also challenging for the student network to mimic the learning process of the teacher network. To overcome these challenges, we propose an online knowledge learning method for crowd counting. Our method builds an end-to-end training framework that integrates two independent networks into a single architecture, which consists of a shared shallow module, a teacher branch, and a student branch. This approach is more efficient than the two-stage training technique of traditional KD. Moreover, we propose a feature relation distillation method which allows the student branch to more effectively comprehend the evolution of inter-layer features by constructing a new interlayer relationship matrix. It is combined with response distillation and feature internal distillation to enhance the transfer of mutually complementary information from the teacher branch to the student branch. Extensive experiments on four challenging crowd counting datasets demonstrate the effectiveness of our method which achieves comparable performance to state-of-theart methods despite using far fewer parameters.

Csrnet: Dilated Convolutional Neural Networks For Understanding The Highly Congested Scenes

Abstract: We propose a network for Congested Scene Recognition called CSRNet to provide a data-driven and deep learning method that can understand highly congested scenes and perform accurate count estimation as well as present high-quality density maps. The proposed CSRNet is composed of two major components: a convolutional neural network (CNN) as the front-end for 2D feature extraction and a dilated CNN for the back-end, which uses dilated kernels to deliver larger reception fields and to replace pooling operations. CSRNet is an easy-trained model because of its pure convolutional structure. We demonstrate CSRNet on four datasets (ShanghaiTech dataset, the UCF_CC_50 dataset, the WorldEXPO'10 dataset, and the UCSD dataset) and we deliver the state-of-the-art performance. In the ShanghaiTech Part_B dataset, CSRNet achieves 47.3% lower Mean Absolute Error (MAE) than the previous state-of-the-art method. We extend the targeted applications for counting other objects, such as the vehicle in TRANCOS dataset. Results show that CSRNet significantly improves the output quality with 15.4% lower MAE than the previous state-of-the-art approach.