As we can see in Table 10 , BERT #REFR have the highest Precision among the models we have tested.

We have used three different pretrained models and collected their Precision, Recall and F1 score.

BERT has Precision value of 0.92, compared to Precision value of 0.91 for both BERT Tiny and GPT #OTHEREFR models.

In case of identifying the fraction of relevant items that can be retrieved, i.e. the Recall score,

Language Model Precision Recall F1 score BERT Tiny 0.91 0.82 0.86 GPT #OTHEREFR 0.91 0.80 0.85 BERT 0.92 0.81 0.86 Table 10 .

Model characterization with pre-trained language models BERT Tiny performs slightly higher 0.82 than the BERT model 0.81.

GPT model has the lowest Recall score of 0.80 among these three models.

Large Language Models Based Automatic Synthesis Of Software Specifications

Abstract: Software configurations play a crucial role in determining the behavior of software systems. In order to ensure safe and error-free operation, it is necessary to identify the correct configuration, along with their valid bounds and rules, which are commonly referred to as software specifications. As software systems grow in complexity and scale, the number of configurations and associated specifications required to ensure the correct operation can become large and prohibitively difficult to manipulate manually. Due to the fast pace of software development, it is often the case that correct software specifications are not thoroughly checked or validated within the software itself. Rather, they are frequently discussed and documented in a variety of external sources, including software manuals, code comments, and online discussion forums. Therefore, it is hard for the system administrator to know the correct specifications of configurations due to the lack of clarity, organization, and a centralized unified source to look at. To address this challenge, we propose SpecSyn a framework that leverages a state-of-the-art large language model to automatically synthesize software specifications from natural language sources. Our approach formulates software specification synthesis as a sequence-to-sequence learning problem and investigates the extraction of specifications from large contextual texts. This is the first work that uses a large language model for end-to-end specification synthesis from natural language texts. Empirical results demonstrate that our system outperforms prior state-of-the-art specification synthesis tool by 21% in terms of F1 score and can find specifications from single as well as multiple sentences.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


Some notable contributions in the field of flow-based models include the Real-valued Non-Volume-Preserving (Real NVP) model, introduced by Dinh et al #REFR , which proposed a flexible and efficient way to construct invertible transformations using neural networks.

#OTHEREFR firstly introduced GANs into pansharpening and performed unsupervised learning of spectral and spatial information based on the reversed loss function upon the network's output.

Zero-GAN #OTHEREFR also relies on GAN and constructed a non-reference loss function, including an adversarial loss, spatial and spectral reconstruction losses, a spatial enhancement loss, and an average constancy loss to specifically supervise the learning process.

Flow-based model is a type of generative model that learns the underlying distribution of data by transforming a simple input distribution into the target distribution through a series of invertible transformations.

This is achieved by representing the probability density function of the target distribution as a sequence of flows, where each flow is an invertible transformation.

Another important contribution is Glow #OTHEREFR , which improved upon previous flow-based models by introducing an invertible 1×1 convolutional operation that greatly increased the expressiveness of the model.

Diffusion model are recently proposed for generation including conditional or unconditional generation #OTHEREFR , #OTHEREFR , #OTHEREFR , text-to-image translation #OTHEREFR , #OTHEREFR , image superresolution #OTHEREFR , image restoration #OTHEREFR , #OTHEREFR and other highlevel image manipulation tasks #OTHEREFR , #OTHEREFR , #OTHEREFR .

Diffusion models can generate images with more details and higher fidelity than the GAN-based model and flow-based model.

The advantages of diffusion models include stable training, minimal mode collapse, and the ability to train with only a single mean squared error (MSE) loss.

In comparison to GAN-based models, which suffer from instability issues in their adversarial training, and flow-based models, which are limited in their network performance due to the requirement for reversibility, diffusion models are easier to train and design.

Ddrf: Denoising Diffusion Model For Remote Sensing Image Fusion

Abstract: Denosing diffusion model, as a generative model, has received a lot of attention in the field of image generation recently, thanks to its powerful generation capability. However, diffusion models have not yet received sufficient research in the field of image fusion. In this article, we introduce diffusion model to the image fusion field, treating the image fusion task as imageto-image translation and designing two different conditional injection modulation modules (i.e., style transfer modulation and wavelet modulation) to inject coarse-grained style information and fine-grained high-frequency and low-frequency information into the diffusion UNet, thereby generating fused images. In addition, we also discussed the residual learning and the selection of training objectives of the diffusion model in the image fusion task. Extensive experimental results based on quantitative and qualitative assessments compared with benchmarks demonstrates state-of-the-art results and good generalization performance in image fusion tasks. Finally, it is hoped that our method can inspire other works and gain insight into this field to better apply the diffusion model to image fusion tasks. Code shall be released for better reproducibility.

Density Estimation Using Real Nvp

Abstract: Unsupervised learning of probabilistic models is a central yet challenging problem in machine learning. Specifically, designing models with tractable learning, sampling, inference and evaluation is crucial in solving this task. We extend the space of such models using real-valued non-volume preserving (real NVP) transformations, a set of powerful invertible and learnable transformations, resulting in an unsupervised learning algorithm with exact log-likelihood computation, exact sampling, exact inference of latent variables, and an interpretable latent space. We demonstrate its ability to model natural images on four datasets through sampling, log-likelihood evaluation and latent variable manipulations.

neural networks
Real-valued Non-Volume-Preserving (Real


Again, expressions (3.11) and (3.12) yield highly accurate numerical values using Bornemann's approach #REFR .

For a random matrix that has its eigenvalue n-point correlation function given as in Proposition 1.1 with a kernel K, the CDF and PDF of the second largest eigenvalue λ 2 are

where the kernel K (s) is given as (1.4).

Proposition 3.2 could be used with the Airy kernel (soft-edge) as well as finite N kernels like the Hermite kernel.

In the hard-edge scaling the interval (0, s) is used instead of (s, ∞) with the Bessel kernel for the second smallest eigenvalue.

Table 4 is the computed first four moments of the two smallest eigenvalues (λ 1 < λ 2 ) in the hard-edge and soft-edge using PDF formulas of Section 3.2 and (3.12).

The Conditional Dpp Approach To Random Matrix Distributions

Abstract: We present the conditional determinantal point process (DPP) approach to obtain new (mostly Fredholm determinantal) expressions for various eigenvalue statistics in random matrix theory. It is well-known that many (especially β = 2) eigenvalue n-point correlation functions are given in terms of n × n determinants, i.e., they are continuous DPPs. We exploit a derived kernel of the conditional DPP which gives the n-point correlation function conditioned on the event of some eigenvalues already existing at fixed locations. Using such kernels we obtain new determinantal expressions for the joint densities of the k largest eigenvalues, probability density functions of the k th largest eigenvalue, density of the first eigenvalue spacing, and more. Our formulae are highly amenable to numerical computations and we provide various numerical experiments. Several numerical values that required hours of computing time could now be computed in seconds with our expressions, which proves the effectiveness of our approach. We also demonstrate that our technique can be applied to an efficient sampling of DR paths of the Aztec diamond domino tiling. Further extending the conditional DPP sampling technique, we sample Airy processes from the extended Airy kernel. Additionally we propose a sampling method for non-Hermitian projection DPPs.

On The Numerical Evaluation Of Fredholm Determinants

Abstract: Some significant quantities in mathematics and physics are most naturally expressed as the Fredholm determinant of an integral operator, most notably many of the distribution functions in random matrix theory. Though their numerical values are of interest, there is no systematic numerical treatment of Fredholm determinants to be found in the literature. Instead, the few numerical evaluations that are available rely on eigenfunction expansions of the operator, if expressible in terms of special functions, or on alternative, numerically more straightforwardly accessible analytic expressions, e.g., in terms of Painleve transcendents, that have masterfully been derived in some cases. In this paper we close the gap in the literature by studying projection methods and, above all, a simple, easily implementable, general method for the numerical evaluation of Fredholm determinants that is derived from the classical Nystrom method for the solution of Fredholm equations of the second kind. Using Gauss-Legendre or Clenshaw-Curtis as the underlying quadrature rule, we prove that the approximation error essentially behaves like the quadrature error for the sections of the kernel. In particular, we get exponential convergence for analytic kernels, which are typical in random matrix theory. The application of the method to the distribution functions of the Gaussian unitary ensemble (GUE), in the bulk and the edge scaling limit, is discussed in detail. After extending the method to systems of integral operators, we evaluate the two-point correlation functions of the more recently studied Airy and Airy1 processes for the first time.


Therefore, we propose the following theorem to further approximate the CRLBs in #REFR , thus obtaining the simplified expressions.

By substituting #OTHEREFR into 23, we have

Thus, the aged CRLBs on the predicted MSEs of θ k,n , distance d k,n , and

However, considering the matrix multiplication of polynomials inG k,n−1 andḠ k,n,i , it is still hard to apply the aged radar tracking performance in #OTHEREFR to study the resource allocation scheme design.

Theorem 3.2: The aged CRLBs on the predicted MSEs of θ k,n , distance d k,n , and velocity v k,n for U R k in block n derived in (27) can be approximated by

a k,n = 1 + n−1 i=1v k,i T d k,i cos(θ k,i ) 2 , b k,n = 1 + n−1 i=2 1 + n−1 i ′ =iv k,i ′ T d k,i ′ cos(θ k,i ′ ) 2 , andθ k,n =θ k,n −ϕ k .

Impact Of Channel Aging On Dual-Function Radar-Communication Systems: Performance Analysis And Resource Allocation

Abstract: In conventional dual-function radar-communication (DFRC) systems, the radar and communication channels are routinely estimated at fixed time intervals based on their worst-case operation scenarios. Such situation-agnostic repeated estimations cause significant training overhead and dramatically degrade the system performance, especially for applications with dynamic sensing/communication demands and limited radio resources. In this paper, we leverage the channel aging characteristics to reduce training overhead and to design a situation-dependent channel re-estimation interval optimization-based resource allocation for performance improvement in a multi-target tracking DFRC system. Specifically, we exploit the channel temporal correlation to predict radar and communication channels for reducing the need of training preamble retransmission. Then, we characterize the channel aging effects on the Cramer-Rao lower bounds (CRLBs) for radar tracking performance analysis and achievable rates with maximum ratio transmission (MRT) and zero-forcing (ZF) transmit beamforming for communication performance analysis. In particular, the aged CRLBs and achievable rates are derived as closed-form expressions with respect to the channel aging time, bandwidth, and power. Based on the analyzed results, we optimize these factors to maximize the average total aged achievable rate subject to individual target tracking precision demand, communication rate requirement, and other practical constraints. Since the formulated problem belongs to a non-convex problem, we develop an efficient one-dimensional search based optimization algorithm to obtain its suboptimal solutions. Finally, simulation results are presented to validate the correctness of the derived theoretical results and the effectiveness of the proposed allocation scheme.

Radar-Assisted Predictive Beamforming For Vehicular Links: Communication Served By Sensing

Abstract: In vehicular networks of the future, sensing and communication functionalities will be intertwined. In this paper, we investigate a radar-assisted predictive beamforming design for vehicle-to-infrastructure (V2I) communication by exploiting the dual-functional radar-communication (DFRC) technique. Aiming for realizing joint sensing and communication functionalities at road side units (RSUs), we present a novel extended Kalman filtering (EKF) framework to track and predict kinematic parameters of each vehicle. By exploiting the radar functionality of the RSU we show that the communication beam tracking overheads can be drastically reduced. To improve the sensing accuracy while guaranteeing the downlink communication sum-rate, we further propose a power allocation scheme for multiple vehicles. Numerical results have shown that the proposed DFRC based beam tracking approach significantly outperforms the communication-only feedback based technique in the tracking performance. Furthermore, the designed power allocation method is able to achieve a favorable performance trade-off between sensing and communication.


Batch Normalization (BN) #REFR is the most common CNN normalization layer, which normalizes layer inputs using moments across the mini-batch dimension.

Normalization layers are essential in training Convolutional Neural Networks (CNNs), as they address the internal covariate shift of the network by normalizing the inputs to layers, so that the input distributions to each layer are stable during training #OTHEREFR .

However, to achieve deterministic behavior during inference, the mini-batch variance and mean are replaced by the global population mean and variance, which are obtained during training using an exponential running average. This works for i.i.d. #OTHEREFR data, but in non-i.i.d.

incremental learning, the BN estimates of the population mean and variance are heavily biased towards the most recent task, resulting in a significant loss of performance on old tasks #OTHEREFR .

Continual Normalization (CN) alleviates this discrepancy by combining Group- #OTHEREFR and Batch Normalization #OTHEREFR .

Principles Of Forgetting In Domain-Incremental Semantic Segmentation In Adverse Weather Conditions

Abstract: Deep neural networks for scene perception in automated vehicles achieve excellent results for the domains they were trained on. However, in real-world conditions, the domain of operation and its underlying data distribution are subject to change. Adverse weather conditions, in particular, can significantly decrease model performance when such data are not available during training. Additionally, when a model is incrementally adapted to a new domain, it suffers from catastrophic forgetting, causing a significant drop in performance on previously observed domains. Despite recent progress in reducing catastrophic forgetting, its causes and effects remain obscure. Therefore, we study how the representations of semantic segmentation models are affected during domain-incremental learning in adverse weather conditions. Our experiments and representational analyses indicate that catastrophic forgetting is primarily caused by changes to low-level features in domain-incremental learning and that learning more general features on the source domain using pre-training and image augmentations leads to efficient feature reuse in subsequent tasks, which drastically reduces catastrophic forgetting. These findings highlight the importance of methods that facilitate generalized features for effective continual learning algorithms.

Batch Normalization: Accelerating Deep Network Training By Reducing Internal Covariate Shift

Abstract: Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.


We use Adam optimizer #REFR and the learning rate is set to 0.002 initially.

To evaluate the performance of cooperative perception with communication interruption, we test the detection results with different V2X packet drop rates (PDRs) (0 ∼ 90%).

The detection results are evaluated by Average Precision (AP) at Intersection-over-Union (IoU) thresholds of 0.50 and 0.70.

To implement V2X-INCOP, we set the number of the past timestep k = 1, the width, length, and height of each voxel of the input to (0.4m, 0.4m, 4m) for OPV2V and Dair-V2X and (0.4m, 0.4m, 5m) for V2X-Sim.

In terms of coefficients in the loss function, we set α = 1, β = 2 and γ = 10000.

For a fair comparison, we implement all baselines and SOTA methods with Pointpillars, the same backbone as our proposed V2X-INCOP.

We train all the models using NVIDIA GeForce RTX 3090 GPU.

Interruption-Aware Cooperative Perception For V2X Communication-Aided Autonomous Driving

Abstract: Cooperative perception enabled by V2X Communication technologies can significantly improve the perception performance of autonomous vehicles beyond the limited perception ability of the individual vehicles, therefore, improving the safety and efficiency of autonomous driving in intelligent transportation systems. However, in order to fully reap the benefits of cooperative perception in practice, the impacts of imperfect V2X communication, i.e., communication errors and disruptions, need to be understood and effective remedies need to be developed to alleviate their adverse impacts. Motivated by this need, we propose a novel INterruption-aware robust COoperative Perception (V2X-INCOP) solution for V2X communication-aided autonomous driving, which leverages historical information to recover missing information due to interruption. To achieve comprehensive recovery, we design a communication adaptive multi-scale spatial-temporal prediction model to extract multiscale features based on V2X communication conditions and capture the most significant information for the prediction of the missing information. To further improve recovery performance, we adopt a knowledge distillation framework to give direct supervision to the prediction model and a curriculum learning strategy to stabilize the training of the model. Our experiments on three public cooperative perception datasets demonstrate that our proposed method is effective in alleviating the impacts of communication interruption on cooperative perception. Index Terms-cooperative perception, 3D object detection, autonomous driving I. INTRODUCTION Autonomous Vehicles (AVs) and Automated Driving Systems (ADSs) promise to improve the safety, efficiency, and accessibility of future transport systems. AVs and ADSs rely on accurate and robust understanding of their surroundings, i.e., perception, for safe, deficient, and comfortable motion planning [60]. Poor perception can result in traffic accidents, threatening human safety [55], [56], and other forms of

Adam: A Method For Stochastic Optimization

Abstract: We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.


Why Does S4 Perform Well On Lm But Not Mt?

This is in line with the observations made by #REFR , who argue that Transformer LMs tend to pay more attention to the previous few tokens, emphasizing the importance of short-term memory over long-term memory.

A natural question to ask is why does S4 perform well on LM #OTHEREFR , but not on MT.

Our intuition is that MT is a more challenging task.

For LM, the model only needs to consider a shorter context to accurately predict the next token, whereas Table 6 : BLEU scores on test set for each architecture in 4 different language pairs. The † on TR-S4A indicates statistically significant results.

for MT, it requires accurate access to the source sentence representations.

As the length of the source sentence increases, a fixed-size state is insufficient to capture fine-grained representations of the source, and thus the model's performance suffers.

State Spaces Aren'T Enough: Machine Translation Needs Attention

Abstract: Structured State Spaces for Sequences (S4) is a recently proposed sequence model with successful applications in various tasks, e.g. vision, language modeling, and audio. Thanks to its mathematical formulation, it compresses its input to a single hidden state, and is able to capture long range dependencies while avoiding the need for an attention mechanism. In this work, we apply S4 to Machine Translation (MT), and evaluate several encoder-decoder variants on WMT'14 and WMT'16. In contrast with the success in language modeling, we find that S4 lags behind the Transformer by approximately 4 BLEU points, and that it counter-intuitively struggles with long sentences. Finally, we show that this gap is caused by S4's inability to summarize the full source sentence in a single hidden state, and show that we can close the gap by introducing an attention mechanism.

Analyzing The Structure Of Attention In A Transformer Language Model

Abstract: The Transformer is a fully attention-based alternative to recurrent networks that has achieved state-of-the-art results across a range of NLP tasks. In this paper, we analyze the structure of attention in a Transformer language model, the GPT-2 small pretrained model. We visualize attention for individual instances and analyze the interaction between attention and syntax over a large corpus. We find that attention targets different parts of speech at different layer depths within the model, and that attention aligns with dependency relations most strongly in the middle layers. We also find that the deepest layers of the model capture the most distant relationships. Finally, we extract exemplar sentences that reveal highly specific patterns targeted by particular attention heads.


Following the strategy in References #REFR , we extracted the users' product purchasing behaviors based on their reviews, i.e., the products they reviewed are the ones they purchased.

Negative samples in the training dataset are set to those impressed products but not clicked by users, and posivtie samples are clicked.

Amazon 2 As there are no large-scale public datasets in the personalized product search area, we use Amazon dataset as our experiment corpus, consistent with existing approaches #OTHEREFR .

In our experiments, we adopted the five-core version provided, whereby the remaining users and products have at least five reviews, respectively.

Besides, we selected three categories with different sizes: Clothing, Shoes & Jewelry, Toys & Games, Electronics.

These datasets both contain several categories so that users may have different interests.

Our model uses the previously purchased products in a neighboring window size to model the short-term user interests.

We further filtered the dataset to make sure each user has at least 20 purchased products (i.e., 20 reviews).

Following the same paradigm used in #OTHEREFR , we construct queries for each purchased item with category information.

This strategy is based on the finding that directed product search is users' search for a producer's name, a brand, or a set of terms describing product category.

We partitioned each of the four datasets into three sets: training, validation and testing sets.

Hierarchically Fusing Long And Short-Term User Interests For Click-Through Rate Prediction In Product Search

Abstract: Estimating Click-Through Rate (CTR) is a vital yet challenging task in personalized product search. However, existing CTR methods still struggle in the product search settings due to the following three challenges including how to more effectively extract users' short-term interests with respect to multiple aspects, how to extract and fuse users' long-term interest with short-term interests, how to address the entangling characteristic of long and short-term interests. To resolve these challenges, in this paper, we propose a new approach named Hierarchical Interests Fusing Network (HIFN), which consists of four basic modules namely Short-term Interests Extractor (SIE), Long-term Interests Extractor (LIE), Interests Fusion Module (IFM) and Interests Disentanglement Module (IDM). Specifically, SIE is proposed to extract user's short-term interests by integrating three fundamental interests encoders within it namely query-dependent, target-dependent and causal-dependent interest encoder, respectively, followed by delivering the resultant representation to the module LIE, where it can effectively capture user longterm interests by devising an attention mechanism with respect to the short-term interests from SIE module. In IFM, the achieved long and short-term interests are further fused in an adaptive manner, followed by concatenating it with original raw context features for the final prediction result. Last but not least, considering the entangling characteristic of long and short-term interests, IDM further devises a self-supervised framework to disentangle long and short-term interests. Extensive offline and online evaluations on a real-world e-commerce platform demonstrate the superiority of HIFN over state-of-the-art methods.

Ups And Downs: Modeling The Visual Evolution Of Fashion Trends With One-Class Collaborative Filtering

Abstract: Building a successful recommender system depends on understanding both the dimensions of people's preferences as well as their dynamics. In certain domains, such as fashion, modeling such preferences can be incredibly difficult, due to the need to simultaneously model the visual appearance of products as well as their evolution over time. The subtle semantics and non-linear dynamics of fashion evolution raise unique challenges especially considering the sparsity and large scale of the underlying datasets. In this paper we build novel models for the One-Class Collaborative Filtering setting, where our goal is to estimate users' fashion-aware personalized ranking functions based on their past feedback. To uncover the complex and evolving visual factors that people consider when evaluating products, our method combines high-level visual features extracted from a deep convolutional neural network, users' past feedback, as well as evolving trends within the community. Experimentally we evaluate our method on two large real-world datasets from Amazon.com, where we show it to outperform state-of-the-art personalized ranking measures, and also use it to visualize the high-level fashion trends across the 11-year span of our dataset.


Appendix D Derivation Of Regularization Terms

CivilComments #REFR : CivilComments is a text classification task that aims to predict whether an online comment is toxic or non-toxic.

The remaining training samples include 3,498 samples from "landbirds on land", and 1,057 samples from "waterbirds on water".

CelebA #OTHEREFR : CelebA includes a number of face images of celebrities.

The classification labels are the hair colors of the celebrities including "blond" or "not blond." The spurious attribute is gender, i.e., male or female.

In CelebA, the minority groups are ("blond", male) and ("not blond", female).

The number of samples for each group are 71,629 (dark hair, female), 66,874 (dark hair, male), 22,880 (blond hair, female), and 1,387 (blond hair, male).

The spurious domain identifications are a number of demographic features, including male, female, LGBTQ, Christian, Muslim, other religion, Black, and White #OTHEREFR .

There are 450,000 comments in CivilComments which are collected from online articles.

The numbers of samples for training, validation, and test datasets are 269,038, 45,180, and 133,782, respectively. Fig.

14 illustrates more images augmented by ICDA, in which samples with various backgrounds and viewpoints are generated.

For example, ICDA generates dogs on the grass, in the water, and even on the sofa. In addition, we present more results for model visualization.

Implicit Counterfactual Data Augmentation For Deep Neural Networks

Abstract: Machine-learning models are prone to capturing the spurious correlations between non-causal attributes and classes, with counterfactual data augmentation being a promising direction for breaking these spurious associations. However, explicitly generating counterfactual data is challenging, with the training efficiency declining. Therefore, this study proposes an implicit counterfactual data augmentation (ICDA) method to remove spurious correlations and make stable predictions. Specifically, first, a novel sample-wise augmentation strategy is developed that generates semantically and counterfactually meaningful deep features with distinct augmentation strength for each sample. Second, we derive an easy-to-compute surrogate loss on the augmented feature set when the number of augmented samples becomes infinite. Third, two concrete schemes are proposed, including direct quantification and meta-learning, to derive the key parameters for the robust loss. In addition, ICDA is explained from a regularization aspect, with extensive experiments indicating that our method consistently improves the generalization performance of popular depth networks on multiple typical learning scenarios that require out-of-distribution generalization.

Nuanced Metrics For Measuring Unintended Bias With Real Data For Text Classification

Abstract: Unintended bias in Machine Learning can manifest as systemic differences in performance for different demographic groups, potentially compounding existing challenges to fairness in society at large. In this paper, we introduce a suite of threshold-agnostic metrics that provide a nuanced view of this unintended bias, by considering the various ways that a classifier's score distribution can vary across designated groups. We also introduce a large new test set of online comments with crowd-sourced annotations for identity references. We use this to show how our metrics can be used to find new and potentially subtle unintended bias in existing public models.


VQA models Since the introductory work on VQA #REFR , several endeavors have extended this paradigm, either by suggesting advanced model architectures or by proposing more challenging datasets.

State-of-the-art models addressing the VQA task are mainly based on VL transformer backbones; thus, models such as ViLBERT #OTHEREFR , VisualBERT #OTHEREFR , FLAVA #OTHEREFR , ALBEF #OTHEREFR , ViLT #OTHEREFR and others have dominated the recent VQA literature demonstrating rapid improvements on relevant benchmark datasets.

Regarding datasets, improvements on the original VQA (VQA-v2) suggest adding similar image pairs corresponding to the same question , but leading to diverging answers #OTHEREFR .

Visual Genome (VG) is another large scale dataset including numerous scene images, object, attribute and relationship annotations, as well as visual question-answer pairs #OTHEREFR .

Our approach is tested on both VQA-v2 and VG datasets.

Other popular VQA datasets are Flickr30k-Entities #OTHEREFR , COCO-QA #OTHEREFR , Visual7W #OTHEREFR and others.

Knowledge-Based Counterfactual Queries For Visual Question Answering

Abstract: Visual Question Answering (VQA) has been a popular task that combines vision and language, with numerous relevant implementations in literature. Even though there are some attempts that approach explainability and robustness issues in VQA models, very few of them employ counterfactuals as a means of probing such challenges in a model-agnostic way. In this work, we propose a systematic method for explaining the behavior and investigating the robustness of VQA models through counterfactual perturbations. For this reason, we exploit structured knowledge bases to perform deterministic, optimal and controllable word-level replacements targeting the linguistic modality, and we then evaluate the model's response against such counterfactual inputs. Finally, we qualitatively extract local and global explanations based on counterfactual responses, which are ultimately proven insightful towards interpreting VQA model behaviors. By performing a variety of perturbation types, targeting different parts of speech of the input question, we gain insights to the reasoning of the model, through the comparison of its responses in different adversarial circumstances. Overall, we reveal possible biases in the decision-making process of the model, as well as expected and unexpected patterns, which impact its performance quantitatively and qualitatively, as indicated by our analysis.

Vqa: Visual Question Answering

Abstract: We propose the task of free-form and open-ended Visual Question Answering (VQA). Given an image and a natural language question about the image, the task is to provide an accurate natural language answer. Mirroring real-world scenarios, such as helping the visually impaired, both the questions and answers are open-ended. Visual questions selectively target different areas of an image, including background details and underlying context. As a result, a system that succeeds at VQA typically needs a more detailed understanding of the image and complex reasoning than a system producing generic image captions. Moreover, VQA is amenable to automatic evaluation, since many open-ended answers contain only a few words or a closed set of answers that can be provided in a multiple-choice format. We provide a dataset containing $$\sim $$~0.25 M images, $$\sim $$~0.76 M questions, and $$\sim $$~10 M answers (www.visualqa.org), and discuss the information it provides. Numerous baselines and methods for VQA are provided and compared with human performance. Our VQA demo is available on CloudCV (http://cloudcv.org/vqa).


In this section, we analyze the latent space of Style-GAN #REFR from the perspective of developing the face swapping model.

Then, we justify the proper combination of latents for a source attribute leakage-free model.

h×w results in improved preservation of expression, head pose, and eye gazing, while the identity undergoes less change.

Overall score is calculated by (ID sim) 3 * (HP dis) * (Exp dis) * (EG dis) which is standardized.

Details for each score metric are described in the supplementary materials.

Robustswap: A Simple Yet Robust Face Swapping Model Against Attribute Leakage

Abstract: Figure 1. Face swapping results of RobustSwap. Our model robustly synthesizes the swapped image from the in-the-wild source and target images, while preserving the target attributes and reflecting source identity. With our model, the target attributes (e.g., eye gazing, hair, background, expression, and head pose) are preserved successfully regardless of source identity images.

Analyzing And Improving The Image Quality Of Stylegan

Abstract: The style-based GAN architecture (StyleGAN) yields state-of-the-art results in data-driven unconditional generative image modeling. We expose and analyze several of its characteristic artifacts, and propose changes in both model architecture and training methods to address them. In particular, we redesign the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images. In addition to improving image quality, this path length regularizer yields the additional benefit that the generator becomes significantly easier to invert. This makes it possible to reliably attribute a generated image to a particular network. We furthermore visualize how well the generator utilizes its output resolution, and identify a capacity problem, motivating us to train larger models for additional quality improvements. Overall, our improved model redefines the state of the art in unconditional image modeling, both in terms of existing distribution quality metrics as well as perceived image quality.


This testing set is not available in real experiments making it hard to trust their behavior online #REFR .

We kickstarted the image classification models using transfer learning or self-supervision, following the industry best practices. Reproducibility.

Our framework is open source, and all experiments results are made available and can be easily run again.

We provide the accuracies and other AL metrics for the most common AL samplers, along with all train, test, and initial batch indices used for those experiments.

Online evaluation of sampling strategies.

Research works rely on the area under the accuracy curve of a left-out test set to evaluate the performance of AL strategies.

OpenAL logs unsupervised metrics to improve the offline strategies' interpretability and be able to interpret their behavior online #OTHEREFR .

We first start by describing the setup of our tasks, the model selection methodology, and the evaluation criteria for sampling strategies.

Then we present the results of our experiments per strategy across all tasks.

We finish by focusing on the metrics observed and how they explain the performance of some strategies.

Finally we open new perspectives on AL experiments and how this benchmark could be useful and extended in the future.

Openal: Evaluation And Interpretation Of Active Learning Strategies

Abstract: Despite the vast body of literature on Active Learning (AL), there is no comprehensive and open benchmark allowing for efficient and simple comparison of proposed samplers. Additionally, the variability in experimental settings across the literature makes it difficult to choose a sampling strategy, which is critical due to the one-off nature of AL experiments. To address those limitations, we introduce OpenAL, a flexible and open-source framework to easily run and compare sampling AL strategies on a collection of realistic tasks. The proposed benchmark is augmented with interpretability metrics and statistical analysis methods to understand when and why some samplers outperform others. Last but not least, practitioners can easily extend the benchmark by submitting their own AL samplers. Preprint. Under review.

Limitations Of Assessing Active Learning Performance At Runtime

Abstract: Classification algorithms aim to predict an unknown label (e.g., a quality class) for a new instance (e.g., a product). Therefore, training samples (instances and labels) are used to deduct classification hypotheses. Often, it is relatively easy to capture instances but the acquisition of the corresponding labels remain difficult or expensive. Active learning algorithms select the most beneficial instances to be labeled to reduce cost. In research, this labeling procedure is simulated and therefore a ground truth is available. But during deployment, active learning is a one-shot problem and an evaluation set is not available. Hence, it is not possible to reliably estimate the performance of the classification system during learning and it is difficult to decide when the system fulfills the quality requirements (stopping criteria). In this article, we formalize the task and review existing strategies to assess the performance of an actively trained classifier during training. Furthermore, we identified three major challenges: 1)~to derive a performance distribution, 2)~to preserve representativeness of the labeled subset, and 3) to correct against sampling bias induced by an intelligent selection strategy. In a qualitative analysis, we evaluate different existing approaches and show that none of them reliably estimates active learning performance stating a major challenge for future research for such systems. All plots and experiments are provided in a Jupyter notebook that is available for download.


To filter massive less informative supervision in background areas, Wang #REFR transferred effective knowledge in sparse imitation regions.

As the seminal work by Hinton #OTHEREFR , knowledge distillation is introduced to boost performance of tiny models.

A line of works #OTHEREFR has verified its potential.

To extend its application to object detection, MIMIC #OTHEREFR proposed to prevent divergence between RPN output of teacher ans student via 2 loss.

To handle imbalanced supervision from foreground and background pixels, Zhang #OTHEREFR and Yang #OTHEREFR encouraged student to learn from crucial pixels of foreground objects and the relationship between them.

To separate informative feature in and out of bounding boxed, FRS #OTHEREFR presented feature-richness score to focus on important features.

To further investigate sophisticated dependence, MasKD #OTHEREFR proposed receptive tokens to build pixel dependency masks for more effective distilling.

Instead of spatial-wise, CWD #OTHEREFR exploited more semantical information in each channel with category-specific masks.

For the sake of fusing different kinds of information, GID #OTHEREFR merged branches of feature/relation/response-based knowledge and developed an all-in-one framework.

Avatar Knowledge Distillation: Self-Ensemble Teacher Paradigm With Uncertainty

Abstract: Knowledge distillation is an effective paradigm for boosting the performance of pocket-size model, especially when multiple teacher models are available, the student would break the upper limit again. However, it is not economical to train diverse teacher models for the disposable distillation. In this paper, we introduce a new concept dubbed Avatars for distillation, which are the inference ensemble models derived from the teacher. Concretely, (1) For each iteration of distillation training, various Avatars are generated by a perturbation transformation. We validate that Avatars own higher upper limit of working capacity and teaching ability, aiding the student model in learning diverse and receptive knowledge perspectives from the teacher model. (2) During the distillation, we propose an uncertainty-aware factor from the variance of statistical differences between the vanilla teacher and Avatars, to adjust Avatars' contribution on knowledge transfer adaptively. Avatar Knowledge Distillation (AKD) is fundamentally different from existing methods and refines with the innovative view of unequal training. Comprehensive experiments demonstrate the effectiveness of our Avatars mechanism, which polishes up the state-of-the-art distillation methods for dense prediction without more extra computational cost. The AKD brings at most 0.7 AP gains on COCO 2017 for Object Detection and 1.83 mIoU gains on Cityscapes for Semantic Segmentation, respectively.

Distilling Object Detectors With Fine-Grained Feature Imitation

Abstract: State-of-the-art CNN based recognition models are often computationally prohibitive to deploy on low-end devices. A promising high level approach tackling this limitation is knowledge distillation, which let small student model mimic cumbersome teacher model's output to get improved generalization. However, related methods mainly focus on simple task of classification while do not consider complex tasks like object detection. We show applying the vanilla knowledge distillation to detection model gets minor gain. To address the challenge of distilling knowledge in detection model, we propose a fine-grained feature imitation method exploiting the cross-location discrepancy of feature response. Our intuition is that detectors care more about local near object regions. Thus the discrepancy of feature response on the near object anchor locations reveals important information of how teacher model tends to generalize. We design a novel mechanism to estimate those locations and let student model imitate the teacher on them to get enhanced performance. We first validate the idea on a developed lightweight toy detector which carries simplest notion of current state-of-the-art anchor based detection models on challenging KITTI dataset, our method generates up to 15% boost of mAP for the student model compared to the non-imitated counterpart. We then extensively evaluate the method with Faster R-CNN model under various scenarios with common object detection benchmark of Pascal VOC and COCO, imitation alleviates up to 74% performance drop of student model compared to teacher. Codes released at https://github.com/twangnh/Distilling-Object-Detectors


In the meta-learner training, we use Adam #REFR to update θ meta , and the meta-optimization step size β is fixed to 0.001, the meta-learning rate γ to 0.004, the metalearner training epochs ite 2 to 150.

Based on QST bases, training images are separated into D base and D meta . We adopt ResNet-18 #OTHEREFR as the baseline model.

All settings and configurations in the base training of QAM are the same as those in the baseline model training.

Specifically, SGD with Nesterov momentum is utilized as optimizer to update θ bnb and θ rem , and the learning rate α is set to 0.1, weight decay to 0.0005, dampening to 0, momentum to 0.9.

The learning rate dropped by 0.2 at 60, 120, and 160 epochs, the base training epochs ite 1 are set to 200, and the minibatch size is set to 128.

All input images are padded by 4 pixels on each side with reflections of the original image and pre-processed with horizontal flips and random crops.

2) Results on CIFAR-10/100-J and CIFAR-10/100-J-S: We compare QAM to the baseline model, which trains ResNet-18 on all training images of CIFAR-10/100-J or CIFAR-10/100-J-S and updates the parameters depending on Equation (1). From the results shown in Fig. 8 and Fig. 9 , we obtain three observations. Firstly, QAM achieves significant improvements, demonstrating its effectiveness.

Secondly, the improvements on images with slighter compression loss are larger.

The reason is that the baseline model is badly misled by heavily compressed images, while QAM utilizes QAC to reduce the interference of quantization.

Thirdly, QAM does not undermine the performance over images with heavy compression but achieves slight improvements.

Moreover, we compare QAM with several state-of-the-art methods, including two ensemble learning based methods (CDI #OTHEREFR and ADL #OTHEREFR ) and two features alignment-based methods (FCT #OTHEREFR and ARF #OTHEREFR ).

Picking Up Quantization Steps For Compressed Image Classification

Abstract: The sensitivity of deep neural networks to compressed images hinders their usage in many real applications, which means classification networks may fail just after taking a screenshot and saving it as a compressed file. In this paper, we argue that neglected disposable coding parameters stored in compressed files could be picked up to reduce the sensitivity of deep neural networks to compressed images. Specifically, we resort to using one of the representative parameters, quantization steps, to facilitate image classification. Firstly, based on quantization steps, we propose a novel quantization aware confidence (QAC), which is utilized as sample weights to reduce the influence of quantization on network training. Secondly, we utilize quantization steps to alleviate the variance of feature distributions, where a quantization aware batch normalization (QABN) is proposed to replace batch normalization of classification networks. Extensive experiments show that the proposed method significantly improves the performance of classification networks on CIFAR-10, CIFAR-100, and ImageNet.


As with the original paper #REFR , we set λ 1 , λ 2 , λ 3 , λ 4 , λ 5 , λ 6 as 10.0, 0.1, 10.0, 0.1, 5.0, and 1.0, respectively.

Thus, the second term represents the loss function when F (I)/L and the luminance of the estimated shade F (S(I)) are correlated.

where m L is the mask, which is 1 for pixels with densified LiDAR intensity and 0 otherwise.

In addition, s x and b x , where x ∈ 1, 2, are trainable parameters for adjusting the scale and bias of LiDAR intensity, respectively. In summary, IID-LI optimizes the loss function in Eq. #OTHEREFR .

In addition, we set λ 7 as 20.0 in this paper.

Unsupervised Intrinsic Image Decomposition With Lidar Intensity

Abstract: Intrinsic image decomposition (IID) is the task that decomposes a natural image into albedo and shade. While IID is typically solved through supervised learning methods, it is not ideal due to the difficulty in observing ground truth albedo and shade in general scenes. Conversely, unsupervised learning methods are currently underperforming supervised learning methods since there are no criteria for solving the ill-posed problems. Recently, light detection and ranging (LiDAR) is widely used due to its ability to make highly precise distance measurements. Thus, we have focused on the utilization of LiDAR, especially LiDAR intensity, to address this issue. In this paper, we propose unsupervised intrinsic image decomposition with LiDAR intensity (IID-LI). Since the conventional unsupervised learning methods consist of image-to-image transformations, simply inputting LiDAR intensity is not an effective approach. Therefore, we design an intensity consistency loss that computes the error between LiDAR intensity and gray-scaled albedo to provide a criterion for the ill-posed problem. In addition, LiDAR intensity is difficult to handle due to its sparsity and occlusion, hence, a LiDAR intensity densification module is proposed. We verified the estimating quality using our own dataset, which include RGB images, LiDAR intensity and human judged annotations. As a result, we achieved an estimation accuracy that outperforms conventional unsupervised learning methods.

Unsupervised Learning For Intrinsic Image Decomposition From A Single Image.

Abstract: Intrinsic image decomposition, which is an essential task in computer vision, aims to infer the reflectance and shading of the scene. It is challenging since it needs to separate one image into two components. To tackle this, conventional methods introduce various priors to constrain the solution, yet with limited performance. Meanwhile, the problem is typically solved by supervised learning methods, which is actually not an ideal solution since obtaining ground truth reflectance and shading for massive general natural scenes is challenging and even impossible. In this paper, we propose a novel unsupervised intrinsic image decomposition framework, which relies on neither labeled training data nor hand-crafted priors. Instead, it directly learns the latent feature of reflectance and shading from unsupervised and uncorrelated data. To enable this, we explore the independence between reflectance and shading, the domain invariant content constraint and the physical constraint. Extensive experiments on both synthetic and real image datasets demonstrate consistently superior performance of the proposed method.


Decoupling Proposal Generation And Roi Classification.

With the generated proposals, the RoI classification stage is trained in the Fast RCNN #REFR manner.

In contrast, the class-specific RoI classification stage cannot even work for novel categories.

Even with language embeddings, it is still biased to base classes.

The distinct properties affect their joint training since the sensitivity of the classification stage to novel classes hampers the universality ability of the proposal generation stage.

Consequently, we decouple these two stages and train them separately to avoid such conflict.

Specifically, the region proposal generation stage is initialized with traditional ImageNet pre-trained parameters and trained in a class-agnostic way. After training, it produces a series of region proposals.

This stage is initialized with imagetext pre-trained parameters for predicting unseen categories.

These two kinds of pre-trained parameters also contain complementary features, which provide more comprehensive information for universal object detection.

To produce generalized proposals in the open world, we present the classagnostic localization network (CLN), as illustrated in Fig.  4 .

Instead of a single RPN module, our CLN contains both the RPN and the RoI head to generate proposals for universal object detection.

Detecting Everything In The Open World: Towards Universal Object Detection

Abstract: In this paper, we formally address universal object detection, which aims to detect every scene and predict every category. The dependence on human annotations, the limited visual information, and the novel categories in the open world severely restrict the universality of traditional detectors. We propose UniDetector, a universal object detector that has the ability to recognize enormous categories in the open world. The critical points for the universality of UniDetector are: 1) it leverages images of multiple sources and heterogeneous label spaces for training through the alignment of image and text spaces, which guarantees sufficient information for universal representations. 2) it generalizes to the open world easily while keeping the balance between seen and unseen classes, thanks to abundant information from both vision and language modalities. 3) it further promotes the generalization ability to novel categories through our proposed decoupling training manner and probability calibration. These contributions allow UniDetector to detect over 7k categories, the largest measurable category size so far, with only about 500 classes participating in training. Our UniDetector behaves the strong zero-shot generalization ability on largevocabulary datasets-it surpasses the traditional supervised baselines by more than 4% on average without seeing any corresponding images. On 13 public detection datasets with various scenes, UniDetector also achieves state-of-the-art performance with only a 3% amount of training data.

Fast R-Cnn

Abstract: This paper proposes a Fast Region-based Convolutional Network method (Fast R-CNN) for object detection. Fast R-CNN builds on previous work to efficiently classify object proposals using deep convolutional networks. Compared to previous work, Fast R-CNN employs several innovations to improve training and testing speed while also increasing detection accuracy. Fast R-CNN trains the very deep VGG16 network 9x faster than R-CNN, is 213x faster at test-time, and achieves a higher mAP on PASCAL VOC 2012. Compared to SPPnet, Fast R-CNN trains VGG16 3x faster, tests 10x faster, and is more accurate. Fast R-CNN is implemented in Python and C++ (using Caffe) and is available under the open-source MIT License at https://github.com/rbgirshick/fast-rcnn.


The new challenge had focused on adversarial attack and defense against machine learning model there had been series of research survey which establish the need for protecting machine learning model against various forms of attach which make it to misclassify #REFR .

Adversarial machine learning deals with the study of attacks on machine learning algorithms, and of the defenses against such attacks.

#OTHEREFR .Many years ago, the focus of machine learning engineers and scientist was on obtaining high www.ijacsa.thesai.org accuracy for correct prediction, and while this had greatly been resolve in the past few years.

During the training of machine learning model, it is usually assumed that the training and test data are generated from the same statistical distribution.

This assumption makes the final model vulnerable to various forms of attack, majority of which includes evasion attacks, [3] data poisoning attacks, [4] Byzantine attacks #OTHEREFR and model extraction [6] .

Adversarial Sampling For Fairness Testing In Deep Neural Network

Abstract: In this research, we focus on the usage of adversarial sampling to test for the fairness in the prediction of deep neural network model across different classes of image in a given dataset. While several framework had been proposed to ensure robustness of machine learning model against adversarial attack, some of which includes adversarial training algorithm. There is still the pitfall that adversarial training algorithm tends to cause disparity in accuracy and robustness among different group. Our research is aimed at using adversarial sampling to test for fairness in the prediction of deep neural network model across different classes or categories of image in a given dataset. We successfully demonstrated a new method of ensuring fairness across various group of input in deep neural network classifier. We trained our neural network model on the original image, and without training our model on the perturbed or attacked image. When we feed the adversarial samplings to our model, it was able to predict the original category/ class of the image the adversarial sample belongs to. We also introduced and used the separation of concern concept from software engineering whereby there is an additional standalone filter layer that filters perturbed image by heavily removing the noise or attack before automatically passing it to the network for classification, we were able to have accuracy of 93.3%. Cifar-10 dataset have ten categories of dataset, and so, in order to account for fairness, we applied our hypothesis across each categories of dataset and were able to get a consistent result and accuracy.

Adversarial Machine Learning -- Industry Perspectives

Abstract: Based on interviews with 28 organizations, we found that industry practitioners are not equipped with tactical and strategic tools to protect, detect and respond to attacks on their Machine Learning (ML) systems. We leverage the insights from the interviews and we enumerate the gaps in perspective in securing machine learning systems when viewed in the context of traditional software security development. We write this paper from the perspective of two personas: developers/ML engineers and security incident responders who are tasked with securing ML systems as they are designed, developed and deployed ML systems. The goal of this paper is to engage researchers to revise and amend the Security Development Lifecycle for industrial-grade software in the adversarial ML era.


In the feedback law (19), we observed no noticeable difference between having ε = 0 or ε being a small number, which agrees with the observations made in #REFR .

In Figure 5 , the mean trajectory of 100 realizations as well as sample trajectories of 10 realizations are shown with the resulting control Hamiltonians. All realizations are initialized with

i| , i.e., a mixed state with probability 1/2 of being in |0 0| and probability 1/2 in uniform superposition over all basis states.

Thus, in the simulations shown here we have ε = 0.

(2013) , we use a quadratic approximation of the nonlinear optimization problem in the feedback law (19) to speed up computations. In particular, we solve the quadratic program

u k = arg min u∈[−ū,ū] 1 2 Tr [H 1 , P ], H 1 ρ k − ε 4 i i|[H 1 , ρ k ]|i 2 u 2 − i Tr [H 1 , P ]ρ k u (29) withū = 0.1.

As can be seen from Figure 5 , the state converges to the state ρ = |n n| in almost all cases within the first 1000 iterations.

Measurement-Based Control For Minimizing Energy Functions In Quantum Systems

Abstract: In variational quantum algorithms (VQAs), the most common objective is to find the minimum energy eigenstate of a given energy Hamiltonian. In this paper, we consider the general problem of finding a sufficient control Hamiltonian structure that, under a given feedback control law, ensures convergence to the minimum energy eigenstate of a given energy function. By including quantum non-demolition (QND) measurements in the loop, convergence to a pure state can be ensured from an arbitrary mixed initial state. Based on existing results on strict control Lyapunov functions, we formulate a semidefinite optimization problem, whose solution defines a non-unique control Hamiltonian, which is sufficient to ensure almost sure convergence to the minimum energy eigenstate under the given feedback law and the action of QND measurements. A numerical example is provided to showcase the proposed methodology.

Feedback Stabilization Of Discrete-Time Quantum Systems Subject To Non-Demolition Measurements With Imperfections And Delays

Abstract: We consider a controlled quantum system whose finite dimensional state is governed by a discrete-time nonlinear Markov process. In open-loop, the measurements are assumed to be quantum non-demolition (QND). The eigenstates of the measured observable are thus the open-loop stationary states: they are used to construct a closed-loop supermartingale playing the role of a strict control Lyapunov function. The parameters of this supermartingale are calculated by inverting a Metzler matrix that characterizes the impact of the control input on the Kraus operators defining the Markov process. The resulting state feedback scheme, taking into account a known constant delay, provides the almost sure convergence of the controlled system to the target state. This convergence is ensured even in the case where the filter equation results from imperfect measurements corrupted by random errors with conditional probabilities given as a left stochastic matrix. Closed-loop simulations corroborated by experimental data illustrate the interest of such nonlinear feedback scheme for the photon box, a cavity quantum electrodynamics system.


B. Transformer Based Method For In Multimodal Emotion Recognition From Physiological Signal

Since their introduction in 2016, Transformers #REFR have emerged as the preferred models in the field of Deep Learning.

However, these approaches are heavily dependent on the quality of handcrafted feature selection and feature extraction processes.

To overcome this challenge, other studies #OTHEREFR - #OTHEREFR proposed the use of Deep Learning techniques for an end-to-end approach, where the model learns to extract features automatically without the need for pre-designed feature descriptors.

With the advancement of Deep Learning, various state-ofthe-art techniques have been employed for Affective Analysis from Physiological signals. Santamaria et al.

#OTHEREFR used Convolutional Neural Networks (CNNs) with 1D Convolution layers for Emotion Detection, while Harper et al.

#OTHEREFR combined CNNs with frequently used Recurrent Neural Networks (RNNs) for Emotion Recognition from ECG signals.

Their robust performance in Natural Language Processing, a type of data that shares some similar characteristics with timeseries data, has demonstrated the potential of Transformers when applied to time-series signals.

As a result, recent research in the Time-Series domain has utilized Transformers as the core module in their model architecture #OTHEREFR - #OTHEREFR .

For physiological signals, some studies have proposed using Transformers and their variants for detecting emotions #OTHEREFR , #OTHEREFR - #OTHEREFR . In the works of Vazquez et al.

#OTHEREFR , #OTHEREFR , they focused on applying pretrained Transformers for Multimodal signal processing.

However, this is still a very basic application of Transformer modules. Wu et al. #OTHEREFR and Yang et al.

Multi-Scale Transformer-Based Network For Emotion Recognition From Multi Physiological Signals

Abstract: This paper presents an efficient Multi-scale Transformer-based approach for the task of Emotion recognition from Physiological data, which has gained widespread attention in the research community due to the vast amount of information that can be extracted from these signals using modern sensors and machine learning techniques. Our approach involves applying a Multi-modal technique combined with scaling data to establish the relationship between internal body signals and human emotions. Additionally, we utilize Transformer and Gaussian Transformation techniques to improve signal encoding effectiveness and overall performance. Our model achieves decent results on the CASE dataset of the EPiC competition, with an RMSE score of 1.45.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


As discussed in the paper, we train a UNet on the Vimeo90K #REFR dataset to generate weight maps to blend the warped frames. Each scene in Vimeo90K consists of 7 frames.

In this section, we provide further details about the blending network training.

We use the corner frames (1 and 7) as reference frames and randomly select an intermediate frame from 2-6.

We apply data augmentation techniques like random horizontal and vertical flip, and cropping to the sequence.

We use Zhang et al.'s optical flow method #OTHEREFR to estimate the flow between intermediate and reference frames.

The reference frames are warped using these flows and the weight maps from the UNet are used to smoothly blend these warped frames.

We use the following loss to train our blending network:

Implicit View-Time Interpolation Of Stereo Videos Using Multi-Plane Disparities And Non-Uniform Coordinates

Abstract: In this paper, we propose an approach for view-time interpolation of stereo videos. Specifically, we build upon X-Fields that approximates an interpolatable mapping between the input coordinates and 2D RGB images using a convolutional decoder. Our main contribution is to analyze and identify the sources of the problems with using X-Fields in our application and propose novel techniques to overcome these challenges. Specifically, we observe that X-Fields struggles to implicitly interpolate the disparities for large baseline cameras. Therefore, we propose multi-plane disparities to reduce the spatial distance of the objects in the stereo views. Moreover, we propose non-uniform time coordinates to handle the non-linear and sudden motion spikes in videos. We additionally introduce several simple, but important, improvements over X-Fields. We demonstrate that our approach is able to produce better results than the state of the art, while running in near real-time rates and having low memory and storage costs.

Video Enhancement With Task-Oriented Flow

Abstract: Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution.


Furthermore, recurrent neural networks have been employed in the research #REFR to learn the temporal information of objects to improve the matching accuracy.

Most of online multi-object tracking methods are tracking by detection #OTHEREFR , #OTHEREFR , which rely on object detection algorithms for object recognition and objection location. Then additional ＊ Corresponding author:

Email: deemozen@163.com convolutional neural network branches are used to obtain the features of detected objects.

Therefore, the similarity between detected objects can be obtained based on their features.

Besides, the Kalman filter algorithm #OTHEREFR is usually used to estimate the motion information of objects.

Finally, Hungarian matching algorithm #OTHEREFR is used to match the objects between frames in terms of the obtained appearance similarity and motion information.

More recently, some works #OTHEREFR - #OTHEREFR use sophisticated object detection architectures to obtain the outstanding tracking accuracy.

In this paper, we focus on exploring effective and efficient matching algorithms without using appearance features of objects.

Even with simple object detection models, our proposed algorithm can ensure fast training speed and online tracking.

The matching process usually necessitates the use of numerous pieces of information about the objects, such as object motion #OTHEREFR and appearance features #OTHEREFR .

However, the extraction of appearance features is time-consuming, and it is difficult to achieve real-time object tracking.

Real-Time Multi-Object Tracking Based On Bi-Directional Matching

Abstract: ：In recent years, anchor-free object detection models combined with matching algorithms are used to achieve real-time muti-object tracking and also ensure high tracking accuracy. However, there are still great challenges in multi-object tracking. For example, when most part of a target is occluded or the target just disappears from images temporarily, it often leads to tracking interruptions for most of the existing tracking algorithms. Therefore, this study offers a bi-directional matching algorithm for multiobject tracking that makes advantage of bi-directional motion prediction information to improve occlusion handling. A stranded area is used in the matching algorithm to temporarily store the objects that fail to be tracked. When objects recover from occlusions, our method will first try to match them with objects in the stranded area to avoid erroneously generating new identities, thus forming a more continuous trajectory. Experiments show that our approach can improve the multi-object tracking performance in the presence of occlusions. In addition, this study provides an attentional up-sampling module that not only assures tracking accuracy but also accelerates training speed. In the MOT17 challenge, the proposed algorithm achieves 63.4% MOTA, 55.3% IDF1, and 20.1 FPS tracking speed.

Tracking The Untrackable: Learning To Track Multiple Cues With Long-Term Dependencies

Abstract: The majority of existing solutions to the Multi-Target Tracking (MTT) problem do not combine cues over a long period of time in a coherent fashion. In this paper, we present an online method that encodes long-term temporal dependencies across multiple cues. One key challenge of tracking methods is to accurately track occluded targets or those which share similar appearance properties with surrounding objects. To address this challenge, we present a structure of Recurrent Neural Networks (RNN) that jointly reasons on multiple cues over a temporal window. Our method allows to correct data association errors and recover observations from occluded states. We demonstrate the robustness of our data-driven approach by tracking multiple targets using their appearance, motion, and even interactions. Our method outperforms previous works on multiple publicly available datasets including the challenging MOT benchmark.

matching accuracy
recurrent neural networks


An initial promising result along these lines is #REFR , but there is substantial need for new research for broad classes of models and corresponding variational families.

Uncertainty quantification of the VB posterior It is well-known that variational posteriors tend to underestimate uncertainty of the posterior, so a central open question is how one can construct computationally efficient VB posteriors producing (a) credible balls with valid frequentist coverage and/or (b) posterior covariance matching that of the true posterior.

There is limited work on theory for statistical inference using the variational posterior, including credible intervals and hypothesis testing.

For this, we need theorems to reveal a limiting distribution of the variational posterior as the sample size goes to infinity, just as the Bernsteinvon Mises (BvM) theorem guarantees that the original posterior distribution converges to a Gaussian distribution under certain regularity conditions.

Theoretical guarantees of gradient-based algorithms Existing theoretical guarantees for VB only apply to the global solution of the variational optimization problem.

In practice, this optimization problem tends to be highly nonconvex and algorithms are only guaranteed to converge to local optima.

For certain variational families and model classes, these local optima can be dramatically different, so that there is a large sensitivity to the starting point of the algorithm.

It is of critical importance to obtain guarantees on the algorithms being used and not just on inaccessible global optima.

For example, can one obtain general theoretical guarantees for gradient-based black-box variational inference with or without warm-start conditions?

Machine Learning And The Future Of Bayesian Computation

Abstract: Bayesian models are a powerful tool for studying complex data, allowing the analyst to encode rich hierarchical dependencies and leverage prior information. Most importantly, they facilitate a complete characterization of uncertainty through the posterior distribution. Practical posterior computation is commonly performed via MCMC, which can be computationally infeasible for high dimensional models with many observations. In this article we discuss the potential to improve posterior computation using ideas from machine learning. Concrete future directions are explored in vignettes on normalizing flows, Bayesian coresets, distributed Bayesian inference, and variational inference.

Frequentist Consistency Of Variational Bayes

Abstract: ABSTRACTA key challenge for modern Bayesian statistics is how to perform scalable inference of posterior distributions. To address this challenge, variational Bayes (VB) methods have emerged as a p...


Our results are applicable to convex, nonconvex and strongly-convex losses, as they appear in the second column of Table 1 , and hold essentially verbatim to those in prior work #REFR , initially developed for SGD with batch size 1.

An important implication of our results is that different mini-batch schedules mainly affect the optimization error (as appears in #OTHEREFR ) rather than the generalization error for the class of Lipschitz and smooth losses.

We also note that in the smooth (possibly non-Lipschitz, nonconvex) case, the question of optimality of batch schedules other than that of full-batch GD (which we prove to be optimal) remains open.

More specifically, we establish upper bounds on the generalization error which are uniform over the class of mini-batch GD training schemes with either deterministic or stochastic, dataindependent, but otherwise arbitrary batch schedules; this class includes both deterministic and stochastic training as special cases.

For instance, the set of algorithms under consideration includes but is not limited to classical SGD with mini-batch of any size m, full-batch GD, round-robin deterministic batch selection, SGD with random reshuffling, SGD with single shuffling, and incremental gradient methods #OTHEREFR .

For this general set of algorithms, we prove generalization error bounds under a unified framework with respect to any data-independent (randomized or deterministic) batch scheduling rule.

To evaluate the performance of the algorithms under consideration, we are also interested in deriving minimax -over mini-batch GD algorithms (inf) and learning problem instances (sup)lower bounds directly on the generalization error.

Prior works have already shown lower bounds on uniform stability (while taking a supremum over the data-set) for non-smooth losses #OTHEREFR and Lipschitz-smooth losses #OTHEREFR , for SGD and full-batch GD.

However, lower bounds on uniform stability do not imply a lower bound on the generalization error; uniform stability is sufficient but not necessary for generalization.

Further, recent work on the class of smooth convex losses #OTHEREFR considered the construction of specific -but not fully parameterized-examples to derive lower bounds on the corresponding excess risk (i.e., generalization + optimization), and showed tightness also for the cases of SGD #OTHEREFR and full-batch GD #OTHEREFR .

Additionally, the construction of an elaborate example to lower bound the excess risk of the (S)GD algorithm appeared in #OTHEREFR .

Select Without Fear: Almost All Mini-Batch Schedules Generalize Optimally

Abstract: We establish matching upper and lower generalization error bounds for mini-batch Gradient Descent (GD) training with either deterministic or stochastic, data-independent, but otherwise arbitrary batch selection rules. We consider smooth Lipschitz-convex/nonconvex/stronglyconvex loss functions, and show that classical upper bounds for Stochastic GD (SGD) also hold verbatim for such arbitrary nonadaptive batch schedules, including all deterministic ones. Further, for convex and strongly-convex losses we prove matching lower bounds directly on the generalization error uniform over the aforementioned class of batch schedules, showing that all such batch schedules generalize optimally. Lastly, for smooth (non-Lipschitz) nonconvex losses, we show that full-batch (deterministic) GD is essentially optimal, among all possible batch schedules within the considered class, including all stochastic ones.

Train Faster, Generalize Better: Stability Of Stochastic Gradient Descent

Abstract: We show that parametric models trained by a stochastic gradient method (SGM) with few iterations have vanishing generalization error. We prove our results by arguing that SGM is algorithmically stable in the sense of Bousquet and Elisseeff. Our analysis only employs elementary tools from convex and continuous optimization. We derive stability bounds for both convex and non-convex optimization under standard Lipschitz and smoothness assumptions. ::: Applying our results to the convex case, we provide new insights for why multiple epochs of stochastic gradient methods generalize well in practice. In the non-convex case, we give a new interpretation of common practices in neural networks, and formally show that popular techniques for training large deep models are indeed stability-promoting. Our findings conceptually underscore the importance of reducing training time beyond its obvious benefit.


FK loss is proposed in #REFR to alleviate the accumulative errors of rotations in local coordinates.

As common in CVAE, we regularize the posterior distribution to normal distribution by optimizing the Kullback-Leibler divergence:

where q(• | •) denotes the inference posterior (encoder).

We calculate global positions by local quaternions with forward kinematics (FK) and get the average L1 norm between the calculated positions and real positions. It The FK loss function is:

where r represents the global root position. Adversarial Loss.

We use a generator-discriminator architecture and employ a pair of long-short discriminators #OTHEREFR to improve the motion quality.

The discriminator is in the form of Least Square GAN #OTHEREFR .

Diverse Motion In-Betweening With Dual Posture Stitching

Abstract: Figure 1. Three motion transitions generated by our method. The poses are rendered every ten frames. Black: key frames. Gray: generated transitions.

Quaternet: A Quaternion-Based Recurrent Model For Human Motion

Abstract: Deep learning for predicting or generating 3D human pose sequences is an active research area. Previous work regresses either joint rotations or joint positions. The former strategy is prone to error accumulation along the kinematic chain, as well as discontinuities when using Euler angle or exponential map parameterizations. The latter requires re-projection onto skeleton constraints to avoid bone stretching and invalid configurations. This work addresses both limitations. Our recurrent network, QuaterNet, represents rotations with quaternions and our loss function performs forward kinematics on a skeleton to penalize absolute position errors instead of angle errors. On short-term predictions, QuaterNet improves the state-of-the-art quantitatively. For long-term generation, our approach is qualitatively judged as realistic as recent neural strategies from the graphics literature.


There have been multiple modifications on attention mechanisms including Soft attention #REFR which focuses on specific parts of the input data when making the prediction.

Markov-Logic Network(MLN) #OTHEREFR uses probabilistic inference to estimate uncertainty in generated captions.

Captions are passed,attention mechanism is used to add visual context and is passed to MLN. This approach generated captions close to human-generated captions.

Initially, Image Captioning was tackled using Recurrent Neural Networks (RNNs) #OTHEREFR and Long-Short Term Memory (LSTM) networks.

Although these methods generated captions effectively, they faced the issue of vanishing gradients where the gradients decreased as the model processed longer sequences.

To address this challenge, researchers proposed the use of an Attention Mechanism in conjunction with RNNs or LSTMs, allowing the network to focus on the most important parts of the input and make predictions based on relevant information.

The word "soft" implies it uses continuous distribution to weigh the importance of the different parts of input data.

On the contrary Hard attention selects one region at random and only focuses on that particular area.

Global attention #OTHEREFR which considers the entire input data globally without giving preference to any specific part.

It allows the model to learn from all parts of the input and make predictions based on a holistic understanding of the data.

In Local Attention #OTHEREFR prior to computing the attention weight in the windows to the left and right of the selected point for alignment, first determines a position for alignment. It then gives the context vector a weight.

Comparative Study Of Transformer And Lstm Network With Attention Mechanism On Image Captioning

Abstract: In a globalized world at the present epoch of generative intelligence, most of the manual labour tasks are automated with increased efficiency. This can support businesses to save time and money. A crucial component of generative intelligence is the integration of vision and language. Consequently, image captioning become an intriguing area of research. There have been multiple attempts by the researchers to solve this problem with different deep learning architectures, although the accuracy has increased, but the results are still not up to standard. This study buckles down to the comparison of Transformer and LSTM with attention block model on MS-COCO dataset, which is a standard dataset for image captioning. For both the models we have used pretrained Inception-V3 CNN encoder for feature extraction of the images. The Bilingual Evaluation Understudy score (BLEU) is used to checked the accuracy of caption generated by both models. Along with the transformer and LSTM with attention block models,CLIP-diffusion model, M2-Transformer model and the X-Linear Attention model have been discussed with state of the art accuracy.

Neural Machine Translation By Jointly Learning To Align And Translate

Abstract: Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.


Extension To Robin Type Transmission Conditions

Similarly to the case of Dirichlet transmission conditions, we apply the restriction operator to problem #REFR .

where the operators A and B are defined as for (3) in (6), #OTHEREFR and the operator C takes the form,

If A and B are continuous operators, A is elliptic, B satisfies the infsup condition of Theorem 2, and C is of the form #OTHEREFR , then, owing to Theorem 4.3.2 of #OTHEREFR , problem (20) admits a unique solution u ∈

Using the notation defined before, we obtain the following abstract problem: given F ∈

We note that in the particular case where C is the identity, as in #OTHEREFR , the perturbation term becomes κ R T Λ, R T w .

The operators A, RB and RCR T satisfy the assumptions of Theorem 4.3.2 of #OTHEREFR .

Reduced Lagrange Multiplier Approach For Non-Matching Coupling Of Mixed-Dimensional Domains

Abstract: Many physical problems involving heterogeneous spatial scales, such as the flow through fractured porous media, the study of fiber-reinforced materials or the modeling of the small circulation in living tissues-just to mention a few examples-can be described as coupled partial differential equations defined in domains of heterogeneous dimensions that are embedded into each other. This formulation is a consequence of geometric model reduction techniques that transform the original problems defined in complex three-dimensional domains into more tractable ones. The definition and the approximation of coupling operators suitable for this class of problems is still a challenge. The main objective of this work is to develop a general mathematical framework for the analysis and the approximation of partial differential equations coupled by nonmatching constraints across different dimensions. Considering the non standard formulation of the coupling conditions, we focus on their enforcement using Lagrange multipliers. In this context we address in abstract and general terms the well posedness, the stability, the robustness of the problem with respect to the smallest characteristic length of the embedded domain. We also address the the numerical approximation of the problem and we discuss the inf-sup stability of the proposed numerical scheme for some representative configuration of the embedded domain. The main message of this work is twofold: from the standpoint of the theory of mixed-dimensional problems, we provide general and abstract mathematical tools to formulate coupled problems across dimensions. From the practical standpoint of the numerical approximation, we show the interplay between the mesh characteristic size, the dimension of the Lagrange multiplier space and the size of the inclusion in representative configurations interesting for applications. The latter analysis is complemented with illustrative numerical examples.

Error Estimates In Weighted Sobolev Norms For Finite Element Immersed Interface Methods

Abstract: When solving elliptic partial differential equations in a region containing immersed interfaces (possibly evolving in time), it is often desirable to approximate the problem using an independent background discretisation, not aligned with the interface itself. Optimal convergence rates are possible if the discretisation scheme is enriched by allowing the discrete solution to have jumps aligned with the surface, at the cost of a higher complexity in the implementation. A much simpler way to reformulate immersed interface problems consists in replacing the interface by a singular force field that produces the desired interface conditions, as done in immersed boundary methods. These methods are known to have inferior convergence properties, depending on the global regularity of the solution across the interface, when compared to enriched methods. In this work we prove that this detrimental effect on the convergence properties of the approximate solution is only a local phenomenon, restricted to a small neighbourhood of the interface. In particular we show that optimal approximations can be constructed in a natural and inexpensive way, simply by reformulating the problem in a distributionally consistent way, and by resorting to weighted norms when computing the global error of the approximation.


The MobileNet backbone and our ResNet-18 backbone have been shown to have very similar performance #REFR .

All of our experiments are based on the Cityscapes dataset #OTHEREFR during training and evaluation in the segmentation task, but they are used to evaluate the uncertainty outputs as they indicate the model's ability to distinguish between in-domain and out-of-domain samples.

Table 1 and Table 2 outline a quantitative comparison between the student's and the teacher's Intersection over Union (IoU) as well as their predictive uncertainties.

The results of Holder and Shafique #OTHEREFR have been included as they are the most relevant previous work on DE-based student-teacher distillation for efficient UQ.

Their teacher is based on 25 DeepLabv3+ models with a MobileNet backbone #OTHEREFR .

Dudes: Deep Uncertainty Distillation Using Ensembles For Semantic Segmentation

Abstract: Deep neural networks lack interpretability and tend to be overconfident, which poses a serious problem in safety-critical applications like autonomous driving, medical imaging, or machine vision tasks with high demands on reliability. Quantifying the predictive uncertainty is a promising endeavour to open up the use of deep neural networks for such applications. Unfortunately, current available methods are computationally expensive. In this work, we present a novel approach for efficient and reliable uncertainty estimation which we call Deep Uncertainty Distillation using Ensembles for Segmentation (DUDES). DUDES applies student-teacher distillation with a Deep Ensemble to accurately approximate predictive uncertainties with a single forward pass while maintaining simplicity and adaptability. Experimentally, DUDES accurately captures predictive uncertainties without sacrificing performance on the segmentation task and indicates impressive capabilities of identifying wrongly classified pixels and out-of-domain samples on the Cityscapes dataset. With DUDES, we manage to simultaneously simplify and outperform previous work on Deep Ensemble-based Uncertainty Distillation.

Benchmark Analysis Of Representative Deep Neural Network Architectures

Abstract: This work presents an in-depth analysis of the majority of the deep neural networks (DNNs) proposed in the state of the art for image recognition. For each DNN multiple performance indices are observed, such as recognition accuracy, model complexity, computational complexity, memory usage, and inference time. The behavior of such performance indices and some combinations of them are analyzed and discussed. To measure the indices we experiment the use of DNNs on two different computer architectures, a workstation equipped with a NVIDIA Titan X Pascal and an embedded system based on a NVIDIA Jetson TX1 board. This experimentation allows a direct comparison between DNNs running on machines with very different computational capacity. This study is useful for researchers to have a complete view of what solutions have been explored so far and in which research directions are worth exploring in the future; and for practitioners to select the DNN architecture(s) that better fit the resource constraints of practical deployments and applications. To complete this work, all the DNNs, as well as the software used for the analysis, are available online.