For MOD a convergence radius of order 1/S 2 was derived in #REFR , however, our results show that even under milder assumptions the basin of convergence is several orders of magnitude larger.

In particular, when assuming that the data follows a random sparse model with an overcomplete generating dictionary Φ, there are no global recovery guarantees, in the sense that one of the algorithms above or a variant will recover Φ with high probability.

Our Contribution In this paper we will cement the theoretical foundations of dictionary learning via alternating minimisation by characterising the convergence basin for both MOD and ODL (aK-SVD) in combination with thresholding as sparse approximation algorithm as summarised in Algorithm 1.1.

In particular we will show that under mild conditions on the generating dictionary, any initialisation within atom-wise ℓ 2 -norm and scaled operator norm distance of order 1/ log K will converge exponentially close to the generating dictionary given enough training signals.

Moreover, we provide further conditions on an initialisation that ensure convergence even for distances close to the maximal distance √ 2.

In case of ODL (aK-SVD) this is the first result of this kind.

Finally, our most important contribution is that we provide the first theoretical results which are valid for data generating models where the non-zero entries of the generating coefficients are not essentially uniformly distributed.

This means that some dictionary elements can be more likely to appear in the data representation than others.

Such a non-homogeneous use of dictionary elements can usually be observed in dictionaries learned on image or audio data.

Our results, proving the stability and convergence of alternating minimisation methods for non-uniform sparse support models, might be an explanation for their practical advantages over graph clustering or element-wise dictionary learning methods.

Organisation The remainder of the paper is organised as follows.

Convergence Of Alternating Minimisation Algorithms For Dictionary Learning

Abstract: In this paper we derive sufficient conditions for the convergence of two popular alternating minimisation algorithms for dictionary learning-the Method of Optimal Directions (MOD) and Online Dictionary Learning (ODL), which can also be thought of as approximative K-SVD. We show that given a well-behaved initialisation that is either within distance at most 1/ log(K) to the generating dictionary or has a special structure ensuring that each element of the initialisation only points to one generating element, both algorithms will converge with geometric convergence rate to the generating dictionary. This is done even for data models with non-uniform distributions on the supports of the sparse coefficients. These allow the appearance frequency of the dictionary elements to vary heavily and thus model real data more closely.

Learning Sparsely Used Overcomplete Dictionaries Via Alternating Minimization

Abstract: We consider the problem of sparse coding, where each sample consists of a sparse linear combination of a set of dictionary atoms, and the task is to learn both the dictionary elements and the mixing coefficients. Alternating minimization is a popular heuristic for sparse coding, where the dictionary and the coefficients are estimated in alternate steps, keeping the other fixed. Typically, the coefficients are estimated via $\ell_1$ minimization, keeping the dictionary fixed, and the dictionary is estimated through least squares, keeping the coefficients fixed. In this paper, we establish local linear convergence for this variant of alternating minimization and establish that the basin of attraction for the global optimum (corresponding to the true dictionary and the coefficients) is $\mathcal{O}(1/s^{2})$, where $s$ is the sparsity level in each sample and the dictionary satisfies restricted isometry property. Combined with the recent results of approximate dictionary estimation, this yields provable guar...


As emphasized in #REFR , the flow of the NODEs cannot represent some functions omnipresently emergent in applications.

#OTHEREFR calculated the gradients by calling an ODE solver with extended ODEs (i.e., concatenating the original state, the adjoint, and the other partial derivatives for the parameters at each time point into a single vector).

Notably, for the regression task of the time series, the loss function probably depends on the state at multiple observational times, such as the form of L(h(t 0 ), h(t 1 ), ..., h(t n )).

Under such a case, we must update the adjoint state instantly by adding the partial derivative of the loss at each observational time point.

Typical examples include the following two-valued function with one argument: g 1-D (1) = −1 and g 1-D (−1) = 1.

Our framework desires to conquer the representation limitation observed in applying the NODEs.

Optimal control As mentioned above, a closed connection between deep neural networks and dynamical systems has been emphasized in the literature and, correspondingly, theories, methods, and tools of dynamical systems have been employed, e.g.

the theory of optimal control #OTHEREFR - #OTHEREFR , #OTHEREFR - #OTHEREFR .

Generally, we model a typical task using a deep neural network and then train the network parameters such that the given loss function can be reduced by some learning algorithm.

Neural Delay Differential Equations: System Reconstruction And Image Classification

Abstract: Neural Ordinary Differential Equations (NODEs), a framework of continuous-depth neural networks, have been widely applied, showing exceptional efficacy in coping with representative datasets. Recently, an augmented framework has been developed to overcome some limitations that emerged in the application of the original framework. In this paper, we propose a new class of continuous-depth neural networks with delay, named Neural Delay Differential Equations (NDDEs). To compute the corresponding gradients, we use the adjoint sensitivity method to obtain the delayed dynamics of the adjoint. Differential equations with delays are typically seen as dynamical systems of infinite dimensions that possess more fruitful dynamics. Compared to NODEs, NDDEs have a stronger capacity for nonlinear representations. We use several illustrative examples to demonstrate this outstanding capacity. Firstly, we successfully model the delayed dynamics where the trajectories in the lower-dimensional phase space could be mutually intersected and even chaotic in a model-free or model-based manner. Traditional NODEs, without any argumentation, are not directly applicable to such modeling. Secondly, we achieve lower loss and higher accuracy not only for the data produced synthetically by complex models but also for the CIFAR10, a well-known image dataset. Our results on the NDDEs demonstrate that appropriately articulating the elements of dynamical systems into the network design is truly beneficial in promoting network performance.

Augmented Neural Odes

Abstract: We show that Neural Ordinary Differential Equations (ODEs) learn representations that preserve the topology of the input space and prove that this implies the existence of functions Neural ODEs cannot represent. To address these limitations, we introduce Augmented Neural ODEs which, in addition to being more expressive models, are empirically more stable, generalize better and have a lower computational cost than Neural ODEs.


Second, to accommodate the pixel-value representation necessary for image processing, we normalize the range of pixel values from (0, 255) to (0, 1) to train the deep learning model correctly #REFR .

To begin, a random 60%, 20%, 10%, and 10% split is made between the training, validation, testing A, and testing B sets from the specified dataset.

It can be characterized as follows: In this paper, we use pre-trained feature extractors for easy modeling because it may be difficult to train a deep learning model for recognizing COVID-19 infection from scratch due to a lack of CT scans of COVID-19 subjects.

The transfer learning method and several different pretrained models address this problem #OTHEREFR .

One of transfer learning's key benefits is that it allows data to be trained with fewer samples and in less time #OTHEREFR .

And it allows a newly trained model to benefit from the experience of a previously trained model #OTHEREFR .

Table 3 presents the pre-trained part and the novelty portion of our MEDNC. DenseNet169.

Mednc: Multi-Ensemble Deep Neural Network For Covid-19 Diagnosis

Abstract: Coronavirus disease 2019 (COVID-19) has spread all over the world for three years, but medical facilities in many areas still aren't adequate. There is a need for rapid COVID-19 diagnosis to identify highrisk patients and maximize the use of limited medical resources. Motivated by this fact, we proposed the deep learning framework MEDNC for automatic prediction and diagnosis of COVID-19 using computed tomography (CT) images. Our model was trained using two publicly available sets of COVID-19 data. And it was built with the inspiration of transfer learning. Results indicated that the MEDNC greatly enhanced the detection of COVID-19 infections, reaching an accuracy of 98.79% and 99.82% respectively. We tested MEDNC on a brain tumor and a blood cell dataset to show that our model applies to a wide range of problems. The outcomes demonstrated that our proposed models attained an accuracy of 99.39% and 99.28%, respectively. This COVID-19 recognition tool could help optimize healthcare resources and reduce clinicians' workload when screening for the virus.

Pixel Normalization From Numeric Data As Input To Neural Networks: For Machine Learning And Image Processing

Abstract: Text to image transformation for input to neural networks requires intermediate steps. This paper attempts to present a new approach to pixel normalization so as to convert textual data into image, suitable as input for neural networks. This method can be further improved by its Graphics Processing Unit (GPU) implementation to provide significant speedup in computational time.


In what follows we will mainly highlight the difference between the construction by #REFR and ours.

We now provide an overview for our additive spanner construction.

First, we apply the clustering algorithm from #OTHEREFR with radius R to G and obtain a set B of balls.

Similar to the construction of sublinear and pairwise additive spanners, while processing unsettled pairs of vertices, we will add demand pairs to the corresponding balls that partition their shortest paths.

It has been shown that O( √ n) demand pairs can be preserved exactly with O(n)

edges #OTHEREFR , and the bound O( √ n) is cannot be improved, so we need to ensure that each ball B collects at most O(|B| 1/2 ) demand pairs.

On the one hand, the small balls B with |B| ≤ R 4/3 can be handled using previous work on subset distance preservers #OTHEREFR .

Almost-Optimal Sublinear Additive Spanners

Abstract: Given an undirected unweighted graph G = (V, E) on n vertices and m edges, a subgraph H ⊆ G is a spanner of G with stretch function f : R + → R + , iff for every pair s, t of vertices in V , dist H (s, t) ≤ f (dist G (s, t)). When f (d) = d + o(d), H is called a sublinear additive spanner ; when f (d) = d + o(n), H is called an additive spanner, and f (d) − d is usually called the additive stretch of H. As our primary result, we show that for any constant δ > 0 and constant integer k ≥ 2, every graph on n vertices has a sublinear additive spanner with stretch function f (d) = d + O(d 1−1/k) and O n 1+ 1+δ 2 k+1 −1 edges. When k = 2, this improves upon the previous spanner construction with stretch function f (d) = d + O(d 1/2) andÕ(n 1+3/17) edges [Chechik, 2013]; for any constant integer k ≥ 3, this improves upon the previous spanner construction with stretch function f (d) = d + O(d 1−1/k) and O n

Better Distance Preservers And Additive Spanners

Abstract: We make improvements to the upper bounds on several popular types of distance preserving graph sketches. These sketches are all various restrictions of the {\em additive pairwise spanner} problem, in which one is given an undirected unweighted graph $G$, a set of node pairs $P$, and an error allowance $+\beta$, and one must construct a sparse subgraph $H$ satisfying $\delta_H(u, v) \le \delta_G(u, v) + \beta$ for all $(u, v) \in P$. The first part of our paper concerns {\em pairwise distance preservers}, which make the restriction $\beta=0$ (i.e. distances must be preserved {\em exactly}). Our main result here is an upper bound of $|H| = O(n^{2/3}|P|^{2/3} + n|P|^{1/3})$ when $G$ is undirected and unweighted. This improves on existing bounds whenever $|P| = \omega(n^{3/4})$, and it is the first such improvement in the last ten years. We then devise a new application of distance preservers to graph clustering algorithms, and we apply this algorithm to {\em subset spanners}, which require $P = S \times S$ for some node subset $S$, and {\em (standard) spanners}, which require $P = V \times V$. For both of these objects, our construction generalizes the best known bounds when the error allowance is constant, and we obtain the strongest polynomial error/sparsity tradeoff that has yet been reported (in fact, for subset spanners, ours is the {\em first} nontrivial construction that enjoys improved sparsity from a polynomial error allowance). We leave open a conjecture that $O(n^{2/3}|P|^{2/3} + n)$ pairwise distance preservers are possible for undirected unweighted graphs. Resolving this conjecture in the affirmative would improve and simplify our upper bounds for all the graph sketches mentioned above.


The other methods were trained on ShapeNet #REFR with 3D supervision since their learning process cannot scale to unknown, large environments.

To compare the different methods, we follow #OTHEREFR and compute the area under the curve of the evolution of the total surface coverage during exploration, after 100 NBV iterations, as presented in Table 1 .

The surface coverage is computed using the ground truth meshes.

AUCs are averaged on multiple trajectories in each scene: We use the same starting camera poses and same sets of candidate camera poses C t for each method for fair comparison.

During this experiment, all weights of MACARONS are frozen and we only perform inference computation.

The model was trained on a set of previous scenes: The 8 scenes above the bar in Table 1 were seen during online training (but with different, random starting camera poses and trajectories), and the 4 scenes below the bar were not.

Even if it only uses RGB images, our model is able to outperform the baselines in large environments since, contrary to other methods, its self-supervised online training strategy allows it to scale its learning process to any kind of environment.

Figures 4 and 5 show examples of trajectories as well as of surface reconstructions computed with MACARONS.

Macarons: Mapping And Coverage Anticipation With Rgb Online Self-Supervision

Abstract: Figure 1. Mapping and Coverage Anticipation with RGB Online Self-Supervision. (a) NBV methods such as [27] rely on a depth sensor to perform path planning (bottom) and scan the environment (top). They need to be trained with explicit 3D supervision, generally on small-scale meshes. (b) Our approach MACARONS instead simultaneously learns to efficiently explore the scene and to reconstruct it (top) using an RGB sensor only. Its self-supervised, online learning process scales to large-scale and complex scenes.

Shapenet: An Information-Rich 3D Model Repository

Abstract: We present ShapeNet: a richly-annotated, large-scale repository of shapes represented by 3D CAD models of objects. ShapeNet contains 3D models from a multitude of semantic categories and organizes them under the WordNet taxonomy. It is a collection of datasets providing many semantic annotations for each 3D model such as consistent rigid alignments, parts and bilateral symmetry planes, physical sizes, keywords, as well as other planned annotations. Annotations are made available through a public web-based interface to enable data visualization of object attributes, promote data-driven geometric analysis, and provide a large-scale quantitative benchmark for research in computer graphics and vision. At the time of this technical report, ShapeNet has indexed more than 3,000,000 models, 220,000 models out of which are classified into 3,135 categories (WordNet synsets). In this report we describe the ShapeNet effort as a whole, provide details for all currently available datasets, and summarize future plans.


Comparison With Sota Lightweight Generic Detectors

TinyDet-L achieves the same 18.3 AP s with YOLOv3 #REFR , but it only has 1/30 computation cost of YOLOv3. Some visualized results are shown in Fig. 7 .

Among the compared methods, ThunderNet #OTHEREFR and Efficient-Det #OTHEREFR are regarded as the most recent SOTA lightweight detectors.

The results show that our models obviously outperform them with fewer computation costs: TinyDet-S surpasses ThunderNet-SNet146 by 2.4% AP with similar FLOPs; TinyDet-M surpasses ThunderNet-SNet535 by 2.3% AP with 76% FLOPs; and TinyDet-L surpasses EfficientDet-D0 by 3.1% AP with similar FLOPs.

Besides, TinyDet has better performance-computation tradeoffs than the automatically searched lightweight detec-tor (i.e., NAS-FPNLite #OTHEREFR ) and the popular YOLO series #OTHEREFR .

Our TinyDet models obtain extraordinary performance on detecting small objects.

With similar computation cost (i.e., FLOPs), TinyDet-M achieves 13.5 AP s , which is over 100% improvement over ThunderNet-SNet535 with 6.5 AP s ; and TinyDet-S achieves 9.6 AP s , which is also over 100% improvement over ThunderNet-SNet146 with 4.6 AP s .

Tinydet: Accurate Small Object Detection In Lightweight Generic Detectors

Abstract: Small object detection requires the detection head to scan a large number of positions on image feature maps, which is extremely hard for computation-and energyefficient lightweight generic detectors. To accurately detect small objects with limited computation, we propose a two-stage lightweight detection framework with extremely low computation complexity, termed as TinyDet. It enables high-resolution feature maps for dense anchoring to better cover small objects, proposes a sparsely-connected convolution for computation reduction, enhances the early stage features in the backbone, and addresses the feature misalignment problem for accurate small object detection. On the COCO benchmark, our TinyDet-M achieves 30.3 AP and 13.5 AP s with only 991 MFLOPs, which is the first detector that has an AP over 30 with less than 1 GFLOPs; besides, TinyDet-S and TinyDet-L achieve promising performance under different computation limitation.

Yolov3: An Incremental Improvement

Abstract: We present some updates to YOLO! We made a bunch of little design changes to make it better. We also trained this new network that's pretty swell. It's a little bigger than last time but more accurate. It's still fast though, don't worry. At 320x320 YOLOv3 runs in 22 ms at 28.2 mAP, as accurate as SSD but three times faster. When we look at the old .5 IOU mAP detection metric YOLOv3 is quite good. It achieves 57.9 mAP@50 in 51 ms on a Titan X, compared to 57.5 mAP@50 in 198 ms by RetinaNet, similar performance but 3.8x faster. As always, all the code is online at this https URL


C.3. Explanation And Intuition For Capacity-Criticality-Tradeoff

Further, we note that the aforementioned observation resonates with the findings made in #REFR -neurons in the middle layers of a network (i.e., Stage2 and Stage3) exhibit fragile co-adaption, which is difficult to re-learn.

To gain a better understanding of why having a higher fraction of Stage1 ReLUs is preferable for achieving superior performance at lower ReLU counts, we examined the ReLU dropping strategies employed in the prevalent ReLU optimization techniques.

Specifically, we examined the strategies employed by DeepReDuce #OTHEREFR , SNL #OTHEREFR , and SENet #OTHEREFR , and found that they consistently demonstrate that Stage1 ReLUs are the least critical, and as such, all Stage1 ReLUs are dropped first to achieve very low ReLU counts.

Consequently, networks with a greater proportion of non-critical ReLUs will drop a lower fraction of their critical ReLUs when aiming for very low ReLU counts, as opposed to networks with a lower proportion of non-critical ReLUs. This can be observed in Figure 17 .

In particular, regardless of the total ReLU count of the network, WRN22x8 (ResNet18(m=16)-4x4x4x) drops a higher fraction of their critical ReLUs compared to ResNet18 (ResNet34) in order to attain a ReLU count of 25K.

Thus, dropping a higher fraction of critical ReLUs lead to a significant loss in accuracy and results in inferior performance.

Consequently, dropping more ReLUs from these stages in a network with lower fraction of non-critical (Stage1) ReLUs would disrupt the fragile co-adaption and significantly reduce performance.

Deepreshape: Redesigning Neural Networks For Efficient Private Inference

Abstract: The increasing demand for privacy and security has driven the advancement of private inference (PI), a cryptographic method enabling inferences directly on encrypted data. However, the computational and storage burdens of non-linear operators (e.g., ReLUs) render it impractical. Despite these limitations, prior ReLU optimization methods consistently relied on classical networks, that are not optimized for PI. Moreover, the selection of baseline networks in these ReLU optimization methods remains enigmatic and fails to provide insights into network attributes contributing to PI efficiency. In this paper, we investigate the desirable network architecture for efficient PI, and key finding is wider networks are superior at higher ReLU counts, while networks with a greater proportion of least-critical ReLUs excel at lower ReLU counts. Leveraging these findings, we develop a novel network redesign technique (DeepReShape) with a complexity of O(1), and synthesize specialized architectures(HybReNet). Compared to the state-of-the-art (SNL on CIFAR-100), we achieve a 2.35% accuracy gain at 180K ReLUs, and for ResNet50 on TinyImageNet our method saves 4.2× ReLUs at iso-accuracy.

How Transferable Are Features In Deep Neural Networks?

Abstract: Many deep neural networks trained on natural images exhibit a curious phenomenon in common: on the first layer they learn features similar to Gabor filters and color blobs. Such first-layer features appear not to be specific to a particular dataset or task, but general in that they are applicable to many datasets and tasks. Features must eventually transition from general to specific by the last layer of the network, but this transition has not been studied extensively. In this paper we experimentally quantify the generality versus specificity of neurons in each layer of a deep convolutional neural network and report a few surprising results. Transferability is negatively affected by two distinct issues: (1) the specialization of higher layer neurons to their original task at the expense of performance on the target task, which was expected, and (2) optimization difficulties related to splitting networks between co-adapted neurons, which was not expected. In an example network trained on ImageNet, we demonstrate that either of these two issues may dominate, depending on whether features are transferred from the bottom, middle, or top of the network. We also document that the transferability of features decreases as the distance between the base task and target task increases, but that transferring features even from distant tasks can be better than using random features. A final surprising result is that initializing a network with transferred features from almost any number of layers can produce a boost to generalization that lingers even after fine-tuning to the target dataset.


An equivalent static graph G = {V , E S , E T } is obtained by modelling temporal edges #REFR .

Since applying convolution on a static graph is generally easier than encoding across multiple snapshots, the DG encoding problem is frequently transformed into encoding a static graph where each node is connected to itself in the adjacent snapshot #OTHEREFR .

This approach can also be interpreted as constructing a time-expanded graph #OTHEREFR or node-oriented ESG (see section 2.2.4) and is widely used to encode Trans-fix V,E cases, i.e. STGs.

In more complex configurations, nodes are also connected with their k-hop neighbours in the adjacent snapshot(s) #OTHEREFR .

A simple example of such a strategy is shown in fig. 8 .

G has Once defined the connection rule for temporal edges, the traditional convolution for static graphs is applicable on ESGs. A state-of-the-art example is the ST-GCN module #OTHEREFR .

To update the hidden states h to h in an ST-GCN layer (see Eqn.

6), a typical spatial GNN structure aggregates the neighbour features with the msg(•) function, computes their weights by the w(•) function, and then sums them after normalisation with the norm(•) function.

Note that, unlike the neighbourhood definition in static graphs, the neighbourhood set N of v t i is defined by: (1) spatially, the shortest path distance with neighbour v j obeys d(v j , v i ) ≤ K and (2) temporally, the time difference between timestamps q and t, i.e. |q − t| is not greater than Γ/2 .

#OTHEREFR , i.e., in a stacked way #OTHEREFR , or incorporates graph convolution when encoding each snapshot across time #OTHEREFR , i.e., in an integrated way #OTHEREFR .

Dynamic Graph Representation Learning With Neural Networks: A Survey

Abstract: In recent years, Dynamic Graph (DG) representations have been increasingly used for modeling dynamic systems due to their ability to integrate both topological and temporal information in a compact representation. Dynamic graphs allow to efficiently handle applications such as social network prediction, recommender systems, traffic forecasting or electroencephalography analysis, that can not be adressed using standard numeric representations. As a direct consequence of the emergence of dynamic graph representations, dynamic graph learning has emerged as a new machine learning problem, combining challenges from both sequential/temporal data processing and static graph learning. In this research area, Dynamic Graph Neural Network (DGNN) has became the state of the art approach and plethora of models have been proposed in the very recent years. This paper aims at providing a review of problems and models related to dynamic graph learning. The various dynamic graph supervised learning settings are analysed and discussed. We identify the similarities and differences between existing models with respect to the way time information is modeled. Finally, general guidelines for a DGNN designer when faced with a dynamic graph learning problem are provided.

Structural-Rnn: Deep Learning On Spatio-Temporal Graphs

Abstract: Deep Recurrent Neural Network architectures, though remarkably capable at modeling sequences, lack an intuitive high-level spatio-temporal structure. That is while many problems in computer vision inherently have an underlying high-level structure and can benefit from it. Spatio-temporal graphs are a popular tool for imposing such high-level intuitions in the formulation of real world problems. In this paper, we propose an approach for combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural Networks~(RNNs). We develop a scalable method for casting an arbitrary spatio-temporal graph as a rich RNN mixture that is feedforward, fully differentiable, and jointly trainable. The proposed method is generic and principled as it can be used for transforming any spatio-temporal graph through employing a certain set of well defined steps. The evaluations of the proposed approach on a diverse set of problems, ranging from modeling human motion to object interactions, shows improvement over the state-of-the-art with a large margin. We expect this method to empower new approaches to problem formulation through high-level spatio-temporal graphs and Recurrent Neural Networks.


There is no evidence that modifying the Gaussian noise standard deviation at onset meaningfully changes the result, which is in line with #REFR who find that varying this parameter at runtime gives no difference.

A sample agent might be denoted as 1-0-1-0-0-0-0-0-0-0 if it won the first and third levels and no others.

1024 agents are possible for most trials, 2 20 = 1, 048, 576 when the key win is treated separately.

• Most Levels Solved refers to the highest count of levels solved by any individual agent. For 1-0-1-0-0-0-0-0-0-0, this number is 2.

Tables 1 through 4 show evaluation for Vanilla Map Elites.

Although longrunning evaluations did give rise to 10 agents, no agent came close to solving all levels.

Switching to a tile map view and including an explore reward did not change the highest count of levels solved.

However, slightly relaxing the steps per level and adding partial map elites cells for gaining a key allowed the agent to solve levels far sooner (at 80K vs 400K iterations).

Including key cells did have a meaningful difference in the number of elites in the map, including 6 elites which acquired the key in a level other than the one solved.

Using prior knowledge to identify intermediate points in level solving is highly beneficial in solving levels more quickly and in increasing the diversity of solutions tracked in the map elites map.

Exploring Novel Quality Diversity Methods For Generalization In Reinforcement Learning

Abstract: The Reinforcement Learning field is strong on achievements and weak on reapplication; a computer playing GO at a superhuman level is still terrible at Tic-Tac-Toe. This paper asks whether the method of training networks improves their generalization. Specifically we explore core quality diversity algorithms, compare against two recent algorithms, and propose a new algorithm to deal with shortcomings in existing methods. Although results of these methods are well below the performance hoped for, our work raises important points about the choice of behavior criterion in quality diversity, the interaction of differential and evolutionary training methods, and the role of offline reinforcement learning and randomized learning in evolutionary search.

Evolution Strategies As A Scalable Alternative To Reinforcement Learning

Abstract: We explore the use of Evolution Strategies (ES), a class of black box optimization algorithms, as an alternative to popular MDP-based RL techniques such as Q-learning and Policy Gradients. Experiments on MuJoCo and Atari show that ES is a viable solution strategy that scales extremely well with the number of CPUs available: By using a novel communication strategy based on common random numbers, our ES implementation only needs to communicate scalars, making it possible to scale to over a thousand parallel workers. This allows us to solve 3D humanoid walking in 10 minutes and obtain competitive results on most Atari games after one hour of training. In addition, we highlight several advantages of ES as a black box optimization technique: it is invariant to action frequency and delayed rewards, tolerant of extremely long horizons, and does not need temporal discounting or value function approximation.


These color shifts disappear when a larger image quality factor λ or a more diverse dataset is used (a similar observation was made by #REFR for image transformations).

Essentially LISO's ability to learn from past optimizations and to stay on the image manifold, allows it to achieve low error rates, high image quality, and initializations (for further L-BFGS optimization) near a good minimum.

Figure 4 shows the steganographic images produced by LISO under different payloads.

Qualitatively we observe that the image quality is quite good even when a large number of bits per pixel are hidden in the cover images.

The CelebA example shows a rare example where the hue changes slightly while the image remains un-pixelated and natural; in contrast, the low-quality steganographic images produced by FNNS tend to be pixelated.

Different from previous image steganography methods that produce unnatural Gaussian noise, LISO's perturbation is smooth and natural because LISO implicitly learns the manifold of natural images and produces steganographic images that stay close it.

We can also explicitly control the trade-off between error rate and image quality by changing the image quality factors λ and µ on the loss terms if desired.

Additional qualitative images obtained by varying these are provided in Appendix J. Optimization with LISO.

In Figure 3 , we compare optimizing with the LISO encoder against popular numerical optimization algorithms including projected gradient descent #OTHEREFR and L-BFGS #OTHEREFR ) on the steganographic encoding task.

Note that L-BFGS-Random is equivalent to FNNS-R and L-BFGS-Pretrained is equivalent to FNNS-D (Kishore et al., 2021).

LISO converges faster than any other optimization algorithm and this validates our hypothesis that LISO can learn a descent direction that is better than the one found by common gradientbased optimization methods; LISO converges to 100× lower loss and 80× lower error rate.

Learning Iterative Neural Optimizers For Im-Age Steganography

Abstract: Image steganography is the process of concealing secret information in images through imperceptible changes. Recent work has formulated this task as a classic constrained optimization problem. In this paper, we argue that image steganography is inherently performed on the (elusive) manifold of natural images, and propose an iterative neural network trained to perform the optimization steps. In contrast to classical optimization methods like L-BFGS or projected gradient descent, we train the neural network to also stay close to the manifold of natural images throughout the optimization. We show that our learned neural optimization is faster and more reliable than classical optimization approaches. In comparison to previous state-of-the-art encoder-decoder based steganography methods, it reduces the recovery error rate by multiple orders of magnitude and achieves zero error up to 3 bits per pixel (bpp) without the need for error-correcting codes.

Deep Feature Interpolation For Image Content Changes

Abstract: We propose Deep Feature Interpolation (DFI), a new data-driven baseline for automatic high-resolution image transformation. As the name suggests, it relies only on simple linear interpolation of deep convolutional features from pre-trained convnets. We show that despite its simplicity, DFI can perform high-level semantic transformations like "make older/younger", "make bespectacled", "add smile", among others, surprisingly well - sometimes even matching or outperforming the state-of-the-art. This is particularly unexpected as DFI requires no specialized network architecture or even any deep network to be trained for these tasks. DFI therefore can be used as a new baseline to evaluate more complex algorithms and provides a practical answer to the question of which image transformation tasks are still challenging in the rise of deep learning.


We inject self-supervision into video similarity learning by adopting the concept of instancediscrimination #REFR , where each video forms its own class, and any transformation of it preserves the class label.

In this work, three cases are considered: i) video copies #OTHEREFR , i.e., edited versions of the same source video, ii) videos of the same incident #OTHEREFR , i.e., videos capturing the same spatio-temporal span, and iii) videos of the same event #OTHEREFR , i.e., videos capturing the same spatial or temporal span. In this work, we target both retrieval and detection.

In the former, only ranking per query matters; therefore, the distribution of similarities can vary among queries.

While in the latter, the ability to apply a similarity threshold and detect relevant videos matters.

Supervised learning of specialised models per task is very demanding in terms of training data collection, especially in the video domain.

Instead, in the proposed method, we are learning a single model via SSL to perform all retrieval and detection tasks (see Figure 1 ) without further fine-tuning.

In this work, we adopt the ViSiL #OTHEREFR architecture for video similarity, which needs labeled video datasets for its development in prior works #OTHEREFR , but we train it in a self-supervised way and argue that instance-discrimination through augmentations is well suited for all the aforementioned tasks.

To pronounce the synergy, we develop an appropriate composition of video augmentations and propose a model-tailored loss combined with a standard SSL loss.

By eliminating the need for video annotations, we are able to train on large video datasets and achieve state-of-the-art results on all target retrieval and detection tasks.

Evaluation is performed on three standard benchmarks, namely, VCDB #OTHEREFR , FIVR #OTHEREFR , and EVVE #OTHEREFR .

In summary, our contributions include the following:

Self-Supervised Video Similarity Learning

Abstract: We introduce S 2 VS, a video similarity learning approach with self-supervision. Self-Supervised Learning (SSL) is typically used to train deep models on a proxy task so as to have strong transferability on target tasks after fine-tuning. Here, in contrast to prior work, SSL is used to perform video similarity learning and address multiple retrieval and detection tasks at once with no use of labeled data. This is achieved by learning via instance-discrimination with tasktailored augmentations and the widely used InfoNCE loss together with an additional loss operating jointly on selfsimilarity and hard-negative similarity. We benchmark our method on tasks where video relevance is defined with varying granularity, ranging from video copies to videos depicting the same incident or event. We learn a single universal model that achieves state-of-the-art performance on all tasks, surpassing previously proposed methods that use labeled data. The code and pretrained models are publicly

A Simple Framework For Contrastive Learning Of Visual Representations

Abstract: This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.


Indeed, several monolingual and multilingual BERT models are available that encompass a number of languages #REFR .

Contemporary NLP methods such as BERT #OTHEREFR , with the large amount of knowledge contained within their parameters, paired with the relatively low computational power required to finetune them for a downstream task, have taken over many NLP applications.

However, the strength of these models is tied to the availability of data, and the large amounts of data required to pre-train such models exclude some languages for which it is difficult to collect large amounts of written text.

The scarcity of data becomes more severe with dialects and language varieties.

In fact, the very nature of dialects as an evolving form of the language, often spoken rather than written, with various social and cultural nuances, can make it difficult to develop systems tailored to specific dialects.

In many applications, users may span a continuum of idiolects, some falling into established dialects and others not.

It may therefore be impossible to train a system on even a small amount of data in every idiolect.

Fine-Tuning Bert With Character-Level Noise For Zero-Shot Transfer To Dialects And Closely-Related Languages

Abstract: In this work, we induce character-level noise in various forms when fine-tuning BERT to enable zero-shot cross-lingual transfer to unseen dialects and languages. We fine-tune BERT on three sentence-level classification tasks and evaluate our approach on an assortment of unseen dialects and languages. We find that character-level noise can be an extremely effective agent of cross-lingual transfer under certain conditions, while it is not as helpful in others. Specifically, we explore these differences in terms of the nature of the task and the relationships between source and target languages, finding that introduction of characterlevel noise during fine-tuning is particularly helpful when a task draws on surface level cues and the source-target cross-lingual pair has a relatively high lexical overlap with shorter (i.e., less meaningful) unseen tokens on average.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


As an alternative to reinforcement learning where model parameters are updated by PPO, the reward model can also be used at inference time by, for instance, rejecting the worst responses generated by the model #REFR .

With the emergence of pre-trained large language models (Zhu, 2022), social chatbots have recently been dominated by transformer-based #OTHEREFR designs #OTHEREFR .

Typically, transformerbased chatbots are fine-tuned on conversational data in a specific domain, e.g. GPT-2 finetuned on conversational reddit data #OTHEREFR .

Over the last few years, larger and better transformerbased models have been trained on conversational data for the development of more sophisticated social chatbots #OTHEREFR .

Learning using Human Feedback: Instead of purely fine-tuning large language models on conversational data, incorporating human feedback in chatbot development has been shown to be an effective method to obtain more human-aligned responses #OTHEREFR Askell et al., 2021; #OTHEREFR .

Popularized by the training of In-structGPT #OTHEREFR , reinforcement learning from human feedback (RLHF; #OTHEREFR ) trains a reward model on user ranks of system responses, followed by reinforcement learning as an agent interacting with the environment.

Human feedback has also been used in several other ways including constrained optimization (Achiam et al., 2017) , control codes #OTHEREFR and expert iteration #OTHEREFR .

Chatbot Evaluation: The traditional method of assessing chatbots is through direct human assessment.

This can be done through rating individual responses #OTHEREFR , scoring aggregated qualities across the multi-turn exchange, or through ranking overall conversational experiences #OTHEREFR .

Alternatively, text overlap based metrics such as BLEU, METEOR and ROUGE #OTHEREFR and F-score metrics #OTHEREFR have also been proposed to assess chatbots.

Other assessment methods #OTHEREFR include sen-tence perplexity #OTHEREFR John et al., 2017; #OTHEREFR , entities per exchange #OTHEREFR , number of questions raised, specificity #OTHEREFR , turns per conversation #OTHEREFR , inconsistency detection, and relevance to history.

Rewarding Chatbots For Real-World Engagement With Millions Of Users

Abstract: The emergence of pretrained large language models has led to the deployment of a range of social chatbots for chitchat. Although these chatbots demonstrate language ability and fluency, they are not guaranteed to be engaging and can struggle to retain users. This work investigates the development of social chatbots that prioritize user engagement to enhance retention, specifically examining the use of human feedback to efficiently develop highly engaging chatbots. The proposed approach uses automatic pseudo-labels collected from user interactions to train a reward model that can be used to reject low-scoring sample responses generated by the chatbot model at inference time. Intuitive evaluation metrics, such as mean conversation length (MCL), are introduced as proxies to measure the level of engagement of deployed chatbots. A/B testing on groups of 10,000 new daily chatbot users on the Chai Research platform shows that this approach increases the MCL by up to 70%, which translates to a more than 30% increase in user retention for a GPT-J 6B model. Future work aims to use the reward model to realise a data flywheel , where the latest user conversations can be used to alternately fine-tune the language model and the reward model 1 .

Plug And Play Language Models: A Simple Approach To Controlled Text Generation

Abstract: Large transformer-based language models (LMs) trained on huge text corpora have shown unparalleled generation capabilities. However, controlling attributes of the generated language (e.g. switching topic or sentiment) is difficult without modifying the model architecture or fine-tuning on attribute-specific data and entailing the significant cost of retraining. We propose a simple alternative: the Plug and Play Language Model (PPLM) for controllable language generation, which combines a pretrained LM with one or more simple attribute classifiers that guide text generation without any further training of the LM. In the canonical scenario we present, the attribute models are simple classifiers consisting of a user-specified bag of words or a single learned layer with 100,000 times fewer parameters than the LM. Sampling entails a forward and backward pass in which gradients from the attribute model push the LM's hidden activations and thus guide the generation. Model samples demonstrate control over a range of topics and sentiment styles, and extensive automated and human annotated evaluations show attribute alignment and fluency. PPLMs are flexible in that any combination of differentiable attribute models may be used to steer text generation, which will allow for diverse and creative applications beyond the examples given in this paper.


Setup and Implementation Details: We validate the proposed method on two image datasets: PACS #REFR , a widelyused benchmark for domain generalization with four domains: Photo (P), Art Painting (A), Cartoon (C), and Sketch (S).

VLCS #OTHEREFR , another commonly adopted benchmark for domain generalization with four different domains: VOC2007 (V), LabelMe (L), Caltech101 (C), SUN09 (S).

In our implementation, we adopt the ResNet-18 #OTHEREFR pretrained on ImageNet #OTHEREFR as backbone. We apply the SGD optimizer with momentum 0.9. The batch size is set to 64. We set the learning rate to 2e-3/1e-3 for PACS/VLCS.

Experiments are conducted on a Tesla P40 GPU with PyTorch-1.5.

Following #OTHEREFR , we split the training domain into training and validation subsets, and select the best-performing model on validation set to report the OOD performances.

Experiment Results: We first conduct experiments on PACS, shown in Table 1 .

Modality-Agnostic Debiasing For Single Domain Generalization

Abstract: Deep neural networks (DNNs) usually fail to generalize well to outside of distribution (OOD) data, especially in the extreme case of single domain generalization (single-DG) that transfers DNNs from single domain to multiple unseen domains. Existing single-DG techniques commonly devise various data-augmentation algorithms, and remould the multi-source domain generalization methodology to learn domain-generalized (semantic) features. Nevertheless, these methods are typically modality-specific, thereby being only applicable to one single modality (e.g., image). In contrast, we target a versatile Modality-Agnostic Debiasing (MAD) framework for single-DG, that enables generalization for different modalities. Technically, MAD introduces a novel two-branch classifier: a biased-branch encourages the classifier to identify the domain-specific (superficial) features, and a general-branch captures domaingeneralized features based on the knowledge from biasedbranch. Our MAD is appealing in view that it is pluggable to most single-DG models. We validate the superiority of our MAD in a variety of single-DG scenarios with different modalities, including recognition on 1D texts, 2D images, 3D point clouds, and semantic segmentation on 2D images. More remarkably, for recognition on 3D point clouds and semantic segmentation on 2D images, MAD improves DSU by 2.82% and 1.5% in accuracy and mIOU.

Deeper, Broader And Artier Domain Generalization

Abstract: The problem of domain generalization is to learn from multiple training domains, and extract a domain-agnostic model that can then be applied to an unseen domain. Domain generalization (DG) has a clear motivation in contexts where there are target domains with distinct characteristics, yet sparse data for training. For example recognition in sketch images, which are distinctly more abstract and rarer than photos. Nevertheless, DG methods have primarily been evaluated on photo-only benchmarks focusing on alleviating the dataset bias where both problems of domain distinctiveness and data sparsity can be minimal. We argue that these benchmarks are overly straightforward, and show that simple deep learning baselines perform surprisingly well on them. In this paper, we make two main contributions: Firstly, we build upon the favorable domain shift-robust properties of deep learning methods, and develop a low-rank parameterized CNN model for end-to-end DG learning. Secondly, we develop a DG benchmark dataset covering photo, sketch, cartoon and painting domains. This is both more practically relevant, and harder (bigger domain shift) than existing benchmarks. The results show that our method outperforms existing DG alternatives, and our dataset provides a more significant DG challenge to drive future research.


We deploy the teleoperation system introduced in [58] to manipulate articulated objects in the Sapien #REFR simulation.

To the best of our knowledge, there is only one largescale dataset #OTHEREFR including 3D hand-articulated object interaction. However, it still holds the following limitations.

(i) HOI4D dataset is only captured in the egocentric view and can not generalize to the third view.

(ii) The annotation of the HOI4D dataset is not accurate enough to provide contact information.

Therefore we build a large handarticulated object interaction dataset with no annotation effort and more accurate pose and contact information.

This simulation system allows us to get accurate pose annotation and contact information using only a mobile phone and a laptop ( Fig. 2 (a) ).

Instead of using professional equipments like Kinect or Real-Sense cameras, the utilization of a mobile phone makes the data collection more accessible and easier to scale up.

Since the teleoperation is in the simulation, the annotations can be automatically recorded, which will make it easy to scale up the size of the dataset. We train our models with this collected dataset.

While the use of the teleoperation system itself is not new, its application in collecting 3D interaction prior data for pose estimation is novel, compared to previous literature on collecting human demonstrations. Dataset collection.

We use the front camera of a mobile phone to stream the RGB-D video at 15 fps. The setup and collection interface are shown in Fig. 2 (a) .

Contactart: Learning 3D Interaction Priors For Category-Level Articulated Object And Hand Poses Estimation

Abstract: Figure 1. Overview. We collect a dataset named ContactArt, which is created by human interacting with the articulated objects in a simulator, using teleoperation. Two interaction priors are learned from ContactArt: (i) a contact prior predicted by a diffusion model to improve 3D hand pose estimation; (ii) an articulation prior with a discriminator to improve category-level articulated object pose estimation. We visualize the pose estimation results in real-world data, leveraging the learned priors.

Sapien: A Simulated Part-Based Interactive Environment

Abstract: Building home assistant robots has long been a pursuit for vision and robotics researchers. To achieve this task, a simulated environment with physically realistic simulation, sufficient articulated objects, and transferability to the real robot is indispensable. Existing environments achieve these requirements for robotics simulation with different levels of simplification and focus. We take one step further in constructing an environment that supports household tasks for training robot learning algorithm. Our work, SAPIEN, is a realistic and physics-rich simulated environment that hosts a large-scale set for articulated objects. Our SAPIEN enables various robotic vision and interaction tasks that require detailed part-level understanding.We evaluate state-of-the-art vision algorithms for part detection and motion attribute recognition as well as demonstrate robotic interaction tasks using heuristic approaches and reinforcement learning algorithms. We hope that our SAPIEN can open a lot of research directions yet to be explored, including learning cognition through interaction, part motion discovery, and construction of robotics-ready simulated game environment.


As M fin and M fin M fin have bifree algebras, we recover the fixpoint operator in the relational model #REFR and obtain that it is unique.

Consider the lifting comonad (−) on Cppo : this comonad freely adjoins a bottom element and its co-Kleisli category is isomorphic to Cppo.

Since the endofunctors (−) and (−) ○ (−) are Cppo-enriched, the category Cppo has a unique dinatural fixpoint operator uniform with respect to the free functor Cppo → Cppo and it is given by the standard formula

f * = ⋁ n∈ω f n ( ) for f ∶ A → A in Cppo.

We can also consider the finite multiset comonad M fin on the category Rel.

It leads to a quantitative model of linear logic where multisets allow to count multiplicities of the inputs for a term #OTHEREFR .

Lastly, the category Lin (also called the linear Scott category ScottL) can be equipped with the ∨-semi-lattice comonad yielding a qualitative model of linear logic where substitution allows for duplication and erasure.

This comonad also verifies the necessary enrichment conditions to obtain a unique uniform dinatural fixpoint operator.

Fixpoint Operators For 2-Categorical Structures

Abstract: Fixpoint operators are tools to reason on recursive programs and data types obtained by induction (e.g. lists, trees) or coinduction (e.g. streams). They were given a categorical treatment with the notion of categories with fixpoints. A theorem by Plotkin and Simpson characterizes existence and uniqueness of fixpoint operators for categories satisfying some conditions on bifree algebras and recovers the standard examples of the category Cppo (ω-complete pointed partial orders and continuous functions) in domain theory and the relational model in linear logic. We present a categorification of this result and develop the theory of 2-categorical fixpoint operators where the 2dimensional framework allows to model the execution steps for languages with (co)recursive principles. We recover the standard categorical constructions of initial algebras and final coalgebras for endofunctors as well as fixpoints of generalized species and polynomial functors.

An Infinitary Model Of Linear Logic

Abstract: In this paper, we construct an infinitary variant of the relational model of linear logic, where the exponential modality is interpreted as the set of finite or countable multisets. We explain how to interpret in this model the fixpoint operator Y as a Conway operator alternatively defined in an inductive or a coinductive way. We then extend the relational semantics with a notion of color or priority in the sense of parity games. This extension enables us to define a new fixpoint operator Y combining both inductive and coinductive policies. We conclude the paper by mentionning a connection between the resulting model of λ-calculus with recursion and higher-order model-checking.


Several approaches that abstract to finite Markov decision processes have been proposed: In #REFR abstractions for uncountable-state discrete-time stochastic processes are proposed for SHA, where all nondeterminism is resolved probabilistically.

The application of model checking methods for probabilistic HA was enabled by CEGAR-style abstractions #OTHEREFR .

Extending decidable subclasses of HA by discrete probability distributions on jumps ( #OTHEREFR , #OTHEREFR ) preserves decidability of reachability.

An extension of probabilistic timed automata is presented in #OTHEREFR , where continuously distributed resets are used and randomized schedulers resolve discrete nondeterminism for a discretized state-space.

Further approaches for (networks of) stochastic timed automata ( #OTHEREFR ) maintain the probabilistic approach of resolving nondeterminism.

Approaches for more general classes either rely on stochastic approximation (also for nondeterminism) #OTHEREFR or on a combination of discretization and randomness #OTHEREFR .

In #OTHEREFR an abstraction to interval Markov decision processes is proposed.

Both approaches feature a stochastic kernel, and can hence not be compared to our work.

The analysis proposed in #OTHEREFR resolves discrete nondeterminism prophetically and continuous nondeterminism via a safe overapproximation (c.f. #OTHEREFR ).

The same approach has been specified for stochastic timed automata in #OTHEREFR .

In #OTHEREFR , discrete nondeterminism is resolved (non-)prophetically for stochastic automata, where all continuous variables are random clocks.

Maximizing Reachability Probabilities In Rectangular Automata With Random Clocks

Abstract: This paper proposes an algorithm to maximize reachability probabilities for rectangular automata with random clocks via a history-dependent prophetic scheduler. This model class incorporates time-induced nondeterminism on discrete behavior and nondeterminism in the dynamic behavior. After computing reachable state sets via a forward flowpipe construction, we use backward refinement to compute maximum reachability probabilities. The feasibility of the presented approach is illustrated on a scalable model.

Faust$^2$: Formal Abstractions Of Uncountable-State Stochastic Processes

Abstract: FAUST $^{\mathsf 2}$ is a software tool that generates formal abstractions of possibly non-deterministic discrete-time Markov processes dtMP defined over uncountable continuous state spaces. A dtMP model is specified in MATLAB and abstracted as a finite-state Markov chain or a Markov decision process. The abstraction procedure runs in MATLAB and employs parallel computations and fast manipulations based on vector calculus, which allows scaling beyond state-of-the-art alternatives. The abstract model is formally put in relationship with the concrete dtMP via a user-defined maximum threshold on the approximation error introduced by the abstraction procedure. FAUST $^{\mathsf 2}$ allows exporting the abstract model to well-known probabilistic model checkers, such as PRISM or MRMC. Alternatively, it can handle internally the computation of PCTL properties e.g. safety or reach-avoid over the abstract model. FAUST $^{\mathsf 2}$ allows refining the outcomes of the verification procedures over the concrete dtMP in view of the quantified and tunable error, which depends on the dtMP dynamics and on the given formula. The toolbox is available at http://sourceforge.net/projects/faust2/


It is interesting to note that the polynomial order d 2 ε −2 that we obtained for L 2 -approximation is the same as the one obtained in #REFR for numerical integration.

f − P [−m,m] d f ∞ ≤ d i=1 S x 1 m • • • S x i−1 m (f − S x i m f ) ∞ ≤ d i=1 (c 1 log(m)) i−1 • c 2 log(m)m −α ≤ (c 3 log(m)) d m α .

This term is smaller than ε for m ≥ exp(c 4 d log

2 (d) log(ε −1 )), thus log N ∞ B d (ε, F 3 d ) ≤ c 5 d 2 log 2 (d) log(ε −1 )

. Now the bound is obtained from Proposition 8.

It is surely an interesting open problem to improve the complexity bounds in Corollaries 11, 12 and 13 and/or provide matching lower bounds.

Tractability Of Sampling Recovery On Unweighted Function Classes

Abstract: It is well-known that the problem of sampling recovery in the L 2-norm on unweighted Korobov spaces (Sobolev spaces with mixed smoothness) as well as classical smoothness classes such as Hölder classes suffers from the curse of dimensionality. We show that the problem is tractable for those classes if they are intersected with the Wiener algebra of functions with summable Fourier coefficients. In fact, this is a relatively simple implication of powerful results by Rauhut and Ward [Appl. Comput. Harmon. Anal. 40 (2016), pp. 321-351]. Tractability is achieved by the use of nonlinear algorithms, while linear algorithms cannot do the job.

Numerical Integration Of H\"Older Continuous, Absolutely Convergent Fourier-, Fourier Cosine-, And Walsh Series

Abstract: We introduce quasi-Monte Carlo rules for the numerical integration of functions f defined on [ 0 , 1 ] s , s ≥ 1 , which satisfy the following properties: the Fourier, Fourier cosine or Walsh coefficients of f are absolutely summable and f satisfies a Holder condition of order α , for some 0 α ≤ 1 . We show a convergence rate of the integration error of order max ( ( s − 1 ) N − 1 / 2 , s α / 2 N − α ) . The construction of the quadrature points is explicit and is based on Weil sums.

L 2 -approximation
numerical integration


B. Clustered Federated Learning Process For Heterogeneous Devices

Model poisoning attacks against FL are outside the scope of this paper, even if already considered in some academic works #REFR .

For the FL deployment, we consider a typical cross-device setting #OTHEREFR with a large number of devices.

However, due to availability guarantees required by many IoT devices, especially in industrial settings, we expect most devices to participate in the FL training process.

This allows the server to maintain persistent state for each client to perform the clustering step.

In this work, we assume that no IoT device is infected prior to the FL model training and that none of them behave in an adversarial manner.

Our clustered FL builds upon the generalized FL setting proposed by Reddi et al. #OTHEREFR .

This generalized FL setting, described in Algorithm 1, improves over standard FL optimization methods such as the popular Federated Averaging (FedAvg) #OTHEREFR by including adaptive optimization methods.

The proposed clustered FL is described in Algorithm 2.

First, the aggregation server initializes the model weights W 0 and selects the training hyperparameters.

Then, the server sends those values to all the participating clients.

Clustered Federated Learning Architecture For Network Anomaly Detection In Large Scale Heterogeneous Iot Networks

Abstract: There is a growing trend of cyberattacks against Internet of Things (IoT) devices; moreover, the sophistication and motivation of those attacks is increasing. The vast scale of IoT, diverse hardware and software, and being typically placed in uncontrolled environments make traditional IT security mechanisms such as signature-based intrusion detection and prevention systems challenging to integrate. They also struggle to cope with the rapidly evolving IoT threat landscape due to long delays between the analysis and publication of the detection rules. Machine learning methods have shown faster response to emerging threats; however, model training architectures like cloud or edge computing face multiple drawbacks in IoT settings, including network overhead and data isolation arising from the large scale and heterogeneity that characterizes these networks. This work presents an architecture for training unsupervised models for network intrusion detection in large, distributed IoT and Industrial IoT (IIoT) deployments. We leverage Federated Learning (FL) to collaboratively train between peers and reduce isolation and network overhead problems. We build upon it to include an unsupervised device clustering algorithm fully integrated into the FL pipeline to address the heterogeneity issues that arise in FL settings. The architecture is implemented and evaluated using a testbed that includes various emulated IoT/IIoT devices and attackers interacting in a complex network topology comprising 100 emulated devices, 30 switches and 10 routers. The anomaly detection models are evaluated on real attacks performed by the testbed's threat actors, including the entire Mirai malware lifecycle, an additional botnet based on the Merlin command and control server and other red-teaming tools performing scanning activities and multiple attacks targeting the emulated devices.

How To Backdoor Federated Learning

Abstract: Federated learning enables thousands of participants to construct a deep learning model without sharing their private training data with each other. For example, multiple smartphones can jointly train a next-word predictor for keyboards without revealing what individual users type. We demonstrate that any participant in federated learning can introduce hidden backdoor functionality into the joint global model, e.g., to ensure that an image classifier assigns an attacker-chosen label to images with certain features, or that a word predictor completes certain sentences with an attacker-chosen word. We design and evaluate a new model-poisoning methodology based on model replacement. An attacker selected in a single round of federated learning can cause the global model to immediately reach 100% accuracy on the backdoor task. We evaluate the attack under different assumptions for the standard federated-learning tasks and show that it greatly outperforms data poisoning. Our generic constrain-and-scale technique also evades anomaly detection-based defenses by incorporating the evasion into the attacker's loss function during training.


Estimation Error Bounds: Local Properties

For instance, if |h(y) − x| ≤ 1 for any h ∈ H, y ∈ R ny and x ∈ R nx , then the condition is satisfied with C = 1 #REFR .

Based on these restricted class of functions, we present the following assumption and theorem (the proof is provided in the supplementary material). Assumption 1.

There exists a constant C ≥ 1 such that for every probability distribution D, and

estimator h ∈ H, E D nx j=1 (h j − h * j ) 2 ≤ CE D nx j=1 (h j ) − (h * j )

, where h j and h * j denote the jth coordinate of the estimators.

As pointed out in #OTHEREFR , this condition usually follows from a uniform convexity condition on the loss function .

Theorem 5 (Estimation error bound of ISTA, ADMM, and ReLU networks).

Consider the class of functions represented by depth-L ISTA networks H L I as detailed in Section 2.1, m i.i.d training samples, and a per-coordinate loss satisfying Assumption 1 with a constant C. Let

||W l − W l, * || ∞ ≤ α √ r for some α > 0. Moreover, B ≥ max{α √ r, 1}.

Then there exists T in the interval T ∈ 0, min √ mB0B L−1 2 L λη , m , where η = LB L−1 (B−1)−B L +1 (B−1) 2

, such that for any s > 0 with probability at least 1 − e −s ,

Generalization And Estimation Error Bounds For Model-Based Neural Networks

Abstract: Model-based neural networks provide unparalleled performance for various tasks, such as sparse coding and compressed sensing problems. Due to the strong connection with the sensing model, these networks are interpretable and inherit prior structure of the problem. In practice, model-based neural networks exhibit higher generalization capability compared to ReLU neural networks. However, this phenomenon was not addressed theoretically. Here, we leverage complexity measures including the global and local Rademacher complexities, in order to provide upper bounds on the generalization and estimation errors of model-based networks. We show that the generalization abilities of model-based networks for sparse recovery outperform those of regular ReLU networks, and derive practical design rules that allow to construct model-based networks with guaranteed high generalization. We demonstrate through a series of experiments that our theoretical insights shed light on a few behaviours experienced in practice, including the fact that ISTA and ADMM networks exhibit higher generalization abilities (especially for small number of training samples), compared to ReLU networks.

Local Rademacher Complexity-Based Learning Guarantees For Multi-Task Learning

Abstract: We show a Talagrand-type concentration inequality for Multi-Task Learning (MTL), using which we establish sharp excess risk bounds for MTL in terms of distribution- and data-dependent versions of the Local Rademacher Complexity (LRC). We also give a new bound on the LRC for norm regularized as well as strongly convex hypothesis classes, which applies not only to MTL but also to the standard i.i.d. setting. Combining both results, one can now easily derive fast-rate bounds on the excess risk for many prominent MTL methods, including---as we demonstrate---Schatten-norm, group-norm, and graph-regularized MTL. The derived bounds reflect a relationship akeen to a conservation law of asymptotic convergence rates. This very relationship allows for trading off slower rates w.r.t. the number of tasks for faster rates with respect to the number of available samples per task, when compared to the rates obtained via a traditional, global Rademacher analysis.


We adopt adversarial training #REFR to enforce the encoder to learn view-invariant discriminative action features by alternatively updating a discriminator and encoder towards an adversarial objective.

We explicitly apply this constraint as regularization 8 for the empirical distribution of the sharpened (sharpening hyper-parameter Sr) unlabeled prediction headỹ i = σ k (ŷ i /Sr) for relatively large batch size.

L var = E xi∈D u |var(ỹ i )− k−1 k 2 | 2 2 (8)

Joint Optimization: We jointly optimize the learning objective 9 of a weighted (λ) sum of the discussed losses and regularization by utilizing both D l and D u in a single stage with supervised contrastive architecture framework.

L = λ ce L ce +λ cl L cl +λ nl L nl +λ H L H +λ var L var (9) View-invariance constraint:

We impose optional additional constraints with the joint optimization to learn viewinvariant features given the availability of the viewpoint specifications.

In our setting, we place a multi-class discriminator f d to retrieve the source viewpoint information [minimizing the loss of equation 10], whereas the encoder network f θ aims to learn viewpoint information free action embedding [minimizing the loss of 11] to make the f d task no better than a random guess than maximize the discriminator loss of 10.

Where, e j denotes one-hot vector for j −th position.

Moreover, we enforce contrastive learning objectives to collapse the distributions from different viewpoints to learn a view-invariant representation that only depends on the underlying action class.

Nev-Ncd: Negative Learning, Entropy, And Variance Regularization Based Novel Action Categories Discovery

Abstract: Novel Categories Discovery (NCD) facilitates learning from a partially annotated label space and enables deep learning (DL) models to operate in an open-world setting by identifying and differentiating instances of novel classes based on the labeled data notions. One of the primary assumptions of NCD is that the novel label space is perfectly disjoint and can be equipartitioned, but it is rarely realized by most NCD approaches in practice. To better align with this assumption, we propose a novel singlestage joint optimization-based NCD method, Negative learning, Entropy, and Variance regularization NCD (NEV-NCD). We demonstrate the efficacy of NEV-NCD in previously unexplored NCD applications of video action recognition (VAR) with the public UCF101 dataset and a curated in-house partial action-space annotated multi-view video dataset. We perform a thorough ablation study by varying the composition of final joint loss and associated hyper-parameters. During our experiments with UCF101 and multi-view action dataset, NEV-NCD achieves ≈ 83% classification accuracy in test instances of labeled data. NEV-NCD achieves ≈ 70% clustering accuracy over unlabeled data outperforming both naive baselines (by ≈ 40%) and state-of-the-art pseudo-labeling-based approaches (by ≈ 3.5%) over both datasets. Further, we propose to incorporate optional view-invariant feature learning with the multiview dataset to identify novel categories from novel viewpoints. Our additional view-invariance constraint improves the discriminative accuracy for both known and unknown categories by ≈ 10% for novel viewpoints.

Abstract: Adversarial learning methods are a promising approach to training robust deep networks, and can generate complex samples across diverse domains. They can also improve recognition despite the presence of domain shift or dataset bias: recent adversarial approaches to unsupervised domain adaptation reduce the difference between the training and test domain distributions and thus improve generalization performance. However, while generative adversarial networks (GANs) show compelling visualizations, they are not optimal on discriminative tasks and can be limited to smaller shifts. On the other hand, discriminative approaches can handle larger domain shifts, but impose tied weights on the model and do not exploit a GAN-based loss. In this work, we first outline a novel generalized framework for adversarial adaptation, which subsumes recent state-of-the-art approaches as special cases, and use this generalized view to better relate prior approaches. We then propose a previously unexplored instance of our general framework which combines discriminative modeling, untied weight sharing, and a GAN loss, which we call Adversarial Discriminative Domain Adaptation (ADDA). We show that ADDA is more effective yet considerably simpler than competing domain-adversarial methods, and demonstrate the promise of our approach by exceeding state-of-the-art unsupervised adaptation results on standard domain adaptation tasks as well as a difficult cross-modality object classification task.

view-invariant discriminative action
adversarial training


Since the seminal work #REFR that introduced adversarial training and encoder-decoder structure into this task, various techniques have been proposed.

Traditional single-view methods #OTHEREFR such as patch-based or diffusion-based work often achieve realistic results but lack high-level image understanding.

These include multi-stage generation #OTHEREFR , gated convolution #OTHEREFR , recurrent methods #OTHEREFR , etc.

More recent work begins to focus on more challenging settings such as pluralistic inpainting #OTHEREFR and large hole filling #OTHEREFR .

Specifically, LaMa #OTHEREFR uses fast Fourier Convolutions and extremely large masks for training.

It achieves high-fidelity results with relatively low training and inference cost.

Single image 3D photography aims to synthesize novel views using image-based rendering techniques #OTHEREFR .

Clutter Detection And Removal In 3D Scenes With View-Consistent Inpainting

Abstract: Removing clutter from scenes is essential in many applications, ranging from privacy-concerned content filtering to data augmentation. In this work, we present an automatic system that removes clutter from 3D scenes and inpaints with coherent geometry and texture. We propose techniques for its two key components: 3D segmentation based on shared properties and 3D inpainting, both of which are important problems. The definition of 3D scene clutter (frequently-moving objects) is not well captured by commonly-studied object categories in computer vision. To tackle the lack of well-defined clutter annotations, we group noisy fine-grained labels, leverage virtual rendering, and impose an instance-level area-sensitive loss. Once clutter is removed, we inpaint geometry and texture in the resulting holes by merging inpainted RGB-D images. This requires novel voting and pruning strategies that guarantee multi-view consistency across individually inpainted images for mesh reconstruction. Experiments on ScanNet and Matterport3D dataset show that our method outperforms baselines for clutter segmentation and 3D inpainting, both visually and quantitatively. Project Page: https://weify627.github.io/clutter/.

Context Encoders: Feature Learning By Inpainting

Abstract: We present an unsupervised visual feature learning algorithm driven by context-based pixel prediction. By analogy with auto-encoders, we propose Context Encoders -- a convolutional neural network trained to generate the contents of an arbitrary image region conditioned on its surroundings. In order to succeed at this task, context encoders need to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing part(s). When training context encoders, we have experimented with both a standard pixel-wise reconstruction loss, as well as a reconstruction plus an adversarial loss. The latter produces much sharper results because it can better handle multiple modes in the output. We found that a context encoder learns a representation that captures not just appearance but also the semantics of visual structures. We quantitatively demonstrate the effectiveness of our learned features for CNN pre-training on classification, detection, and segmentation tasks. Furthermore, context encoders can be used for semantic inpainting tasks, either stand-alone or as initialization for non-parametric methods.


Handle Extrem Memory Constraints With Partial Tranining

This mechanism would not incur significant performance degradation because the input-side layers learn similar representations on different datasets #REFR , and clients with sufficient memory will provide model parameters of these input-side layers in the FL aggregation phase. Figure 6 emeprically validates such a strategy.

According to the memory consumption analysis in Section 3.2, the memory bottleneck of training a neural network is related to the block with the largest activations, which Figure 6 : Correspondences between layers of different local neural networks trained from private datasets in FL under non-IID distribution.

We observe that early layers, but not later layers, learn similar representations. some devices may still not afford.

These "large" blocks are usually the layers close to the input side.

To tackle this issue, we borrow the idea of partial training in FEDEPTH, where we skip the first few blocks that are too large to fit even after the finnest blocks decomposition.

We train a customized 14-layer ResNet (13 convolution layers and one classifier) on MNIST with 20 clients under non-IID distribution, respectively and measure the similarity of neural network representations, using both canonical correlation analysis (CCA) and centered kernel alignment (CKA) #OTHEREFR .

Memory-Adaptive Depth-Wise Heterogenous Federated Learning

Abstract: Federated learning is a promising paradigm that allows multiple clients to collaboratively train a model without sharing the local data. However, the presence of heterogeneous devices in federated learning, such as mobile phones and IoT devices with varying memory capabilities, would limit the scale and hence the performance of the model could be trained. The mainstream approaches to address memory limitations focus on width-slimming techniques, where different clients train subnetworks with reduced widths locally and then the server aggregates the subnetworks. The global model produced from these methods suffers from performance degradation due to the negative impact of the actions taken to handle the varying subnetwork widths in the aggregation phase. In this paper, we introduce a memory-adaptive depth-wise learning solution in FL called FEDEPTH, which adaptively decomposes the full model into blocks according to the memory budgets of each client and trains blocks sequentially to obtain a full inference model. Our method outperforms state-of-the-art approaches, achieving 5% and more than 10% improvements in top-1 accuracy on CIFAR-10 and CIFAR-100, respectively. We also demonstrate the effectiveness of depth-wise fine-tuning on ViT.

Similarity Of Neural Network Representations Revisited

Abstract: Recent work has sought to understand the behavior of neural networks by comparing representations between layers and between different trained models. We examine methods for comparing neural network representations based on canonical correlation analysis (CCA). We show that CCA belongs to a family of statistics for measuring multivariate similarity, but that neither CCA nor any other statistic that is invariant to invertible linear transformation can measure meaningful similarities between representations of higher dimension than the number of data points. We introduce a similarity index that measures the relationship between representational similarity matrices and does not suffer from this limitation. This similarity index is equivalent to centered kernel alignment (CKA) and is also closely connected to CCA. Unlike CCA, CKA can reliably identify correspondences between representations in networks trained from different initializations.

input-side layers
similar representations


Second, we overlooked YT channels shared on Twitter to safeguard our analysis from Twitter users who just advertise their own (or others) YouTube channel #REFR .

There is a number of limitations in our study.

First, neither Twitter nor YouTube provides any supplementary information about account suspension and video moderation, as well as we do not know when these interventions occurred.

As a result, there is no guarantee that YT videos were still online when re-shared through retweets on Twitter, but we can confidently assume they were not moderated yet when shared in an original tweet.

In addition, we acknowledge our analyses, like several previous works #OTHEREFR , might be biased towards moderated YT content which includes not only videos violating YouTube policies but also those removed by their publishers for any reason.

However, this choice might prevent us to consider another potential source of harmful YT content on Twitter.

Third, the partition strategy to define the two groups of mobilizers is quite conservative as we considered Twitter users who never share moderated videos (NMYT mobilizers) and those who mostly share moderated videos (MYT mobilizers).

The investigation of the users sharing a limited number of moderated videos is left for future research.

The Interconnected Nature Of Online Harm And Moderation: Investigating The Cross-Platform Spread Of Harmful Content Between Youtube And Twitter

Abstract: The proliferation of harmful content shared online poses a threat to online information integrity and the integrity of discussion across platforms. Despite various moderation interventions adopted by social media platforms, researchers and policymakers are calling for holistic solutions. This study explores how a target platform could leverage content that has been deemed harmful on a source platform by investigating the behavior and characteristics of Twitter users responsible for sharing moderated YouTube videos. Using a large-scale dataset of 600M tweets related to the 2020 U.S. election, we find that moderated Youtube videos are extensively shared on Twitter and that users who share these videos also endorse extreme and conspiratorial ideologies. A fraction of these users are eventually suspended by Twitter, but they do not appear to be involved in state-backed information operations. The findings of this study highlight the complex and interconnected nature of harmful cross-platform information diffusion, raising the need for cross-platform moderation strategies. CCS Concepts: • Information systems → Social networks; • Human-centered computing → Empirical studies in collaborative and social computing.

Who Watches (And Shares) What On Youtube? And When? Using Twitter To Understand Youtube Viewership

Abstract: By combining multiple social media datasets, it is possible to gain insight into each dataset that goes beyond what could be obtained with either individually. In this paper we combine user-centric data from Twitter with video-centric data from YouTube to build a rich picture of who watches and shares what on YouTube. We study 87K Twitter users, 5.6 million YouTube videos and 15 million video sharing events from user-, video- and sharing-event-centric perspectives. We show that features of Twitter users correlate with YouTube features and sharing-related features. For example, urban users are quicker to share than rural users. We find a superlinear relationship between initial Twitter shares and the final amounts of views. We discover that Twitter activity metrics play more role in video popularity than mere amount of followers. We also reveal the existence of correlated behavior concerning the time between video creation and sharing within certain timescales, showing the time onset for a coherent response, and the time limit after which collective responses are extremely unlikely. Response times depend on the category of the video, suggesting Twitter video sharing is highly dependent on the video content. To the best of our knowledge, this is the first large-scale study combining YouTube and Twitter data, and it reveals novel, detailed insights into who watches (and shares) what on YouTube, and when.


In comparison with #REFR , this theorem shows the separate effects of the sensor noise levels on the accuracy of the trained NN, allowing noise-aware optimization of the drone trajectory.

Iterative application of (1) over a series of T aggregation rounds, gives a bound on the expected convergence speed and final performance of the trained NN.

Assuming {Φt} T t=1 < 1, the speed of convergence is determined only by {Φt} T t=1 values, while the expected final performance of the trained NN is determined by the interplay of the collective effects from Jt, Kt, and Φt terms over T aggregation rounds.

In an ideal scenario without packet errors and sensor noises, i.e.

assuming ei,t = 0 and σ 2 i = 0, the expected training loss would asymptotically converge to the optimum loss value, i.e.

limT →∞ E(F (ωT +1)) = F (ω * ), while the packet errors and sensor noises have a destructive effect on the convergence speed and final performance of the trained NN.

The packet errors increase Φt values, thereby slowing down the convergence.

The sensor noises seem to have no effect on the convergence speed, but degrade the final performance of the trained NN by introducing the additional error terms Kt values.

To speed up convergence and improve the final performance of the NN, the drone should position itself closer to devices with larger local datasets and less noisy data.

In the following, we study the optimum placement and trajectory of the drone, based on the above insights.

Collaborative Learning With A Drone Orchestrator

Abstract: In this paper, the problem of drone-assisted collaborative learning is considered. In this scenario, swarm of intelligent wireless devices train a shared neural network (NN) model with the help of a drone. Using its sensors, each device records samples from its environment to gather a local dataset for training. The training data is severely heterogeneous as various devices have different amount of data and sensor noise level. The intelligent devices iteratively train the NN on their local datasets and exchange the model parameters with the drone for aggregation. For this system, the convergence rate of collaborative learning is derived while considering data heterogeneity, sensor noise levels, and communication errors, then, the drone trajectory that maximizes the final accuracy of the trained NN is obtained. The proposed trajectory optimization approach is aware of both the devices data characteristics (i.e., local dataset size and noise level) and their wireless channel conditions, and significantly improves the convergence rate and final accuracy in comparison with baselines that only consider data characteristics or channel conditions. Compared to state-of-the-art baselines, the proposed approach achieves an average 3.85% and 3.54% improvement in the final accuracy of the trained NN on benchmark datasets for image recognition and semantic segmentation tasks, respectively. Moreover, the proposed framework achieves a significant speedup in training, leading to an average 24% and 87% saving in the drone's hovering time, communication overhead, and battery usage, respectively for these tasks.

A Joint Learning And Communications Framework For Federated Learning Over Wireless Networks

Abstract: In this paper, the problem of training federated learning (FL) algorithms over a realistic wireless network is studied. In particular, in the considered model, wireless users execute an FL algorithm while training their local FL models using their own data and transmitting the trained local FL models to a base station (BS) that will generate a global FL model and send it back to the users. Since all training parameters are transmitted over wireless links, the quality of the training will be affected by wireless factors such as packet errors and the availability of wireless resources. Meanwhile, due to the limited wireless bandwidth, the BS must select an appropriate subset of users to execute the FL algorithm so as to build a global FL model accurately. This joint learning, wireless resource allocation, and user selection problem is formulated as an optimization problem whose goal is to minimize an FL loss function that captures the performance of the FL algorithm. To address this problem, a closed-form expression for the expected convergence rate of the FL algorithm is first derived to quantify the impact of wireless factors on FL. Then, based on the expected convergence rate of the FL algorithm, the optimal transmit power for each user is derived, under a given user selection and uplink resource block (RB) allocation scheme. Finally, the user selection and uplink RB allocation is optimized so as to minimize the FL loss function. Simulation results show that the proposed joint federated learning and communication framework can reduce the FL loss function value by up to 10% and 16%, respectively, compared to: 1) An optimal user selection algorithm with random resource allocation and 2) a standard FL algorithm with random user selection and resource allocation.


C3AE #REFR is a lightweight network with a multi-scale image input, aiming to show that the designed DAA is also a lightweight operation.

The DAA operation we designed can be transplanted to different feature extraction networks for age estimation. Our methods are expressed as Resnet18+DAA and C3AE+DAA.

Following the design in C3AE #OTHEREFR , we also give the "plain" configuration, applying single-scale input.

As is shown in Table  1 , the upper part of the table is the method with large parameters, while the lower part is the lightweight network.

Compared with Ranking-CNN, AP, MV, and PML, our proposed method achieved the best MAEs value of 2.06 with the pre-trained operation.

At the same time, the parameters of our approach are significantly reduced compared with other methods.

Daa: A Delta Age Adain Operation For Age Estimation Via Binary Code Transformer

Abstract: Naked eye recognition of age is usually based on comparison with the age of others. However, this idea is ignored by computer tasks because it is difficult to obtain representative contrast images of each age. Inspired by the transfer learning, we designed the Delta Age AdaIN (DAA) operation to obtain the feature difference with each age, which obtains the style map of each age through the learned values representing the mean and standard deviation. We let the input of transfer learning as the binary code of age natural number to obtain continuous age feature information. The learned two groups of values in Binary code mapping are corresponding to the mean and standard deviation of the comparison ages. In summary, our method consists of four parts: FaceEncoder, DAA operation, Binary code mapping, and AgeDecoder modules. After getting the delta age via AgeDecoder, we take the average value of all comparison ages and delta ages as the predicted age. Compared with state-of-the-art methods, our method achieves better performance with fewer parameters on multiple facial age datasets.

C3Ae: Exploring The Limits Of Compact Model For Age Estimation

Abstract: Age estimation is a classic learning problem in computer vision. Many larger and deeper CNNs have been proposed with promising performance, such as AlexNet, VggNet, GoogLeNet and ResNet. However, these models are not practical for the embedded/mobile devices. Recently, MobileNets and ShuffleNets have been proposed to reduce the number of parameters, yielding lightweight models. However, their representation has been weakened because of the adoption of depth-wise separable convolution. In this work, we investigate the limits of compact model for small-scale image and propose an extremely Compact yet efficient Cascade Context-based Age Estimation model(C3AE). This model possesses only 1/9 and 1/2000 parameters compared with MobileNets/ShuffleNets and VggNet, while achieves competitive performance. In particular, we re-define age estimation problem by two-points representation, which is implemented by a cascade model. Moreover, to fully utilize the facial context information, multi-branch CNN network is proposed to aggregate multi-scale context. Experiments are carried out on three age estimation datasets. The state-of-the-art performance on compact model has been achieved with a relatively large margin.