As shown in Table 1 , our model achieves 29.6 mAP on the AudioSet evaluation set, which outperforms the CNN10 model (when also trained on balanced set) as shown in #REFR .

The teacher model ft uses the CNN10 architecture designed in #OTHEREFR .

In order to achieve good performance, we train the teacher model using data augmentation techniques (e.g., mixup #OTHEREFR and specaugment #OTHEREFR ) up to 400K iterations using batch size of 64 on AudioSet balanced train set.

We compare OFA training with two baseline approaches: Baseline: Training each model from scratch using BCE loss and specaugment, which is a standard way to train an AEC model. Fig. 3 . Comparison of different search spaces for the OFA supernet.

We selectively apply ED on certain blocks, indicated with "X".

Knowledge Distillation (KD): Training each model with knowledge distillation and specaugment, which is a stronger baseline.

For the architecture of individual models, based on the supernet search space, we set each model with a fixed width ratio and depth across all the dynamic blocks.

For example, "width ratio = 0.4, depth = 2" for all three dynamic CNN blocks.

Weight-Sharing Supernet For Searching Specialized Acoustic Event Classification Networks Across Device Constraints

Abstract: Acoustic Event Classification (AEC) has been widely used in devices such as smart speakers and mobile phones for home safety or accessibility support [1]. As AEC models run on more and more devices with diverse computation resource constraints, it became increasingly expensive to develop models that are tuned to achieve optimal accuracy/computation trade-off for each given computation resource constraint. In this paper, we introduce a Once-For-All (OFA) Neural Architecture Search (NAS) framework for AEC. Specifically, we first train a weight-sharing supernet that supports different model architectures, followed by automatically searching for a model given specific computational resource constraints. Our experimental results showed that by just training once, the resulting model from NAS significantly outperforms both models trained individually from scratch and knowledge distillation (25.4% and 7.3% relative improvement). We also found that the benefit of weightsharing supernet training of ultra-small models comes not only from searching but from optimization.

Panns: Large-Scale Pretrained Audio Neural Networks For Audio Pattern Recognition

Abstract: Audio pattern recognition is an important research topic in the machine learning area, and includes several tasks such as audio tagging, acoustic scene classification and sound event detection. Recently neural networks have been applied to solve audio pattern recognition problems. However, previous systems focus on small datasets, which limits the performance of audio pattern recognition systems. Recently in computer vision and natural language processing, systems pretrained on large datasets have generalized well to several tasks. However, there is limited research on pretraining neural networks on large datasets for audio pattern recognition. In this paper, we propose large-scale pretrained audio neural networks (PANNs) trained on AudioSet. We propose to use Wavegram, a feature learned from waveform, and the mel spectrogram as input. We investigate the performance and complexity of a variety of convolutional neural networks. Our proposed AudioSet tagging system achieves a state-of-the-art mean average precision (mAP) of 0.439, outperforming the best previous system of 0.392. We transferred a PANN to six audio pattern recognition tasks and achieve state-of-the-art performance in many tasks. Source code and pretrained models have been released.


The work on MSC is extended in #REFR , which proves that while MSC corresponds to synchronized automata, the µ-fragment of the modal µcalculus similarly captures asynchronous distributed automata.

As already mentioned, descriptive complexity of distributed computing has been largely initiated in #OTHEREFR , which characterizes a range of related complexity classes via modal logics.

It is shown, for example, that graded modal logic captures the class MB(1) containing problems solvable in constant time by algorithms whose recognition capacity is sufficient all the way up to distinguishing between multisets of incoming messages, but no further.

In the paper, the link to logic helps also in separating some of the studied classes.

The constant-time limitation is lifted in #OTHEREFR , which shows that finite distributed message passing automata (FMPAs) correspond to modal substitution calculus MSC, which is the logic studied also in the current paper.

Distributed computing with identifiers has been studied from the point of view of logic earlier in #OTHEREFR .

The paper #OTHEREFR approaches identifiers via a uniform logical characterization of a certain class of algorithms using IDs, while our work is based on the circuit-style paradigm with formulas and circuits being given based on model size.

Thus, the two approaches are not comparable in any uniquely obvious way.

Nevertheless, one simple difference between our work and #OTHEREFR is that we treat IDs bit by bit as concrete bit strings.

Thus, we can express, e.g., that the current ID has a bit 1 without implying that the current node cannot have the smallest ID in the system.

Descriptive Complexity For Distributed Computing With Circuits

Abstract: We consider distributed algorithms in the realistic scenario where distributed message passing is operated via circuits. We show that within this setting, modal substitution calculus MSC captures the expressive power of circuits. The translations between circuits and MSC-programs are linear in both directions. Furthermore, we show that the colouring algorithm based on Cole-Vishkin can be specified via logarithmic size programs.

Asynchronous Distributed Automata: A Characterization Of The Modal Mu-Fragment

Abstract: We establish the equivalence between a class of asynchronous distributed automata and a small fragment of least fixpoint logic, when restricted to finite directed graphs. More specifically, the logic we consider is (a variant of) the fragment of the modal $\mu$-calculus that allows least fixpoints but forbids greatest fixpoints. The corresponding automaton model uses a network of identical finite-state machines that communicate in an asynchronous manner and whose state diagram must be acyclic except for self-loops. Exploiting the connection with logic, we also prove that the expressive power of those machines is independent of whether or not messages can be lost.


For multi-view RGB frames, we directly filter them using HRNet #REFR to acquire 2D feature maps F i .

Specifically, we follow #OTHEREFR to unproject depth image into pointcloud, voxelize and filter using 3D convolutional as coarse feature volume.

For a query point X, we acquire its local 3D feature F D by trilinear interpolation based on point coordinates and then share it with both visibility and occupancy MLPs.

Intuitively, the 3D feature originates from voxelized point cloud, which can be viewed as noisy samples of the underlying surface (decision boundary of the occupancy field).

By applying 3D convolution, the network reasons the coarse geometric surface feature over a sufficiently large receptive field, consequently aiding visibility inference.

3D feature is necessary for accurate visibility prediction as ablated in Fig. 8 .

The local 2D features are then extracted in the pixel-aligned fashion as in #OTHEREFR , by projecting the point coordinate X onto each view as the image coordinate π(X), then bilinearly interpolating the corresponding feature maps F i (π(X)).

Compared to feature volume, 2D feature maps have much higher resolution and thus grant better details.

(4), we infer V φ|ω (n) by providing 3D feature F D and averaged 2D feature F avg :

Learning Visibility Field For Detailed 3D Human Reconstruction And Relighting

Abstract: Figure 1. Given sparse-view RGBD input (left), our method reconstructs detailed geometry (middle left), albedo (middle right), and visibility for realistic 3D human relighting under different illuminations (right).

Deep High-Resolution Representation Learning For Human Pose Estimation

Abstract: This is an official pytorch implementation of Deep High-Resolution Representation Learning for Human Pose Estimation. In this work, we are interested in the human pose estimation problem with a focus on learning reliable high-resolution representations. Most existing methods recover high-resolution representations from low-resolution representations produced by a high-to-low resolution network. Instead, our proposed network maintains high-resolution representations through the whole process. We start from a high-resolution subnetwork as the first stage, gradually add high-to-low resolution subnetworks one by one to form more stages, and connect the mutli-resolution subnetworks in parallel. We conduct repeated multi-scale fusions such that each of the high-to-low resolution representations receives information from other parallel representations over and over, leading to rich high-resolution representations. As a result, the predicted keypoint heatmap is potentially more accurate and spatially more precise. We empirically demonstrate the effectiveness of our network through the superior pose estimation results over two benchmark datasets: the COCO keypoint detection dataset and the MPII Human Pose dataset. The code and models have been publicly available at \url{this https URL}.


For the individual classifiers, we examined different PLMs such as BERT #REFR , RoBERTa , and DeBERTa.

Similar to the full pipeline, Module 2 can also be broken down into its individual constituents, which are (i) the pre-processing module, (ii) the individual classifiers, and (iii) the ensembling method to combine the decision of these classifiers.

Firstly, a pre-processing step is needed, as the data for the tasks was collected from noisy resources (Reddit, and Gab).

For this, we used the same common techniques for all models.

In particular, we converted all uppercase characters to lowercase, removed repetitive patterns like "heeeey" and additional spaces, eliminated special characters like emojis and hashtags (#), and deleted numbers.

each of these models was initially fine-tuned using the entire dataset.

Lastly, we employed an ensemble of the three best-performing classifiers from the previous step for the final submission, namely BERTweet-large, DeBERTa-v3-large, and RoBERTa-large.

Ensembling multiple models can potentially prevent egregious mistakes made by a single model #OTHEREFR .

We used two ensemble approaches: majority voting, a hard voting method where the prediction of each classifier is treated as a vote, and the class with the most votes is ultimately selected as the predicted class, and soft average ensemble, in which the output of each model is averaged as shown in Equation 1.

Nlp-Ltu At Semeval-2023 Task 10: The Impact Of Data Augmentation And Semi-Supervised Learning Techniques On Text Classification Performance On An Imbalanced Dataset

Abstract: In this paper, we propose a methodology for task 10 of SemEval23, focusing on detecting and classifying online sexism in social media posts. The task is tackling a serious issue, as detecting harmful content on social media platforms is crucial for mitigating the harm of these posts on users. Our solution for this task is based on an ensemble of fine-tuned transformer-based models (BERTweet, RoBERTa, and DeBERTa). To alleviate problems related to class imbalance, and to improve the generalization capability of our model, we also experiment with data augmentation and semi-supervised learning. In particular, for data augmentation, we use backtranslation, either on all classes, or on the underrepresented classes only. We analyze the impact of these strategies on the overall performance of the pipeline through extensive experiments. while for semi-supervised learning, we found that with a substantial amount of unlabelled, in-domain data available, semi-supervised learning can enhance the performance of certain models. Our proposed method (for which the source code is available on Github 12) attains an F 1-score of 0.8613 for sub-taskA, which ranked us 10th in the competition.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


In #REFR it is presented a tethered UAV-UGV team that navigates through unstructured or confined spaces.

However, the tether is assumed to be taut and the planner generates trajectories with sweeping patterns only for the UAV system.

A tether state estimation technique is presented in #OTHEREFR , which describes a relative localization system for a marsupial tethered UGV-UAV pair.

The estimation of the tether angles, as measured by the cable's provider SDK, and a catenary model #OTHEREFR are used to estimate the relative position of the UAV with respect to the UGV.

The same authors propose in #OTHEREFR a PRM planner in which reachable space is constrained by the tether in two ways: avoiding any contacts of the tether with obstacles, or allowing up to two different contact points to extend the effective range.

Nevertheless, again the planner assumes a tensed tether at any time, enabling collision checking based on ray-tracing.

The UAV navigates autonomously and performs as visual assistant for UGV, while UGV is teleoperated. This approach also assumes a taut tether.

Regarding path planning, #OTHEREFR presents a method for autonomous exploration of unknown cavities in three dimensions (3D) that focuses on minimizing the distance traveled and the length of the unwound tether.

The method uses a hierarchical framework composed of a global level of exploration path planning that solves a Traveling Salesman Problem (TSP) and a local level of path planning whose parameters make it possible to adjust the trade-off between the tether unwinding and the path cost.

This approach also assumes the tether to be always in a taut condition.

In #OTHEREFR , we proposed a trajectory planning method for an UAV tied to a fixed element.

Path And Trajectory Planning Of A Tethered Uav-Ugv Marsupial Robotic System*

Abstract: This letter addresses the problem of trajectory planning in a marsupial robotic system consisting of an unmanned aerial vehicle (UAV) linked to an unmanned ground vehicle (UGV) through a non-taut tether with controllable length. To the best of our knowledge, this is the first method that addresses the trajectory planning of a marsupial UGV-UAV with a non-taut tether. The objective is to determine a synchronized collision-free trajectory for the three marsupial system agents: UAV, UGV, and tether. First, we present a path planning solution based on optimal Rapidly-exploring Random Trees (RRT*) with novel sampling and steering techniques to speed-up the computation. This algorithm is able to obtain collision-free paths for the UAV and the UGV, taking into account the 3D environment and the tether. Then, the letter presents a trajectory planner based on non-linear least squares. The optimizer takes into account aspects not considered in the path planning, like temporal constraints of the motion imposed by limits on the velocities and accelerations of the robots, or raising the tether's clearance. Simulated and field test results demonstrate that the approach generates obstacle-free, smooth, and feasible trajectories for the marsupial system.

Autonomous Visual Assistance For Robot Operations Using A Tethered Uav

Abstract: This paper develops an autonomous tethered aerial visual assistant for robot operations in unstructured or confined environments. Robotic tele-operation in remote environments is difficult due to lack of sufficient situational awareness, mostly caused by the stationary and limited field-of-view and lack of depth perception from the robot's onboard camera. The emerging state of the practice is to use two robots, a primary and a secondary that acts as a visual assistant to overcome the perceptual limitations of the onboard sensors by providing an external viewpoint. However, problems exist when using a tele-operated visual assistant: extra manpower, manually chosen suboptimal viewpoint, and extra teamwork demand between primary and secondary operators. In this work, we use an autonomous tethered aerial visual assistant to replace the secondary robot and operator, reducing human robot ratio from 2:2 to 1:2. This visual assistant is able to autonomously navigate through unstructured or confined spaces in a risk-aware manner, while continuously maintaining good viewpoint quality to increase the primary operator's situational awareness. With the proposed co-robots team, tele-operation missions in nuclear operations, bomb squad, disaster robots, and other domains with novel tasks or highly occluded environments could benefit from reduced manpower and teamwork demand, along with improved visual assistance quality based on trustworthy risk-aware motion in cluttered environments.


Influence Of Backbone Architecture On Fsl

One can conclude that the graph-based network DGCNN #REFR achieves higher classification accuracy than other networks on these two datasets.

• Pointwise-based: PointNet #OTHEREFR and PointNet++ #OTHEREFR .

• Convolution-based: PointCNN [15] , RSCNN [18] and DensePoint [17] .

For fair comparisons, we remove the last fully connected layers and train the networks from scratch with the same training strategy as suggested by these methods.

After that, we feed the embedding features generated by these networks into ProtoNet #OTHEREFR for few-shot classification and the comparison results are shown in Table 2 .

The reason may be that graph-based methods dynamically update the point-wise connection graph in feature space and extracts edge features with EdgeConv layers hence can learn more discriminative features.

Therefore, we will use ProtoNet #OTHEREFR with DGCNN #OTHEREFR as the strong baseline for benchmarking with 3D FSL.

What Makes For Effective Few-Shot Point Cloud Classification?

Abstract: Due to the emergence of powerful computing resources and large-scale annotated datasets, deep learning has seen wide applications in our daily life. However, most current methods require extensive data collection and retraining when dealing with novel classes never seen before. On the other hand, we humans can quickly recognize new classes by looking at a few samples, which motivates the recent popularity of few-shot learning (FSL) in machine learning communities. Most current FSL approaches work on 2D image domain, however, its implication in 3D perception is relatively under-explored. Not only needs to recognize the unseen examples as in 2D domain, 3D few-shot learning is more challenging with unordered structures, high intraclass variances, and subtle inter-class differences. Moreover, different architectures and learning algorithms make it difficult to study the effectiveness of existing 2D methods when migrating to the 3D domain. In this work, for the first time, we perform systematic and extensive studies of recent 2D FSL and 3D backbone networks for benchmarking few-shot point cloud classification, and we suggest a strong baseline and learning architectures for 3D FSL. Then, we propose a novel plug-and-play component called Cross-Instance Adaptation (CIA) module, to address the high intra-class variances and subtle inter-class differences issues, which can be easily inserted into current baselines with significant performance improvement. Extensive experiments on two newly introduced benchmark datasets, ModelNet40-FS and ShapeNet70-FS, demonstrate the superiority of our proposed network for 3D FSL.

Dynamic Graph Cnn For Learning On Point Clouds

Abstract: Point clouds provide a flexible geometric representation suitable for countless applications in computer graphics; they also comprise the raw output of most 3D data acquisition devices. While hand-designed features on point clouds have long been proposed in graphics and vision, however, the recent overwhelming success of convolutional neural networks (CNNs) for image analysis suggests the value of adapting insight from CNN to the point cloud world. Point clouds inherently lack topological information, so designing a model to recover topology can enrich the representation power of point clouds. To this end, we propose a new neural network module dubbed EdgeConv suitable for CNN-based high-level tasks on point clouds, including classification and segmentation. EdgeConv acts on graphs dynamically computed in each layer of the network. It is differentiable and can be plugged into existing architectures. Compared to existing modules operating in extrinsic space or treating each point independently, EdgeConv has several appealing properties: It incorporates local neighborhood information; it can be stacked applied to learn global shape properties; and in multi-layer systems affinity in feature space captures semantic characteristics over potentially long distances in the original embedding. We show the performance of our model on standard benchmarks, including ModelNet40, ShapeNetPart, and S3DIS.


Rather than re-implementing ESGD from scratch, we replicate the same image recognition experiment used by #REFR and compare PAPA to ESGD in this setting.

We also see that PAPA is additionally improved by soups (an advantage not shared by training a single model for p times more epochs).

This demonstrates that PAPA can bring more performance gain than training models for more epochs.

This suggests that PAPA could provide a better and more efficient way of training large models on large data by parallelizing the training length over multiple PAPA networks trained for less time.

#OTHEREFR ] is a method that leverages a population of networks through a complicated genetic algorithm. In this section, we compare PAPA to ESGD.

Since ESGD did not release their code as open source, direct comparison to PAPA is difficult.

From the paper, the experiments were done with CIFAR-10, using the same data processing as our main experiments using a different schedule (160 epochs, learning starts at 0.1 and is decayed by a scaling of 0.1 at 81 epochs and again at 122 epochs).

The authors do not mention what batch size they used, but having tried 32, 64, and 128, 128 gives us numbers closest to the one from their paper; thus, we use a batch size of 128.

We do not report soups since we can only compare the results with respect to the mean, min, and max since these were the metrics reported in #OTHEREFR .

For the PAPA data augmentations and regularizations, we use a random draw from Mixup α ∈ {0, 0.5, 1} and Label smoothing α ∈ {0, 0.05, 0.1}

The results are shown in the table below.

Population Parameter Averaging (Papa)

Abstract: Ensemble methods combine the predictions of multiple models to improve performance, but they require significantly higher computation costs at inference time. To avoid these costs, multiple neural networks can be combined into one by averaging their weights (model soups). However, this usually performs significantly worse than ensembling. Weight averaging is only beneficial when weights are similar enough (in weight or feature space) to average well but different enough to benefit from combining them. Based on this idea, we propose PopulAtion Parameter Averaging (PAPA): a method that combines the generality of ensembling with the efficiency of weight averaging. PAPA leverages a population of diverse models (trained on different data orders, augmentations, and regularizations) while occasionally (not too often, not too rarely) replacing the weights of the networks with the population average of the weights. PAPA reduces the performance gap between averaging and ensembling, increasing the average accuracy of a population of models by up to 1.1% on CIFAR-10, 2.4% on CIFAR-100, and 1.9% on ImageNet when compared to training independent (non-averaged) models.

Evolutionary Stochastic Gradient Descent For Optimization Of Deep Neural Networks

Abstract: We propose a population-based Evolutionary Stochastic Gradient Descent (ESGD) framework for optimizing deep neural networks. ESGD combines SGD and gradient-free evolutionary algorithms as complementary algorithms in one framework in which the optimization alternates between the SGD step and evolution step to improve the average fitness of the population. With a back-off strategy in the SGD step and an elitist strategy in the evolution step, it guarantees that the best fitness in the population will never degrade. In addition, various SGD-based optimizers with distinct hyper-parameters are considered in the SGD step as competing species in a coevolution setting such that their complementarity is also taken into account. The effectiveness of ESGD is demonstrated across multiple applications including speech recognition, image recognition and language modeling using networks with a variety of deep architectures.


In #REFR Kempa and Prezza also show that computing the smallest k-attractor is NP -complete for any k ≥ 3, by giving a reduction from k-set cover, and extend this proof to non-constant k, especially for k = n.

For a string of length n, an attractor is a set of positions Γ ⊆ [1, n] covering all distinct substrings, that is, every distinct substring has an occurrence crossing at least one of the selected positions.

If only distinct substrings up to a certain length k need to be covered, we speak of the k-attractor.

For the string T = abbcabccac, the set of markings Γ = {2, 7, 9} is a 2-attractor for T .

It is not a valid 3-attractor and therefore not an attractor because the substrings bca and cab are not covered.

Adding either 4 or 5 to Γ results in an attractor, and there is no smaller attractor for T .

Thus, the problem is solvable in polynomial time for k = 1, and the complexity for k = 2 was left as an open problem.

Variants of the problem have been introduced, such as the sharp k-attractor in #OTHEREFR , which only considers distinct substrings of length exactly k, and the circular attractor in #OTHEREFR , which also requires to cover circular substrings.

Interestingly, the sharp k-attractor problem can also be reduced from and to k-set cover and is therefore NP -complete for k ≥ 3.

However, the sharp 2-attractor problem reduces to 2-set cover, which is equivalent to edge cover and therefore solvable in polynomial time.

Notably, the sharp variant does not exhibit the same gap as the k-attractor problem.

The 2-Attractor Problem Is Np-Complete

Abstract: A k-attractor is a combinatorial object unifying dictionary-based compression. It allows to compare the repetitiveness measures of different dictionary compressors such as Lempel-Ziv 77, the Burrows-Wheeler transform, straight line programs and macro schemes. For a string T ∈ Σ n , the k-attractor is defined as a set of positions Γ ⊆ [1, n], such that every distinct substring of length at most k is covered by at least one of the selected positions. Thus, if a substring occurs multiple times in T, one position suffices to cover it. A 1-attractor is easily computed in linear time, while Kempa and Prezza [STOC 2018] have shown that for k ≥ 3, it is NP-complete to compute the smallest k-attractor by a reduction from k-set cover. The main result of this paper answers the open question for the complexity of the 2-attractor problem, showing that the problem remains NP-complete. Kempa and Prezza's proof for k ≥ 3 also reduces the 2-attractor problem to the 2-set cover problem, which is equivalent to edge cover, but that does not fully capture the complexity of the 2-attractor problem. For this reason, we extend edge cover by a color function on the edges, yielding the colorful edge cover problem. Any edge cover must then satisfy the additional constraint that each color is represented. This extension raises the complexity such that colorful edge cover becomes NP-complete while also more precisely modeling the 2-attractor problem. We obtain a reduction showing k-attractor to be NP-complete for any k ≥ 2.

At The Roots Of Dictionary Compression: String Attractors

Abstract: A well-known fact in the field of lossless text compression is that high-order entropy is a weak model when the input contains long repetitions. Motivated by this fact, decades of research have generated myriads of so-called dictionary compressors: algorithms able to reduce the text’s size by exploiting its repetitiveness. Lempel-Ziv 77 is one of the most successful and well-known tools of this kind, followed by straight-line programs, run-length Burrows-Wheeler transform, macro schemes, collage systems, and the compact directed acyclic word graph. In this paper, we show that these techniques are different solutions to the same, elegant, combinatorial problem: to find a small set of positions capturing all distinct text’s substrings. We call such a set a string attractor. We first show reductions between dictionary compressors and string attractors. This gives the approximation ratios of dictionary compressors with respect to the smallest string attractor and allows us to uncover new asymptotic relations between the output sizes of different dictionary compressors. We then show that the k-attractor problem — deciding whether a text has a size-t set of positions capturing all substrings of length at most k — is NP-complete for k≥ 3. This, in particular, includes the full string attractor problem. We provide several approximation techniques for the smallest k-attractor, show that the problem is APX-complete for constant k, and give strong inapproximability results. To conclude, we provide matching lower and upper bounds for the random access problem on string attractors. The upper bound is proved by showing a data structure supporting queries in optimal time. Our data structure is universal: by our reductions to string attractors, it supports random access on any dictionary-compression scheme. In particular, it matches the lower bound also on LZ77, straight-line programs, collage systems, and macro schemes, and therefore essentially closes (at once) the random access problem for all these compressors.


It is adapted from ResNet #REFR by replacing the pooling and stride-convolution layers with a lowpass filter based on Discrete Wavelet Transform (DWT).

Authors in #OTHEREFR developed efficient low-resolution face recognition models with very low computational cost by distilling the most informative facial features from teacher to student stream.

More recently, #OTHEREFR In this paper, a cross-resolution knowledge distillation framework is first employed, where the targeted student network is trained with multi-scale LR data and optimized with both face recognition and distillation losses.

Despite the guidance of the prior knowledge extracted from HR face images, the huge resolution disparity between HR and LR images makes it difficult for the student network to capture informative features.

Frequency analysis in #OTHEREFR has shown that the high-frequency information in a facial image, such as edge and noise, is eliminated during the resolution reduction, while the low-frequency subbands still preserve the most discriminative features.

This work proposes an identity-preserving network, WaveResNet, to capture the discriminative information stored in low-frequency components of the LR images.

The high-frequency subbands of the intermediate feature maps are filtered out to remove ambiguous and noisy information and enforce the network to focus on the more discriminative lowfrequency information.

In addition, a wavelet similarity loss is designed as an auxiliary distillation loss in order to further enhance attention in low-frequency subbands.

Moreover, a degradation model is designed to simulate real-world LR training data and develop a more robust recognition system.

The proposed method has been evaluated on four datasets in a variety of resolutions and it outperforms the baseline model and some other related work.

Identity-Preserving Knowledge Distillation For Low-Resolution Face Recognition

Abstract: Low-resolution face recognition (LRFR) has become a challenging problem for modern deep face recognition systems. Existing methods mainly leverage prior information from high-resolution (HR) images by either reconstructing facial details with super-resolution techniques or learning a unified feature space. To address this issue, this paper proposes a novel approach which enforces the network to focus on the discriminative information stored in the lowfrequency components of a low-resolution (LR) image. A cross-resolution knowledge distillation paradigm is first employed as the learning framework. An identity-preserving network, WaveResNet, and a wavelet similarity loss are then designed to capture low-frequency details and boost performance. Finally, an image degradation model is conceived to simulate more realistic LR training data. Consequently, extensive experimental results show that the proposed method consistently outperforms the baseline model and other stateof-the-art methods across a variety of image resolutions.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

Discrete Wavelet Transform
stride-convolution layers


Our long-term forecasting experiments require ∼ 10 18 floating point operations for training, model selection and hyperparameter tuning, a figure comparable to the scale of other recent large-scale machine learning benchmarks #REFR .

Traditional methods include standard linear regression, autoregressive moving averages (ARIMA), exponential smoothing, Kalman filtering, Fourier mode extrapolation, boosted random forest models #OTHEREFR , and newly-introduced linear models that account for trends and distribution shift #OTHEREFR .

Current state-of-the-art models for general time series forecasting are based on deep neural networks: the transformer model #OTHEREFR , long-short-termmemory networks (LSTM), block recurrent neural networks (RNN), temporal convolutional neural networks #OTHEREFR , and neural basis expansion/neural hierarchical interpolation (NBEATS/NHiTS) #OTHEREFR .

The latter methods generates forecasts hierarchically by aggregating separate forecasts at distinct timescales (NBEATS), and can explicitly coarse-grain the time series to further reduce computational costs (NHiTS).

We also consider hybrid physics-motivated methods such as neural ordinary differential equations #OTHEREFR , which approximate the continuous-time differential equation underlying time series, as well as echo-state networks and their generalization, nonlinear vector autoregressive models (nVAR), which use a trainable linear recurrent structure with fixed "reservoir" of random nonlinearities #OTHEREFR .

Our main forecasting results are summarized in Figure  2 .

For brevity, we report here only results for the top 14 forecasting models in terms of the sMAPE error, and defer the full tabular results and accuracy metrics to the supplementary material and our open-source repository.

We observe that large, domain-agnostic forecasting methods successfully forecast diverse chaotic systems, with the strongest methods succeeding consistently across diverse systems and forecasting horizons.

The strong relative performance of machine learning models is highlighted in Fig 2C, where NBEATS successfully forecasts the Mackey-Glass equation for ∼22 Lyapunov times without losing track of the global phase.

This and other strongly-performing methods, such as the transformer and LSTM, comprise large models originally designed for generic sequential datasets, and which do not make any assumptions regarding whether observed time series data arises from a dynamical system.

This observation suggests that more flexible, generic architectures may prove preferable for problems where some physical structure is present (e.g.

Large Statistical Learning Models Effectively Forecast Diverse Chaotic Systems

Abstract: Chaos and unpredictability are traditionally synonymous, yet recent advances in statistical forecasting suggest that large machine learning models can derive unexpected insight from extended observation of complex systems. Here, we study the forecasting of chaos at scale, by performing a large-scale comparison of 24 representative state-of-the-art multivariate forecasting methods on a crowdsourced database of 135 distinct low-dimensional chaotic systems. We find that large, domain-agnostic time series forecasting methods based on artificial neural networks consistently exhibit strong forecasting performance, in some cases producing accurate predictions lasting for dozens of Lyapunov times. Best-in-class results for forecasting chaos are achieved by recently-introduced hierarchical neural basis function models, though even generic transformers and recurrent neural networks perform strongly. However, physics-inspired hybrid methods like neural ordinary equations and reservoir computers contain inductive biases conferring greater data efficiency and lower training times in data-limited settings. We observe consistent correlation across all methods despite their widely-varying architectures, as well as universal structure in how predictions decay over long time intervals. Our results suggest that a key advantage of modern forecasting methods stems not from their architectural details, but rather from their capacity to learn the large-scale structure of chaotic attractors.

An Analysis Of Deep Neural Network Models For Practical Applications

Abstract: Since the emergence of Deep Neural Networks (DNNs) as a prominent technique in the field of computer vision, the ImageNet classification challenge has played a major role in advancing the state-of-the-art. While accuracy figures have steadily increased, the resource utilisation of winning models has not been properly taken into account. In this work, we present a comprehensive analysis of important metrics in practical applications: accuracy, memory footprint, parameters, operations count, inference time and power consumption. Key findings are: (1) power consumption is independent of batch size and architecture; (2) accuracy and inference time are in a hyperbolic relationship; (3) energy constraint are an upper bound on the maximum achievable accuracy and model complexity; (4) the number of operations is a reliable estimate of the inference time. We believe our analysis provides a compelling set of information that helps design and engineer efficient DNNs.

long-term forecasting experiments
learning benchmarks


The message-passing is split into two steps, one for each set of nodes, similar to the approach of #REFR .

This allows for more complex features to be extracted at each model layer.

Algorithm 1: Application of an M P operator with K convolutions to update the features of a node v through message-passing.

Target node v, node features h v , neighbors' features H N (v) , edge weights w u,v , ∀u ∈ N (v). Result: Updated node features h * v .

h (0) ← h v for k ← 1 to K do h (k) ← U (k) h (k−1) , Aggregation (k) M (k) (h u , w u,v ) : u ∈ N (v) end h * v ← h (K)

At each layer, the messages are propagated first from the variable nodes to the constraint nodes and then from the constraint nodes to the variable nodes, exploiting the bipartite nature of the graph. Algorithm 2 describes this process with further detail.

Finally, the output of model is generated from the last hidden feature vectors of the variable nodes, generating an output in the same shape as the problem's variable vector.

The hidden features are fed to an MLP with two hidden layers and ReLU activations NN out , which maps each d-dimensional vector into a single output, i.e., Figure 2 shows an overview of the architecture. Data:

Bipartite graph (V var , V con , E), node features H (0) ∈ R (n+m)×d , edge weights w e ∈ R, ∀e ∈ E, number of message passing iterations L. Result: Node embeddings H (L) .

A Graph Neural Network Approach To Nanosatellite Task Scheduling: Insights Into Learning Mixed-Integer Models

Abstract: This study investigates how to schedule nanosatellite tasks more efficiently using Graph Neural Networks (GNN). In the Offline Nanosatellite Task Scheduling (ONTS) problem, the goal is to find the optimal schedule for tasks to be carried out in orbit while taking into account Quality-of-Service (QoS) considerations such as priority, minimum and maximum activation events, execution time-frames, periods, and execution windows, as well as constraints on the satellite's power resources and the complexity of energy harvesting and management. The ONTS problem has been approached using conventional mathematical formulations and precise methods, but their applicability to challenging cases of the problem is limited. This study examines the use of GNNs in this context, which has been effectively applied to many optimization problems, including traveling salesman problems, scheduling problems, and facility placement problems. Here, we fully represent MILP instances of the ONTS problem in bipartite graphs. We apply a feature aggregation and message-passing methodology allied to a ReLU activation function to learn using a classic deep learning model, obtaining an optimal set of parameters. Furthermore, we apply Explainable AI (XAI), another emerging field of research, to determine which features-nodes, constraints-had the most significant impact on learning performance, shedding light on the inner workings and decision process of such models. We also explored an early fixing approach, obtaining an accuracy above 80% both in predicting the feasibility of a solution and the probability of a decision variable value being in the optimal solution. Our results point to GNNs as a potentially effective method for scheduling nanosatellite tasks and shed light on the advantages of explainable machine learning models for challenging combinatorial optimization problems.

Exact Combinatorial Optimization With Graph Convolutional Neural Networks

Abstract: Combinatorial optimization problems are typically tackled by the branch-and-bound paradigm. We propose a new graph convolutional neural network model for learning branch-and-bound variable selection policies, which leverages the natural variable-constraint bipartite graph representation of mixed-integer linear programs. We train our model via imitation learning from the strong branching expert rule, and demonstrate on a series of hard problems that our approach produces policies that improve upon state-of-the-art machine-learning methods for branching and generalize to instances significantly larger than seen during training. Moreover, we improve for the first time over expert-designed branching rules implemented in a state-of-the-art solver on large problems. Code for reproducing all the experiments can be found at this https URL.


Iv. Methodology A. Adapting The Safe Set Algorithm

Thus we decide to consider the current positions, estimate future positions of all approaching obstacles, and modify the direction of safe control by tuning parameters in the QP problem #REFR .

Vanilla SSA would output safe control that drives the system to the currently safest direction, which may not be an efficient direction in the long run, see fig. 1a .

Besides, in multi-obstacle environment, it's not safe to directly add constraints for every dangerous obstacle whose φ is positive.

In detail, since vanilla SSA only considers these dangerous obstacles, it may push the robot to the direction that is safe now but risky in the next step if there are unconsidered obstacles (φ values are negative) in that direction, see fig. 1b .

After adapting the projection direction, we expect to generate control signal that is safe to all approaching obstacles, even these φ values are negative, and efficient for the longer time horizon.

In detail, we define the approaching obstacles as those whose distances to the robot are smaller than a threshold value.

With the current position of the robot (x 0 , y 0 ) and the current positions of approaching obstacles (x i , y i ), i = 1, 2, ..., n, we first predict the next k steps positions of each obstacle (x j i , y j i ), j = 1, 2, ..., k using constant velocity model.

Then we solve the line l θ : − sin(θ)x + cos(θ)y = 0 that maximizes the distance to all approaching obstacles

Safe And Sample-Efficient Reinforcement Learning For Clustered Dynamic Environments

Abstract: This study proposes a safe and sample-efficient reinforcement learning (RL) framework to address two major challenges in developing applicable RL algorithms: satisfying safety constraints and efficiently learning with limited samples. To guarantee safety in real-world complex environments, we use the safe set algorithm (SSA) to monitor and modify the nominal controls, and evaluate SSA+RL in a clustered dynamic environment which is challenging to be solved by existing RL algorithms. However, the SSA+RL framework is usually not sample-efficient especially in reward-sparse environments, which has not been addressed in previous safe RL works. To improve the learning efficiency, we propose three techniques: (1) avoiding behaving overly conservative by adapting the SSA; (2) encouraging safe exploration using random network distillation with safety constraints; (3) improving policy convergence by treating SSA as expert demonstrations and directly learn from that. The experimental results show that our framework can achieve better safety performance compare to other safe RL methods during training and solve the task with substantially fewer episodes. Project website: https://hychennaza.github.io/projects/Safe RL/.

Constrained Policy Optimization

Abstract: For many applications of reinforcement learning it can be more convenient to specify both a reward function and constraints, rather than trying to design behavior through the reward function. For example, systems that physically interact with or around humans should satisfy safety constraints. Recent advances in policy search algorithms (Mnih et al., 2016, Schulman et al., 2015, Lillicrap et al., 2016, Levine et al., 2016) have enabled new capabilities in high-dimensional control, but do not consider the constrained setting. ::: We propose Constrained Policy Optimization (CPO), the first general-purpose policy search algorithm for constrained reinforcement learning with guarantees for near-constraint satisfaction at each iteration. Our method allows us to train neural network policies for high-dimensional control while making guarantees about policy behavior all throughout training. Our guarantees are based on a new theoretical result, which is of independent interest: we prove a bound relating the expected returns of two policies to an average divergence between them. We demonstrate the effectiveness of our approach on simulated robot locomotion tasks where the agent must satisfy constraints motivated by safety.


In #REFR , a study is performed evaluating thirty-three different OPE methods by measuring the distance between the estimated policy value and the true policy value.

Generally, it is difficult to perform policy rollouts in real-world settings, due to costs or safety concerns.

This is especially true for evaluating policies that are still being trained.

Performing evaluation before the training of the policy to determine the ideal value of hyper-parameters, would allow for changing the values of hyper-parameters and safe time and resources, instead of training a complete policy.

Hence, this is another reason restricting demonstration learning to simulators.

Off-policy methods aim at evaluating a policy from previous data, without requiring interaction with the environment.

The study concluded that evaluation with the state-action function outperforms the remaining methods.

To help accelerate the development of OPE methods, #OTHEREFR propose the Deep Off-Policy Evaluation (DOPE) benchmark.

The available options are using inaccurate OPE methods or training the model for a fixed number of steps and adjusting the values based on the early results.

The challenge of the selection of hyper-parameter values is common to other machine-learning paradigms.

However, demonstration learning needs to overcome this problem if it truly wishes to reduce the requirement of expert programming.

A Survey Of Demonstration Learning

Abstract: With the fast improvement of machine learning, reinforcement learning (RL) has been used to automate human tasks in different areas. However, training such agents is difficult and restricted to expert users. Moreover, it is mostly limited to simulation environments due to the high cost and safety concerns of interactions in the real world. Demonstration Learning is a paradigm in which an agent learns to perform a task by imitating the behavior of an expert shown in demonstrations. It is a relatively recent area in machine learning, but it is gaining significant traction due to having tremendous potential for learning complex behaviors from demonstrations. Learning from demonstration accelerates the learning process by improving sample efficiency, while also reducing the effort of the programmer. Due to learning without interacting with the environment, demonstration learning would allow the automation of a wide range of real world applications such as robotics and healthcare. This paper provides a survey of demonstration learning, where we formally introduce the demonstration problem along with its main challenges and provide a comprehensive overview of the process of learning from demonstrations from the creation of the demonstration data set, to learning methods from demonstrations, and optimization by combining demonstration learning with different machine learning methods. We also review the existing benchmarks and identify their strengths and limitations. Additionally, we discuss the advantages and disadvantages of the paradigm as well as its main applications. Lastly, we discuss our perspective on open problems and research directions for this rapidly growing field.

Empirical Study Of Off-Policy Policy Evaluation For Reinforcement Learning

Abstract: Off-policy policy evaluation (OPE) is the problem of estimating the online performance of a policy using only pre-collected historical data generated by another policy. Given the increasing interest in deploying learning-based methods for safety-critical applications, many recent OPE methods have recently been proposed. Due to disparate experimental conditions from recent literature, the relative performance of current OPE methods is not well understood. In this work, we present the first comprehensive empirical analysis of a broad suite of OPE methods. Based on thousands of experiments and detailed empirical analyses, we offer a summarized set of guidelines for effectively using OPE in practice, and suggest directions for future research.


PixBis #REFR employs a binary supervisory strategy at pixel-level to simplify the problem and obviate the need for a computationally intensive synthesis of depth maps.

ResNet #OTHEREFR is one of the most popular backbone architectures used in face PAD algorithm design #OTHEREFR .

We report the results of a model trained from scratch based on the ResNet-18 model architecture.

A cross-entropy loss function is used in the training phase and formulated as follows:

where y is the ground truth (1 for bona fide and 0 for attack in our case) and p is the predicted score.

Two dense blocks of DenseNet121 #OTHEREFR are utilized as the model backbone and a combination of two binary cross-entropy loss functions is used to train the model for both pixel-wise and binary output.

The combined loss equation for the training of all models is formed as:

where L pixel−wise CE refers to the loss based on the pixel-wise output and L binary CE refers to the loss based on the binary output.

Synthaspoof: Developing Face Presentation Attack Detection Based On Privacy-Friendly Synthetic Data

Abstract: Recently, significant progress has been made in face presentation attack detection (PAD), which aims to secure face recognition systems against presentation attacks, owing to the availability of several face PAD datasets. However, all available datasets are based on privacy and legallysensitive authentic biometric data with a limited number of subjects. To target these legal and technical challenges, this work presents the first synthetic-based face PAD dataset, named SynthASpoof, as a large-scale PAD development dataset. The bona fide samples in SynthASpoof are synthetically generated and the attack samples are collected by presenting such synthetic data to capture systems in a real attack scenario. The experimental results demonstrate the feasibility of using SynthASpoof for the development of face PAD. Moreover, we boost the performance of such a solution by incorporating the domain generalization tool MixStyle into the PAD solutions. Additionally, we showed the viability of using synthetic data as a supplement to enrich the diversity of limited authentic training data and consistently enhance PAD performances. The SynthASpoof dataset, containing 25,000 bona fide and 78,800 attack samples, the implementation, and the pre-trained weights are made publicly available 1 .

Deep Pixel-Wise Binary Supervision For Face Presentation Attack Detection

Abstract: Face recognition has evolved as a prominent biometric authentication modality. However, vulnerability to presentation attacks curtails its reliable deployment. Automatic detection of presentation attacks is essential for secure use of face recognition technology in unattended scenarios. In this work, we introduce a Convolutional Neural Network (CNN) based framework for presentation attack detection, with deep pixel-wise supervision. The framework uses only frame level information making it suitable for deployment in smart devices with minimal computational and time overhead. We demonstrate the effectiveness of the proposed approach in public datasets for both intra as well as cross-dataset experiments. The proposed approach achieves an HTER of 0% in Replay Mobile dataset and an ACER of 0.42% in Protocol-1 of OULU dataset outperforming state of the art methods.

computationally intensive synthesis
pixel-level


Specifically, we use a ResNet50 #REFR with an extra two-layer MLP as the encoder and train it on ImageNet.

Our experiments are performed using PyTorch #OTHEREFR , and are all conducted on 8 Nvidia V100 GPUs.

Following the classical pre-training and fine-tuning paradigm, we run the whole experiments in two stages:

We inherit most settings from MoCoV2 #OTHEREFR , while modify the pretext task and degradation process for the quality-aware purpose.

We use SGD optimizer with weight decay of 0.0001, momentum of 0.9 and a mini-batch size of 256.

The initial learning rate is 0.03 and multiplied by 0.1 at 120 and 160 epoch. The hyper-parameters, β and τ in Eq. 4 Fine-tuning.

Following the scheme from #OTHEREFR , we resize the smaller edge of images to 340 while maintaining aspect ratio, and randomly sample and horizontally flip patches with size 320x320 pixels.

We use AdamW optimizer with weight decay of 0.01 and mini-batch size of 64.

The Learning rate is initialized with 0.0002, and decayed by the cosine annealing strategy.

Quality-Aware Pre-Trained Models For Blind Image Quality Assessment

Abstract: Blind image quality assessment (BIQA) aims to automatically evaluate the perceived quality of a single image, whose performance has been improved by deep learningbased methods in recent years. However, the paucity of labeled data somewhat restrains deep learning-based BIQA methods from unleashing their full potential. In this paper, we propose to solve the problem by a pretext task customized for BIQA in a self-supervised learning manner, which enables learning representations from orders of magnitude more data. To constrain the learning process, we propose a quality-aware contrastive loss based on a simple assumption: the quality of patches from a distorted image should be similar, but vary from patches from the same image with different degradations and patches from different images. Further, we improve the existing degradation process and form a degradation space with the size of roughly 2 × 10 7. After pre-trained on ImageNet using our method, models are more sensitive to image quality and perform significantly better on downstream BIQA tasks. Experimental results show that our method obtains remarkable improvements on popular BIQA datasets.


If q ≡ 1 (mod 4), then the self-dual quasi-cyclic codes of index 2 over F and the LCD quasi-cyclic codes of index 2 over F of rate #REFR 2 are both asymptotically good.

Assume that q is even or 4 | (q − 1) (i.e. q ≡ 3 (mod 4)).

Then the self-dual consta-dihedral codes over F are asymptotically good.

In particular, self-dual quasi-cyclic codes of index 2 over F are asymptotically good.

It is known that if q is odd then LCD dihedral codes of rate 1 2 are asymptotically good, see #OTHEREFR Theorem 1.2] . We get the following consequence.

Consta-Dihedral Codes Over Finite Fields

Abstract: It is proved in a reference (Fan, Lin, IEEE TIT, vol.67, pp.5016-5025) that the self-dual (LCD respectively) dihedral codes over a finite field F with |F | = q are asymptotically good if q is even (odd respectively). In this paper, we investigate the algebraic property and the asymptotic property of conta-dihedral codes over F , and show that: if q is even or 4 | (q − 1), then the self-dual consta-dihedral codes are asymptotically good; otherwise, the LCD consta-dihedral codes are asymptotically good. And, with the help of a technique developed in this paper, some errors in the reference mentioned above are corrected.

On Self-Dual Double Circulant Codes

Abstract: Self-dual double circulant codes of odd dimension are shown to be dihedral in even characteristic and consta-dihedral in odd characteristic. Exact counting formulae are derived for them, generalizing some old results of MacWilliams on the enumeration of circulant orthogonal matrices. These formulae, in turn, are instrumental in deriving a Varshamov–Gilbert bound on the relative minimum distance of this family of codes.


Models employing the pipeline architecture have highlighted the effects of errors arising from the ASR component #REFR .

Spoken Question Answering Spoken Question Answering (SQA) #OTHEREFR involves a set of approaches aimed at appropriately answering a spoken question.

There are two types of SQA systems: in context SQA systems #OTHEREFR and open domain SQA systems.

While open domain SQA systems are built with the aim to answer questions without context, in context SQA systems typically take the form of Machine Reading Comprehension (MRC) #OTHEREFR where the question is accompanied by a passage.

#OTHEREFR explored the use of sub-word strategy while #OTHEREFR utilized BERT based model to learn joint audio-text features with the aim of mitigating the errors from ASR.

However, such techniques do not explicitly consider the mapping between ASR transcript and the corresponding human transcription.

In our work, we present experiments with models explicitly aimed at exploiting this mapping as well, by attempting to directly learn the said mapping in a translation fashion.

In #OTHEREFR authors present a method to learn representations using contrastive learning and show improved performance on three SQA datasets.

While the augmentation strategies presented may implicitly contribute to improvements in representations learned, it is not clear how they explicitly improve the performance of an SQA system.

Heysquad: A Spoken Question Answering Dataset

Abstract: Human-spoken questions are critical to evaluating the performance of spoken question answering (SQA) systems that serve several realworld use cases including digital assistants. We present a new large-scale communityshared SQA dataset, HeySQuAD that consists of 76k human-spoken questions and 97k machine-generated questions and corresponding textual answers derived from the SQuAD QA dataset. The goal of HeySQuAD is to measure the ability of machines to understand noisy spoken questions and answer the questions accurately. To this end, we run extensive benchmarks on the human-spoken and machine-generated questions to quantify the differences in noise from both sources and its subsequent impact on the model and answering accuracy. Importantly, for the task of SQA, where we want to answer human-spoken questions, we observe that training using the transcribed human-spoken and original SQuAD questions leads to significant improvements (12.51%) over training using only the original SQuAD textual questions.

Spoken Squad: A Study Of Mitigating The Impact Of Speech Recognition Errors On Listening Comprehension

Abstract: Reading comprehension has been widely studied. One of the most representative reading comprehension tasks is Stanford Question Answering Dataset (SQuAD), on which machine is already comparable with human. On the other hand, accessing large collections of multimedia or spoken content is much more difficult and time-consuming than plain text content for humans. It's therefore highly attractive to develop machines which can automatically understand spoken content. In this paper, we propose a new listening comprehension task - Spoken SQuAD. On the new task, we found that speech recognition errors have catastrophic impact on machine comprehension, and several approaches are proposed to mitigate the impact.


In this work, we rather consider the use of local self-attention layers, and more specifically the stand-alone self-attention layers proposed by #REFR .

Several strategies have been proposed, including training a subnetwork to identify the important regions #OTHEREFR , or increasing the receptive field of CNNs.

The methods based on the latest strategy use self-attention mechanisms, which were introduced in computer vision tasks by #OTHEREFR .

They lead to impressive results compared to the performance achieved so far using CNNs, especially when it comes to the use of Vision Transformers (ViT) for various visual tasks #OTHEREFR .

However, this process is computationally very expensive, and it also requires a lot of training data.

In addition, for small object detection, the spatial dependencies are mainly local.

Deep-Nfa: A Deep A Contrario Framework For Small Object Detection

Abstract: The detection of small objects is a challenging task in computer vision. Conventional object detection methods have difficulty in finding the balance between high detection and low false alarm rates. In the literature, some methods have addressed this issue by enhancing the feature map responses, but without guaranteeing robustness with respect to the number of false alarms induced by background elements. To tackle this problem, we introduce an a contrario decision criterion into the learning process to take into account the unexpectedness of small objects. This statistic criterion enhances the feature map responses while controlling the number of false alarms (NFA) and can be integrated into any semantic segmentation neural network. Our add-on NFA module not only allows us to obtain competitive results for small target and crack detection tasks respectively, but also leads to more robust and interpretable results.

Stand-Alone Self-Attention In Vision Models

Abstract: Convolutions are a fundamental building block of modern computer vision systems. Recent approaches have argued for going beyond convolutions in order to capture long-range dependencies. These efforts focus on augmenting convolutional models with content-based interactions, such as self-attention and non-local means, to achieve gains on a number of vision tasks. The natural question that arises is whether attention can be a stand-alone primitive for vision models instead of serving as just an augmentation on top of convolutions. In developing and testing a pure self-attention vision model, we verify that self-attention can indeed be an effective stand-alone layer. A simple procedure of replacing all instances of spatial convolutions with a form of self-attention to ResNet-50 produces a fully self-attentional model that outperforms the baseline on ImageNet classification with 12% fewer FLOPS and 29% fewer parameters. On COCO object detection, a fully self-attention model matches the mAP of a baseline RetinaNet while having 39% fewer FLOPS and 34% fewer parameters. Detailed ablation studies demonstrate that self-attention is especially impactful when used in later layers. These results establish that stand-alone self-attention is an important addition to the vision practitioner's toolbox.

local self-attention layers
stand-alone self-attention layers


Iv. Insufficiently Separated Markov Chains

This Lemma 3 bound seems strikingly similar to the bounding result of Lemma 2 in #REFR with relative entropy rate between chains replacing the role of relative entropy between measurement densities in that result.

For sufficiently large k, we can now expand the second term of (5) as

where the first line follows from the definition of expectation operation, the 2nd line follows from application of (6), the second last line follows from the definition of relative entropy rate between two chains and the last line follows becausê M b k−1 ≤ 1 and relative entropy rates such as R(A a |A b ) are non-negative.

The Lemma statement then follows from (5) and (7) by noting that for any δ < 0 there is a h δ > 0 such that for allM k−1 < h δ we have E π γ k M b k−1 < δ as log is monotonically increasing and the elements A i,j b and A i,j a are bounded in size (no greater than 1).

The lemma's assumptions on p(X 0 ) being the stationary distribution of A b and the requirement for sufficiently largely k have been included to simplify analysis rather than being fundamental to the bounding mechanism.

We would expect similar bounded behaviour under relaxation of these two assumptions.

The lemma's assumption of the existence of stationary distributions is ensured under this paper's standing assumption of aperiodic and irreducible chains #OTHEREFR Ch. 4] .

Whilst such ergodic chains are a large and useful class, the lemma does exclude non-ergodic chains such as those with transient states, absorbing states or chains that exhibit periodic behaviours.

We now investigate a phenomenon that occurs when the Markov chains A b and A a are too close in the sense of having small relative entropy rate R(A b |A a ) and are unable to overcome the change event's geometric prior information.

Exactly Optimal Quickest Change Detection Of Markov Chains

Abstract: This paper establishes that an exactly optimal rule for Bayesian Quickest Change Detection (QCD) of Markov chains is a threshold test on the no change posterior. We also provide a computationally efficient scalar filter for the no change posterior whose effort is independent of the dimension of the chains. We establish that an (undesirable) weak practical super-martingale phenomenon can be exhibited by the no change posterior when the before and after chains are too close in a relative entropy rate sense. The proposed detector is examined in simulation studies.

On The Informativeness Of Measurements In Shiryaev'S Bayesian Quickest Change Detection

Abstract: This paper provides the first description of a weak practical super-martingale phenomenon that can emerge in the test statistic in Shiryaev's Bayesian quickest change detection (QCD) problem. We establish that this super-martingale phenomenon can emerge under a condition on the relative entropy between pre and post change densities when the measurements are insufficiently informative to overcome the change time's geometric prior. We illustrate this super-martingale phenomenon in a simple Bayesian QCD problem which highlights the unsuitability of Shiryaev's test statistic for detecting subtle change events.


Random initialization of prototypes, proposed in #REFR , fails in the incremental learning (see Table 5 ).

L IR = H i=0 W j=0 |sim(p t−1 , z t i,j ) − sim(p t , z t i,j )| • S i,j (1)

is computed for the model stored before training task t, and S is a binary mask of size H × W , indicating the representation pixels with the highest similarity (γ quantile of those pixels).

Such similarity distillation gives higher plasticity when learning a new task but, at the same time, reduces the interpretability drift. Proximity-based prototype initialization.

Most probably because new prototypes can be created far from the old ones, which are only slightly different (e.g. wing prototypes of various bird species).

Therefore, we introduce initialization that sets new prototypes close to the old ones (see Figure 4) .

We start by passing training samples of task t We introduce a new proximity-based prototype initialization.

It starts by passing training samples of task t through the network (green dots) and choosing representations closest to existing prototypes (violet diamonds).

This results in many points, which we cluster (purple circles) to obtain the initial locations of task t prototypical parts (yellow diamonds).

Icicle: Interpretable Class Incremental Continual Learning

Abstract: Continual learning enables incremental learning of new tasks without forgetting those previously learned, resulting in positive knowledge transfer that can enhance performance on both new and old tasks. However, continual learning poses new challenges for interpretability, as the rationale behind model predictions may change over time, leading to interpretability concept drift. We address this problem by proposing Interpretable Class-InCremental LEarning (ICICLE), an exemplar-free approach that adopts a prototypical part-based approach. It consists of three crucial novelties: interpretability regularization that distills previously learned concepts while preserving user-friendly positive reasoning; proximity-based prototype initialization strategy dedicated to the fine-grained setting; and taskrecency bias compensation devoted to prototypical parts. Our experimental results demonstrate that ICICLE reduces the interpretability concept drift and outperforms the existing exemplar-free methods of common class-incremental learning when applied to concept-based models. We make the code available.

This Looks Like That: Deep Learning For Interpretable Image Recognition

Abstract: When we are faced with challenging image classification tasks, we often explain our reasoning by dissecting the image, and pointing out prototypical aspects of one class or another. The mounting evidence for each of the classes helps us make our final decision. In this work, we introduce a deep network architecture -- prototypical part network (ProtoPNet), that reasons in a similar way: the network dissects the image by finding prototypical parts, and combines evidence from the prototypes to make a final classification. The model thus reasons in a way that is qualitatively similar to the way ornithologists, physicians, and others would explain to people on how to solve challenging image classification tasks. The network uses only image-level labels for training without any annotations for parts of images. We demonstrate our method on the CUB-200-2011 dataset and the Stanford Cars dataset. Our experiments show that ProtoPNet can achieve comparable accuracy with its analogous non-interpretable counterpart, and when several ProtoPNets are combined into a larger network, it can achieve an accuracy that is on par with some of the best-performing deep models. Moreover, ProtoPNet provides a level of interpretability that is absent in other interpretable deep models.


In #REFR , experiments were performed with class counts that decay linearly or decay with a step-function.

Established benchmarks for long-tail image recognition #OTHEREFR have shaped the progress of long-tail methods.

These followed earlier efforts that investigated the desired data distribution characteristics for long-tail benchmarks.

They noted that a larger imbalance between majority (now known as 'head') and minority (i.e.

'tail') classes increases difficulty and that a longer tail negatively affects classifier performance for both linear and step class count decays.

Interestingly, imbalance was shown to affect higher complexity tasks (e.g. CIFAR) significantly more than lower complexity tasks (e.g. MNIST).

Step and exponential class count decays were also investigated in #OTHEREFR , with similar conclusions.

In #OTHEREFR , multiple long-tail versions of CIFAR #OTHEREFR were curated by changing the minimum class size.

Use Your Head: Improving Long-Tail Video Recognition

Abstract: This paper presents an investigation into long-tail video recognition. We demonstrate that, unlike naturallycollected video datasets and existing long-tail image benchmarks, current video benchmarks fall short on multiple long-tailed properties. Most critically, they lack few-shot classes in their tails. In response, we propose new video benchmarks that better assess long-tail recognition, by sampling subsets from two datasets: SSv2 and VideoLT. We then propose a method, Long-Tail Mixed Reconstruction (LMR), which reduces overfitting to instances from few-shot classes by reconstructing them as weighted combinations of samples from head classes. LMR then employs label mixing to learn robust decision boundaries. It achieves state-of-the-art average class accuracy on EPIC-KITCHENS and the proposed SSv2-LT and VideoLT-LT.

A Systematic Study Of The Class Imbalance Problem In Convolutional Neural Networks

Abstract: In this study, we systematically investigate the impact of class imbalance on classification performance of convolutional neural networks (CNNs) and compare frequently used methods to address the issue. Class imbalance is a common problem that has been comprehensively studied in classical machine learning, yet very limited systematic research is available in the context of deep learning. In our study, we use three benchmark datasets of increasing complexity, MNIST, CIFAR-10 and ImageNet, to investigate the effects of imbalance on classification and perform an extensive comparison of several methods to address the issue: oversampling, undersampling, two-phase training, and thresholding that compensates for prior class probabilities. Our main evaluation metric is area under the receiver operating characteristic curve (ROC AUC) adjusted to multi-class tasks since overall accuracy metric is associated with notable difficulties in the context of imbalanced data. Based on results from our experiments we conclude that (i) the effect of class imbalance on classification performance is detrimental; (ii) the method of addressing class imbalance that emerged as dominant in almost all analyzed scenarios was oversampling; (iii) oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance; (iv) as opposed to some classical machine learning models, oversampling does not cause overfitting of CNNs; (v) thresholding should be applied to compensate for prior class probabilities when overall number of properly classified cases is of interest.


Next, we conduct experiments to investigate the relation between performance and source corpora choice #REFR . According to the results, we have the following observations.

Meanwhile, each source corpus should possess as many identities to the target corpus as possible.

This idea could be achieved with properly designed similarity criteria, where the performance of models should show a monotonous change with corpora chosen along the similarity rankings.

Based on this idea, we consider the following hypotheses that may help in source corpora choice.

1) Semantics: the document embedding similarity, 2) Word overlapping: token cosine similarity, 3) Information coverage: ROUGE recall, 4) Information density: ROUGE precision, 5) Length: absolute difference in token length between articles.

Given a target corpus, we rank the source corpora and create meta-datasets in different ranking sections for each criterion.

First, the statistical similarity is more important than semantic similarity.

Second, corpora from the same domain may not always provide better knowledge.

Third, rather than choosing source corpora that can cover all used words in the target corpus, source corpora should have a high information density.

Fourth, source corpora with similar average article lengths are better than source corpora with different average article lengths.

With the above observations, we use the average ranking of the following criteria: 1) cosine similarity, 2) ROUGE precision, and 3) article length to choose our source corpora for the meta-dataset.

Spec: Summary Preference Decomposition For Low-Resource Abstractive Summarization

Abstract: Neural abstractive summarization has been widely studied and achieved great success with large-scale corpora. However, the considerable cost of annotating data motivates the need for learning strategies under low-resource settings. In this paper, we investigate the problems of learning summarizers with only few examples and propose corresponding methods for improvements. First, typical transfer learning methods are prone to be affected by data properties and learning objectives in the pretext tasks. Therefore, based on pretrained language models, we further present a meta learning framework to transfer few-shot learning processes from source corpora to the target corpus. Second, previous methods learn from training examples without decomposing the content and preference. The generated summaries could therefore be constrained by the preference bias in the training set, especially under low-resource settings. As such, we propose decomposing the contents and preferences during learning through the parameter modulation, which enables control over preferences during inference. Third, given a target application, specifying required preferences could be non-trivial because the preferences may be difficult to derive through observations. Therefore, we propose a novel decoding method to automatically estimate suitable preferences and generate corresponding summary candidates from the few training examples. Extensive experiments demonstrate that our methods achieve state-of-the-art performance on six diverse corpora with 30.11%/33.95%/27.51% and 26.74%/31.14%/24.48% average improvements on ROUGE-1/2/L under 10-and 100example settings.

Attractive Or Faithful? Popularity-Reinforced Learning For Inspired Headline Generation

Abstract: With the rapid proliferation of online media sources and published news, headlines have become increasingly important for attracting readers to news articles, since users may be overwhelmed with the massive information. In this paper, we generate inspired headlines that preserve the nature of news articles and catch the eye of the reader simultaneously. The task of inspired headline generation can be viewed as a specific form of Headline Generation (HG) task, with the emphasis on creating an attractive headline from a given news article. To generate inspired headlines, we propose a novel framework called POpularity-Reinforced Learning for inspired Headline Generation (PORL-HG). PORL-HG exploits the extractive-abstractive architecture with 1) Popular Topic Attention (PTA) for guiding the extractor to select the attractive sentence from the article and 2) a popularity predictor for guiding the abstractor to rewrite the attractive sentence. Moreover, since the sentence selection of the extractor is not differentiable, techniques of reinforcement learning (RL) are utilized to bridge the gap with rewards obtained from a popularity score predictor. Through quantitative and qualitative experiments, we show that the proposed PORL-HG significantly outperforms the state-of-the-art headline generation models in terms of attractiveness evaluated by both human (71.03%) and the predictor (at least 27.60%), while the faithfulness of PORL-HG is also comparable to the state-of-the-art generation model.


The purpose of zero-shot semantic segmentation is to enable the model to predict new categories exclusively from auxiliary semantic information without any extra pixel-level annotations of new categories #REFR , further relaxing the annotation requirements for new categories.

The semantic segmentation task is to make pixel-by-pixel category predictions for an image on a fixed category label #OTHEREFR .

Few-shot semantic segmentation is to relax the restrictions of fixed categories #OTHEREFR .

By providing several images of new categories and pixel-level annotations (support set), the dense prediction ability of the network can be generalized to new categories.

However, few-shot semantic segmentation needs to provide at least one annotation of new categories.

Therefore, the model must establish the correlation between the image and auxiliary semantic information from seen categories.

The rise of CLIP provides a fresh perspective on the interaction between auxiliary semantic information and image information.

Prompt learning was first widely used in NLP #OTHEREFR .

The initial approach was to design different hard prompts to bridge the gap between the downstream task and the pre-trained model, guiding the model to complete the corresponding downstream task #OTHEREFR .

Cls] Token Is All You Need For Zero-Shot Semantic Segmentation

Abstract: In this paper, we propose an embarrassingly simple yet highly effective zero-shot semantic segmentation (ZS3) method, based on the pre-trained vision-language model CLIP. First, our study provides a couple of key discoveries: (i) the global tokens (a.k.a [CLS] tokens in Transformer) of the text branch in CLIP provide a powerful representation of semantic information and (ii) these text-side [CLS] tokens can be regarded as category priors to guide CLIP visual encoder pay more attention on the corresponding region of interest. Based on that, we build upon the CLIP model as a backbone which we extend with a One-Way [CLS] token navigation from text to the visual branch that enables zero-shot dense prediction, dubbed ClsCLIP. Specifically, we use the [CLS] token output from the text branch, as an auxiliary semantic prompt, to replace the [CLS] token in shallow layers of the ViT-based visual encoder. This one-way navigation embeds such global category prior earlier and thus promotes semantic segmentation. Furthermore, to better segment tiny objects in ZS3, we further enhance ClsCLIP with a local zoom-in strategy, which employs a region proposal pre-processing and we get ClsCLIP+. Extensive experiments demonstrate that our proposed ZS3 method achieves a SOTA performance, and it is even comparable with those few-shot semantic segmentation methods.

Zero-Shot Semantic Segmentation

Abstract: Semantic segmentation models are limited in their ability to scale to large numbers of object classes. In this paper, we introduce the new task of zero-shot semantic segmentation: learning pixel-wise classifiers for never-seen object categories with zero training examples. To this end, we present a novel architecture, ZS3Net, combining a deep visual segmentation model with an approach to generate visual representations from semantic word embeddings. By this way, ZS3Net addresses pixel classification tasks where both seen and unseen categories are faced at test time (so called generalized zero-shot classification). Performance is further improved by a self-training step that relies on automatic pseudo-labeling of pixels from unseen classes. On the two standard segmentation datasets, Pascal-VOC and Pascal-Context, we propose zero-shot benchmarks and set competitive baselines. For complex scenes as ones in the Pascal-Context dataset, we extend our approach by using a graph-context encoding to fully leverage spatial context priors coming from class-wise segmentation maps.


A.2. Additional Experiments A.2.1 Evaluation Of Light-Net

To evaluate the performance of Light-Net, we randomly sampled a testing set of 200 images from LIME #REFR .

It is a synthetic dataset of Bigbird #OTHEREFR and ShapeNet #OTHEREFR objects with ground truth normal, albedo, and shading.

We use lighting coefficients predicted by Light-Net to render shading with ground truth normal maps.

By comparing the rendered shading and ground truths, we can evaluate the accuracy of the estimated lighting coefficients. For quantitative

Weakly-Supervised Single-View Image Relighting

Abstract: We present a learning-based approach to relight a single image of Lambertian and low-frequency specular objects. Our method enables inserting objects from photographs into new scenes and relighting them under the new environment lighting, which is essential for AR applications. To relight the object, we solve both inverse rendering and re-rendering. To resolve the ill-posed inverse rendering, we propose a weakly-supervised method by a low-rank constraint. To facilitate the weakly-supervised training, we contribute Relit, a large-scale (750K images) dataset of videos with aligned objects under changing illuminations. For re-rendering, we propose a differentiable specular rendering layer to render low-frequency non-Lambertian materials under various illuminations of spherical harmonics. The whole pipeline is end-to-end and efficient, allowing for a mobile app implementation of AR object insertion. Extensive evaluations demonstrate that our method achieves state-of-the-art performance. Project

Lime: Live Intrinsic Material Estimation

Abstract: We present the first end to end approach for real time material estimation for general object shapes with uniform material that only requires a single color image as input. In addition to Lambertian surface properties, our approach fully automatically computes the specular albedo, material shininess, and a foreground segmentation. We tackle this challenging and ill posed inverse rendering problem using recent advances in image to image translation techniques based on deep convolutional encoder decoder architectures. The underlying core representations of our approach are specular shading, diffuse shading and mirror images, which allow to learn the effective and accurate separation of diffuse and specular albedo. In addition, we propose a novel highly efficient perceptual rendering loss that mimics real world image formation and obtains intermediate results even during run time. The estimation of material parameters at real time frame rates enables exciting mixed reality applications, such as seamless illumination consistent integration of virtual objects into real world scenes, and virtual material cloning. We demonstrate our approach in a live setup, compare it to the state of the art, and demonstrate its effectiveness through quantitative and qualitative evaluation.


GraphSAGE's message-passing layer #REFR does not directly aggregate central node features with incoming messages.

GNNs utilize the concept of message-passing to encode semantically rich features within network-structured data.

Over time, multiple variations of message-passing layers have been proposed [40, #OTHEREFR .

For instance, GCN's message-passing layer #OTHEREFR weighs each incoming message with a fixed coefficient, the node degree, before aggregation.

In contrast, GAT's message-passing layer #OTHEREFR learns aggregation weights dynamically based on attention scores.

Instead, it distinguishes these two kinds of features and learns two different transformations, one on the central node and another on incoming messages.

EdgeConv #OTHEREFR aggregates the feature difference between the central node and its neighbors combined with the central node's features.

Thus, EdgeConv draws parallels to aggregating spatial vectors if the nodes embed spatial positions.

However, our proposed GAV layer differs significantly from EdgeConv, as our method explicitly constrains the update of vector embeddings to imitate the simplified dynamics of physical flow in spatial networks.

Importantly, only a few works tried to adapt the message-passing paradigm to spatial networks #OTHEREFR .

Link Prediction For Flow-Driven Spatial Networks

Abstract: Link prediction algorithms predict the existence of connections between nodes in network-structured data and are typically applied to refine the connectivity among nodes by proposing meaningful new links. In this work, we focus on link prediction for flow-driven spatial networks, which are embedded in a Euclidean space and relate to physical exchange and transportation processes (e.g., blood flow in vessels or traffic flow in road networks). To this end, we propose the Graph Attentive Vectors (GAV) link prediction framework. GAV models simplified dynamics of physical flow in spatial networks via an attentive, neighborhoodaware message-passing paradigm, updating vector embeddings in a constrained manner. We evaluate GAV on eight flow-driven spatial networks given by whole-brain vessel graphs and road networks. GAV demonstrates superior performances across all datasets and metrics and outperforms the current state-of-the-art on the ogbl-vessel benchmark by more than 18% (98.38 vs. 83.07 AUC).

Inductive Representation Learning On Large Graphs

Abstract: Low-dimensional embeddings of nodes in large graphs have proved extremely useful in a variety of prediction tasks, from content recommendation to identifying protein functions. However, most existing approaches require that all nodes in the graph are present during training of the embeddings; these previous approaches are inherently transductive and do not naturally generalize to unseen nodes. Here we present GraphSAGE, a general, inductive framework that leverages node feature information (e.g., text attributes) to efficiently generate node embeddings for previously unseen data. Instead of training individual embeddings for each node, we learn a function that generates embeddings by sampling and aggregating features from a node's local neighborhood. Our algorithm outperforms strong baselines on three inductive node-classification benchmarks: we classify the category of unseen nodes in evolving information graphs based on citation and Reddit post data, and we show that our algorithm generalizes to completely unseen graphs using a multi-graph dataset of protein-protein interactions.

central node features
GraphSAGE's message-passing layer


The metasurface cell in #REFR consists of two layers of dielec- tric substrate, as shown in Fig.15 (a).

To isolate the DC feed port from the microwave signal, three 30 nH inductors are used.

In this cell design, the phase difference between two adjacent states is (90

• ) in the range of frequency around 3.2 GHz.

Thus, this cell can be viewed as a 2-bit digital coded metasurface cell at all digital states around 3.2 GHz, and can achieve reflectance of more than 85%.

The square patch at the top is responsible for reflecting the incident electromagnetic wave, and integrates a PIN diode, which is connected to the ground plane through a via.

The chip inductance L = 33 nH is used to suppress the coupling between AC and ground.

Simulation results and experimental results are compared in Fig.15 Fig.

16 (a) demonstrated, the cell structure in #OTHEREFR consists of two pairs of rectangular patches with a slotted copper ground floor on the back of the dielectric plate.

Each pair of metal patches is connected by a varactor diode, and the junction capacitance is strongly affected by the external bias voltage.

Design Of Reconfigurable Intelligent Surfaces For Wireless Communication: A Review

Abstract: Existing literature reviews predominantly focus on the theoretical aspects of reconfigurable intelligent surfaces (RISs), such as algorithms and models, while neglecting a thorough examination of the associated hardware components. To bridge this gap, this research paper presents a comprehensive overview of the hardware structure of RISs. The paper provides a classification of RIS cell designs and prototype systems, offering insights into the diverse configurations and functionalities. Moreover, the study explores potential future directions for RIS development. Notably, a novel RIS prototype design is introduced, which integrates seamlessly with a communication system for performance evaluation through signal gain and image formation experiments. The results demonstrate the significant potential of RISs in enhancing communication quality within signal blind zones and facilitating effective radio wave imaging.

Intelligent Metasurface Imager And Recognizer

Abstract: It is ever-increasingly demanded to remotely monitor people in daily life using radio-frequency probing signals. However, conventional systems can hardly be deployed in real-world settings since they typically require objects to either deliberately cooperate or carry a wireless active device or identification tag. To accomplish the complicated successive tasks using a single device in real time, we propose a smart metasurface imager and recognizer simultaneously, empowered by a network of artificial neural networks (ANNs) for adaptively controlling data flow. Here, three ANNs are employed in an integrated hierarchy: transforming measured microwave data into images of whole human body; classifying the specifically designated spots (hand and chest) within the whole image; and recognizing human hand signs instantly at Wi-Fi frequency of 2.4 GHz. Instantaneous in-situ imaging of full scene and adaptive recognition of hand signs and vital signs of multiple non-cooperative people have been experimentally demonstrated. We also show that the proposed intelligent metasurface system work well even when it is passively excited by stray Wi-Fi signals that ubiquitously exist in our daily lives. The reported strategy could open a new avenue for future smart cities, smart homes, human-device interactive interfaces, healthy monitoring, and safety screening free of visual privacy issues.