Progress has been made in this direction in #REFR where a methodology for equating carefully structured convolutional layers to finite difference quotients is developed.

Indeed, this has led to the development of several new technologies, such as physics informed neural networks (PINNs) #OTHEREFR .

Such models are powerful tools for approximating PDE dynamics as they typically build some level of PDE structure into the network, and may be exploited to both solve PDEs, and to identify an underlying PDE.

Outside of PINNs, significant progress has also been made in solving the inverse problem #OTHEREFR .

While PINNs have proven effective for solving given (families of) PDEs, they have not been shown to be efficient solution methods when using real data.

An important step towards incorporating physical knowledge into more standard networks is to construct neural networks which capture the structure of families of PDEs but are trained on data which does not necessarily arise from a PDE.

Indeed, by ensuring the network contains a variety of finite difference quotients one may train a neural network which accurately simulates dynamics for remarkably large classes of PDEs based on simulated data, although the network itself is highly restricted.

One crucial observation we make in this work (and, to a certain extent, in #OTHEREFR ) is that by imposing no restrictions on the convolutional layers one may build a convolutional neural network which contains within it a large class of finite difference discretisations of PDEs.

Specifically, here we require particular choices of activation functions and design the network with sufficiently wide convolutional layers.

We shall exploit this connection in our analysis in section 3.

An additional benefit of working with unrestricted networks is they contain more than discrete solutions to PDEs, and as such are more amenable to real world datasets.

Predictions Based On Pixel Data: Insights From Pdes And Finite Differences

Abstract: Neural networks are the state-of-the-art for many approximation tasks in high-dimensional spaces, as supported by an abundance of experimental evidence. However, we still need a solid theoretical understanding of what they can approximate and, more importantly, at what cost and accuracy. One network architecture of practical use, especially for approximation tasks involving images, is convolutional (residual) networks. However, due to the locality of the linear operators involved in these networks, their analysis is more complicated than for generic fully connected neural networks. This paper focuses on sequence approximation tasks, where a matrix or a higher-order tensor represents each observation. We show that when approximating sequences arising from space-time discretisations of PDEs we may use relatively small networks. We constructively derive these results by exploiting connections between discrete convolution and finite difference operators. Throughout, we design our network architecture to, while having guarantees, be similar to those typically adopted in practice for sequence approximation tasks. Our theoretical results are supported by numerical experiments which simulate linear advection, the heat equation, and the Fisher equation. The implementation used is available at the repository associated to the paper [30].

Pde-Net 2.0: Learning Pdes From Data With A Numeric-Symbolic Hybrid Deep Network

Abstract: Partial differential equations (PDEs) are commonly derived based on empirical observations. However, recent advances of technology enable us to collect and store massive amount of data, which offers new opportunities for data-driven discovery of PDEs. In this paper, we propose a new deep neural network, called PDE-Net 2.0, to discover (time-dependent) PDEs from observed dynamic data with minor prior knowledge on the underlying mechanism that drives the dynamics. The design of PDE-Net 2.0 is based on our earlier work [1] where the original version of PDE-Net was proposed. PDE-Net 2.0 is a combination of numerical approximation of differential operators by convolutions and a symbolic multi-layer neural network for model recovery. Comparing with existing approaches, PDE-Net 2.0 has the most flexibility and expressive power by learning both differential operators and the nonlinear response function of the underlying PDE model. Numerical experiments show that the PDE-Net 2.0 has the potential to uncover the hidden PDE of the observed dynamics, and predict the dynamical behavior for a relatively long time, even in a noisy environment.


For gradinet based simulations we used the standard BinaryNet implementation (BNET) (see e.g. #REFR ).

We report here the details for the various algorithms used in text. Spherical Perceptron algorithms.

For SGD we initialized the weights uniformly on the unit sphere and optimized the cross entropy loss using batchsize 200 and learning rate 1, with a maximum number of allowed epochs equal to 20000.

In the fBP simulations (see #OTHEREFR for details) we used = 10 replicas and an increasing coupling constant with 15 exponentially spaced values in the range [0.1, 10].

For each value of we initialized fBP with the messages obtained at the previous coupling value.

We than made it run until the maximum absolute difference between message was less than 10 −5 or until it reached 500 messages updates. Binary Perceptron algorithms.

We then optimized the cross entropy loss using full batch gradient descent with learing rate 1 and a maximum number of 20000 epochs.

For Stochastic Belief Propagation Inspired (SBPI) simulations (see #OTHEREFR ) we set the maximum number of allowed iterations to 1000 and used a threshold = 2 and a probability = 0.3 of updating the synapses with a stability 0 ≤ Δ ≤ .

For fBP simulations we used the same parameters as in the continuous weights case.

Typical And Atypical Solutions In Non-Convex Neural Networks With Discrete And Continuous Weights

Abstract: We study the binary and continuous negative-margin perceptrons as simple non-convex neural network models learning random rules and associations. We analyze the geometry of the landscape of solutions in both models and find important similarities and differences. Both models exhibit subdominant minimizers which are extremely flat and wide. These minimizers coexist with a background of dominant solutions which are composed by an exponential number of algorithmically inaccessible small clusters for the binary case (the frozen 1-RSB phase) or a hierarchical structure of clusters of different sizes for the spherical case (the full RSB phase). In both cases, when a certain threshold in constraint density is crossed, the local entropy of the wide flat minima becomes non-monotonic, indicating a break-up of the space of robust solutions into disconnected components. This has a strong impact on the behavior of algorithms in binary models, which cannot access the remaining isolated clusters. For the spherical case the behaviour is different, since even beyond the disappearance of the wide flat minima the remaining solutions are shown to always be surrounded by a large number of other solutions at any distance, up to capacity. Indeed, we exhibit numerical evidence that algorithms seem to find solutions up to the SAT/UNSAT transition, that we compute here using an 1RSB approximation. For both models, the generalization performance as a learning device is shown to be greatly improved by the existence of wide flat minimizers even when trained in the highly underconstrained regime of very negative margins.

Binarized Neural Networks

Abstract: We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time. At train-time the binary weights and activations are used for computing the parameter gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substantially improve power-efficiency. To validate the effectiveness of BNNs, we conducted two sets of experiments on the Torch7 and Theano frameworks. On both, BNNs achieved nearly state-of-the-art results over the MNIST, CIFAR-10 and SVHN datasets. We also report our preliminary results on the challenging ImageNet dataset. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available on-line.

gradinet based simulations
standard BinaryNet implementation


Besides 2D images, prior work also applied NST in spherical images for 3D movies and VR #REFR .

Prior work explored how to reconstruct realistic scenes in VR to elicit users' different affective responses. For example, Felnhofer et al.

#OTHEREFR created five VR parks with different environments, in which each scene elicited specific affective responses, e.g., joy, sadness, boredom, anger and anxiety.

However, customizing all details in VR environments is challenging, especially for eliciting specific affective responses.

Researchers have developed the neural style transfer (NST) to separate the content and style from a style image A and apply image A's style to image B's content to generate a new image with the same content as B but the same style as A #OTHEREFR .

Here, the style image refers to an image with a specific style reference, such as some famous painters' line strokes from existing artworks.

Although prior work altered users' perceptions by manipulating visual elements in VR, their affective state manipulation is based on their subjective judgment.

Therefore, our study aims to generate style-transferred 360°images that can affect users' affective states (valence and arousal levels) in VR.

We created an affective VR generating system using the NST approach ( Fig.

1) conducted a within-subject user study to evaluate the effectiveness of our system and approach.

The Exploration And Evaluation Of Generating Affective 360°Panoramic Vr Environments Through Neural Style Transfer

Abstract: Affective virtual reality (VR) environments with varying visual style can impact users' valence and arousal responses. We applied Neural Style Transfer (NST) to generate 360°VR environments that elicited users' varied valence and arousal responses. From a user study with 30 participants, findings suggested that generative VR environments changed participants' arousal responses but not their valence levels. The generated visual features, e.g., textures and colors, also altered participants' affective perceptions. Our work contributes novel insights about how users respond to generative VR environments and provided a strategy for creating affective VR environments without altering content.

Stereoscopic Neural Style Transfer

Abstract: This paper presents the first attempt at stereoscopic neural style transfer, which responds to the emerging demand for 3D movies or AR/VR. We start with a careful examination of applying existing monocular style transfer methods to left and right views of stereoscopic images separately. This reveals that the original disparity consistency cannot be well preserved in the final stylization results, which causes 3D fatigue to the viewers. To address this issue, we incorporate a new disparity loss into the widely adopted style loss function by enforcing the bidirectional disparity constraint in non-occluded regions. For a practical real-time solution, we propose the first feed-forward network by jointly training a stylization sub-network and a disparity sub-network, and integrate them in a feature level middle domain. Our disparity sub-network is also the first end-to-end network for simultaneous bidirectional disparity and occlusion mask estimation. Finally, our network is effectively extended to stereoscopic videos, by considering both temporal coherence and disparity consistency. We will show that the proposed method clearly outperforms the baseline algorithms both quantitatively and qualitatively.


Because the study #REFR shows no recommendations on how systematically identify the filter technique which fits best the concrete dataset, in the current study, we will empirically find an optimal for our dataset a filter method.

In the current work, based on the results of studies #OTHEREFR , to balance a dataset, we will apply the over-sampling method SMOTE.

We also use a learning algorithm called Random Forest Classifier to build a model.

The mentioned enhanced methods: SMOTEBoost and Weighted Random Forest, won't be used to avoid over complication of our model.

We will use the ROC convex hull method to evaluate the built model's ability to recommend an elicitation technique.

However, classification task metrics like precision and recall will be calculated to compare the effect of the proposed enhancements with the results achieved in the work #OTHEREFR .

Approaches To Improving The Accuracy Of Machine Learning Models In Requirements Elicitation Techniques Selection

Abstract: Selecting techniques is a crucial element of the business analysis approach planning in IT projects. Particular attention is paid to the choice of techniques for requirements elicitation. One of the promising methods for selecting techniques is using machine learning algorithms trained on the practitioners' experience considering different projects' contexts. The effectiveness of ML models is significantly affected by the balance of the training dataset, which is violated in the case of popular techniques. The paper aims to analyze the efficiency of the Synthetic Minority Over-sampling Technique usage in Machine Learning models for elicitation technique selection in case of the imbalanced training dataset and possible ways for positive feature importance selection. The computational experiment results confirmed the effectiveness of using the proposed approaches to improve the accuracy of machine learning models for selecting requirements elicitation techniques. Proposed approaches can be used to build Machine Learning models for business analysis activities planning in IT projects.

Feature Selection: A Data Perspective

Abstract: Feature selection, as a data preprocessing strategy, has been proven to be effective and efficient in preparing data (especially high-dimensional data) for various data-mining and machine-learning problems. The objectives of feature selection include building simpler and more comprehensible models, improving data-mining performance, and preparing clean, understandable data. The recent proliferation of big data has presented some substantial challenges and opportunities to feature selection. In this survey, we provide a comprehensive and structured overview of recent advances in feature selection research. Motivated by current challenges and opportunities in the era of big data, we revisit feature selection research from a data perspective and review representative feature selection algorithms for conventional data, structured data, heterogeneous data and streaming data. Methodologically, to emphasize the differences and similarities of most existing feature selection algorithms for conventional data, we categorize them into four main groups: similarity-based, information-theoretical-based, sparse-learning-based, and statistical-based methods. To facilitate and promote the research in this community, we also present an open source feature selection repository that consists of most of the popular feature selection algorithms (http://featureselection.asu.edu/). Also, we use it as an example to show how to evaluate feature selection algorithms. At the end of the survey, we present a discussion about some open problems and challenges that require more attention in future research.


Voice Emotion Maintenance Generating the speech with a specific emotion is a difficult task for speech synthesis because conventional TTS methods require TTS data with emotion labels to train #REFR .

Besides, we conduct a subjection evaluation to label foreign accents from 1 to 5 on randomly selected 20 synthesized speech for both English and Chinese, where each sample is measured with a score from 1 to 5 denoting high-status foreign speakers, low-status foreign speakers, middle-status speakers, low-status native speakers, and high-status native speakers, respectively.

As summarized in the last column of Table 6 , we observed that our VALL-E X can control the accent for the target speech by LID modules.

For example, in English→Chinese, VALL-E X Trans with right LID and without LID get the score of 4.03 and 2.35, respectively.

This indicates that by using correct LID embedding, VALL-E X Trans is able to alleviate the foreign accent problem.

Please also see the demo for audio examples of VALL-E X Trans with or without language ID.

Moreover, it is more tempting to reserve the source speaker's emotion in generated target speech for the S2ST task, which is not explored in previous S2ST work.

In these experiments, we adopt the source prompts from the emotional voices dataset EmoV-DB #OTHEREFR as inputs of VALL-E X Trans to generate the translated target speech, whose samples are listed on our demo page.

We found that the proposed VALL-E X can maintain emotional consistency to a certain extent between the source prompt and the synthesized speech.

The underlying reasons are (1) our VALL-E X is trained with large-scale multi-lingual multi-speaker speech-transcription data, which contains various emotional speech records, and (2) the strong in-context learning ability of VALL-E X, like GPT-3 #OTHEREFR , promotes the generated speech to reserve the characteristic of the source prompt.

Code-Switch Speech Synthesis It is a common phenomenon to use code-switch utterances in bilingual or multi-lingual communities #OTHEREFR .

Speak Foreign Languages With Your Own Voice: Cross-Lingual Neural Codec Language Modeling

Abstract: We propose a cross-lingual neural codec language model, VALL-E X, for crosslingual speech synthesis. Specifically, we extend VALL-E [Wang et al., 2023] and train a multilingual conditional codec language model to predict the acoustic token sequences of the target language speech by using both the source language speech and the target language text as prompts. VALL-E X inherits strong incontext learning capabilities and can be applied for zero-shot cross-lingual text-tospeech synthesis and zero-shot speech-to-speech translation tasks. Experimental results show that it can generate high-quality speech in the target language via just one speech utterance in the source language as a prompt while preserving the unseen speaker's voice, emotion, and acoustic environment. Moreover, VALL-E X effectively alleviates the foreign accent problems, which can be controlled by a language ID. Audio samples are available at https://aka.ms/vallex.

Emotional Speech Synthesis With Rich And Granularized Control

Abstract: This paper proposes an effective emotion control method for an end-to-end text-to-speech (TTS) system. To flexibly control the distinct characteristic of a target emotion category, it is essential to determine embedding vectors representing the TTS input. We introduce an inter-to-intra emotional distance ratio algorithm to the embedding vectors that can minimize the distance to the target emotion category while maximizing its distance to the other emotion categories. To further enhance the expressiveness of a target speech, we also introduce an effective interpolation technique that enables the intensity of a target emotion to be gradually changed to that of neutral speech. Subjective evaluation results in terms of emotional expressiveness and controllability show the superiority of the proposed algorithm to the conventional methods.

speech synthesis
Voice Emotion Maintenance


The output of the previous action is a tensor of size B × N × nPDE, as shown in Fig. 2 . Using automatic differentiation #REFR , the governing PDEs (Eqs.

In practice, the input of PIPN is a three-dimensional tensor (i.e., numeric array) of size B × N × 2.

Afterward, there are two sequential shared MLPs with sizes of (ns ×64, ns ×64) and (ns ×64, ns ×128, ns ×1024), as displayed in Fig. 2 .

In the next step, the symmetric operator (s) forms the global feature with a size of ns × 1024. As exhibited in Fig.

2 , the global feature is concatenated to the intermediate feature tensor of size B × N × (ns × 64), resulting in a tensor of size B × N × (ns × 1088).

Next, two other sequential shared MLPs, respectively, with sizes of (ns × 512, ns × 256, ns × 128) and (ns × 128, nPDE) operate in the PIPN architecture.

1-2) are built and fed into the PIPN loss function.

Note that reasonable choices for ns could be 0.125, 0.25, 0.5, 1.0, 2.0, and 4.0 such that a positive integer number represents the size of resulting shared MLPs.

Physics-Informed Pointnet: On How Many Irregular Geometries Can It Solve An Inverse Problem Simultaneously? Application To Linear Elasticity

Abstract: Regular physics-informed neural networks (PINNs) predict the solution of partial differential equations using sparse labeled data but only over a single domain. On the other hand, fully supervised learning models are first trained usually over a few thousand domains with known solutions (i.e., labeled data) and then predict the solution over a few hundred unseen domains. Physics-informed PointNet (PIPN) is primarily designed to fill this gap between PINNs (as weakly supervised learning models) and fully supervised learning models. In this article, we demonstrate that PIPN predicts the solution of desired partial differential equations over a few hundred domains simultaneously, while it only uses sparse labeled data. This framework benefits fast geometric designs in the industry when only sparse labeled data are available. Particularly, we show that PIPN predicts the solution of a plane stress problem over more than 500 domains with different geometries, simultaneously. Moreover, we pioneer implementing the concept of remarkable batch size (i.e., the number of geometries fed into PIPN at each sub-epoch) into PIPN. Specifically, we try batch sizes of 7, 14, 19, 38, 76, and 133. Additionally, the effect of the PIPN size, symmetric function in the PIPN architecture, and static and dynamic weights for the component of the sparse labeled data in the loss function are investigated.

Tensorflow: Large-Scale Machine Learning On Heterogeneous Distributed Systems

Abstract: TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms. A computation expressed using TensorFlow can be executed with little or no change on a wide variety of heterogeneous systems, ranging from mobile devices such as phones and tablets up to large-scale distributed systems of hundreds of machines and thousands of computational devices such as GPU cards. The system is flexible and can be used to express a wide variety of algorithms, including training and inference algorithms for deep neural network models, and it has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer science and other fields, including speech recognition, computer vision, robotics, information retrieval, natural language processing, geographic information extraction, and computational drug discovery. This paper describes the TensorFlow interface and an implementation of that interface that we have built at Google. The TensorFlow API and a reference implementation were released as an open-source package under the Apache 2.0 license in November, 2015 and are available at www.tensorflow.org.


However, the aggregation-like operations defined in #REFR result in the dependency of each node on its neighbors and the amount of dependent nodes for one node increases exponentially with the growth of layers, which results in the unexpected increases of batch size.

In recent years, neural networks #OTHEREFR , due to the impressive performance, have been extended to graph data, known as graph neural networks (GNN) #OTHEREFR .

As GNNs significantly improve the results of graph tasks, it has been extensively investigated from different aspects, such as graph convolution network (GCN) #OTHEREFR , graph attention networks (GAT) #OTHEREFR , spatial-temporal GNN (STGNN) #OTHEREFR , graph auto-encoder [9, #OTHEREFR , graph contrastive learning #OTHEREFR , etc.

Except for the variants that originate from different perspectives, an important topic is motivated by the well-known inefficiency of GNN.

In classical neural networks #OTHEREFR , the optimization is usually based on stochastic algorithms with limited batch #OTHEREFR since samples are independent of each other.

Some works are proposed based on neighbor sampling #OTHEREFR and graph approximation #OTHEREFR to limit the batch size, while some methods #OTHEREFR attempt to directly apply high-order graph operation and sacrifice the most non-linearity.

The training stability is a problem for neighbor sampling methods #OTHEREFR though VRGCN #OTHEREFR has attempted to control the variance via improving sampling.

Note that the required nodes may still grow (slowly) with the increase of depth.

ClusterGCN #OTHEREFR finds an approximate Figure 1 : Illustration of a stacked graph neural network decoupled from an L-layer GNN.

To train each module individually, some loss (e.g., semi-supervised loss, unsupervised loss, contrastive loss) is required and it is denoted by L graph with plenty of connected components so that the batch size is strictly upper-bounded.

Decouple Graph Neural Networks: Train Multiple Simple Gnns Simultaneously Instead Of One

Abstract: Graph neural networks (GNN) suffer from severe inefficiency due to the exponential growth of node dependency with the increase of layers. It extremely limits the application of stochastic optimization algorithms so that the training of GNN is usually time-consuming. To address this problem, we propose to decouple a multi-layer GNN as multiple simple modules for more efficient training, which is comprised of classical forward training (FT) and designed backward training (BT). Under the proposed framework, each module can be trained efficiently in FT by stochastic algorithms without distortion of graph information owing to its simplicity. To avoid the only unidirectional information delivery of FT and sufficiently train shallow modules with the deeper ones, we develop a backward training mechanism that makes the former modules perceive the latter modules, inspired by the classical backward propagation algorithm. The backward training introduces the reversed information delivery into the decoupled modules as well as the forward information delivery. To investigate how the decoupling and greedy training affect the representational capacity, we theoretically prove that the error produced by linear modules will not accumulate on unsupervised tasks in most cases. The theoretical and experimental results show that the proposed framework is highly efficient with reasonable performance, which may deserve more investigation. Preprint. Under review.

Inductive Representation Learning On Large Graphs

Abstract: Low-dimensional embeddings of nodes in large graphs have proved extremely useful in a variety of prediction tasks, from content recommendation to identifying protein functions. However, most existing approaches require that all nodes in the graph are present during training of the embeddings; these previous approaches are inherently transductive and do not naturally generalize to unseen nodes. Here we present GraphSAGE, a general, inductive framework that leverages node feature information (e.g., text attributes) to efficiently generate node embeddings for previously unseen data. Instead of training individual embeddings for each node, we learn a function that generates embeddings by sampling and aggregating features from a node's local neighborhood. Our algorithm outperforms strong baselines on three inductive node-classification benchmarks: we classify the category of unseen nodes in evolving information graphs based on citation and Reddit post data, and we show that our algorithm generalizes to completely unseen graphs using a multi-graph dataset of protein-protein interactions.


We will now describe the enhancements to the three stages of the anomaly detection framework compared to #REFR .

The objective of the anomaly detection framework is to generate anomaly scores s i for classification of data sample x i given a threshold AD th such that:

y i =        0, if s i < AD th 1, if s i ≥ AD th

y i = 0 indicates that the sample x i is a normal sample, whereas y i = 1 indicates that the sample x i is an anomaly.

The details about the anomaly score s i are discussed in Section 3.4.

Few-Shot Weakly-Supervised Cybersecurity Anomaly Detection

Abstract: With increased reliance on Internet based technologies, cyberattacks compromising users' sensitive data are becoming more prevalent. The scale and frequency of these attacks are escalating rapidly, affecting systems and devices connected to the Internet. The traditional defense mechanisms may not be sufficiently equipped to handle the complex and ever-changing new threats. The significant breakthroughs in the machine learning methods including deep learning, had attracted interests from the cybersecurity research community for further enhancements in the existing anomaly detection methods. Unfortunately, collecting labelled anomaly data for all new evolving and sophisticated attacks is not practical. Training and tuning the machine learning model for anomaly detection using only a handful of labelled data samples is a pragmatic approach. Therefore, few-shot weakly supervised anomaly detection is an encouraging research direction. In this paper, we propose an enhancement to an existing few-shot weakly-supervised deep learning anomaly detection framework. This framework incorporates data augmentation, representation learning and ordinal regression. We then evaluated and showed the performance of our implemented framework on three benchmark datasets: NSL-KDD, CIC-IDS2018, and TON IoT.

Weakly-Supervised Deep Anomaly Detection With Pairwise Relation Learning.

Abstract: This paper studies a rarely explored but critical anomaly detection problem: weakly-supervised anomaly detection with limited labeled anomalies and a large unlabeled data set. This problem is very important because it (i) enables anomaly-informed modeling which helps identify anomalies of interests and address the notorious high false positives in unsupervised anomaly detection, and (ii) eliminates the reliance on large-scale and complete labeled anomaly data in fully-supervised settings. However, the problem is especially challenging since we have only limited labeled data for a single class, and moreover, the seen anomalies often cannot cover all types of anomalies (i.e., unseen anomalies). We address this problem by formulating the problem as a pairwise relation learning task. Particularly, our approach defines a two-stream ordinal regression network to learn the relation of randomly selected instance pairs, i.e., whether the instance pair contains labeled anomalies or just unlabeled data instances. The resulting model leverages both the labeled and unlabeled data to effectively augment the data and learn generalized representations of both normality and abnormality. Extensive empirical results show that our approach (i) significantly outperforms state-of-the-art competing methods in detecting both seen and unseen anomalies and (ii) is substantially more data-efficient.


We use a conventional machine-learning method, namely, Extreme Gradient Boost (XGBoost), as our baseline #REFR .

The XGBoost algorithm is a boosting-based method for integrated learning.

In our XGBoost model, we select the optimal hyperparameters using the GridSearch method in scikit-learn #OTHEREFR .

Additionally, we use neural networks as our second model.

Since the data distribution is not particularly complex, we construct a simple multi-layer perception with three hidden layers as our neural network model.

As we have two input features, i.e., the distance between charging coils and the charging duration, we set the number of neurons in the input layer to two.

Monitoring Efficiency Of Iot Wireless Charging

Abstract: Crowdsourcing wireless energy is a novel and convenient solution to charge nearby IoT devices. Several applications have been proposed to enable peer-to-peer wireless energy charging. However, none of them considered the energy efficiency of the wireless transfer of energy. In this paper, we propose an energy estimation framework that predicts the actual received energy. Our framework uses two machine learning algorithms, namely XGBoost and Neural Network, to estimate the received energy. The result shows that the Neural Network model is better than XGBoost at predicting the received energy. We train and evaluate our models by collecting a real wireless energy dataset.

Xgboost: A Scalable Tree Boosting System

Abstract: Tree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.


These connections could be established using the same carrier frequencies (intra-frequency MC) or different carrier frequencies (inter-frequency MC) #REFR .

In #OTHEREFR , this requirement is referred as uniform experience.

However, maintaining uniform connection performance for all the users during their entire sessions is challenging, especially for high mobility use cases.

For instance, varying connection bandwidth, addressed via dynamic rate adaption, leads to varying resolution #OTHEREFR , which, coupled with latency variation, results in fluctuating XR experience or even motion sickness.

There are two approaches to achieving an ultra-reliable connection: simultaneous multi-connectivity (MC) #OTHEREFR and massive multiple input multiple output (MIMO) #OTHEREFR .

MC's main objective is to improve the connection's data rates and reliability, i.e., achieve uniform experience by using a diversity of connections.

An example of the intra-frequency MC is single-frequency networks (SFNs), where several transmitters use the same frequency to send the same signal to the same user jointly.

In inter-frequency MC, a transmitter (or several transmitters) uses multiple frequencies simultaneously to send the same signal.

An example of interfrequency MC is when a transmitter is connected to multiple 5G/4G base stations and Wi-Fi access points simultaneously to increase the connection reliability.

Although MC enhances reliability significantly, it comes with the major drawbacks of more resource consumption and additional delays #OTHEREFR .

When a transmitter utilizes multiple resources, it will negatively impact the delay experienced by the remaining transmitters.

Traffic Characteristics Of Extended Reality

Abstract: This tutorial paper analyzes the traffic characteristics of immersive experiences with extended reality (XR) technologies, including Augmented reality (AR), virtual reality (VR), and mixed reality (MR). The current trend in XR applications is to offload the computation and rendering to an external server and use wireless communications between the XR head-mounted display (HMD) and the access points. This paradigm becomes essential owing to (1) its high flexibility (in terms of user mobility) compared to remote rendering through a wired connection, and (2) the high computing power available on the server compared to local rendering (on HMD). The requirements to facilitate a pleasant XR experience are analyzed in three aspects: capacity (throughput), latency, and reliability. For capacity, two VR experiences are analyzed: a human eyelike experience and an experience with the Oculus Quest 2 HMD. For latency, the key components of the motion-to-photon (MTP) delay are discussed. For reliability, the maximum packet loss rate (or the minimum packet delivery rate) is studied for different XR scenarios. Specifically, the paper reviews optimization techniques that were proposed to reduce the latency, conserve the bandwidth, extend the scalability, and/or increase the reliability to satisfy the stringent requirements of the emerging XR applications.

Towards Low-Latency And Ultra-Reliable Virtual Reality

Abstract: Virtual Reality (VR) is expected to be one of the killer-applications in 5G networks. However, many technical bottlenecks and challenges need to be overcome to facilitate its wide adoption. In particular, VR requirements in terms of high-throughput, low-latency and reliable communication call for innovative solutions and fundamental research cutting across several disciplines. In view of this, this article discusses the challenges and enablers for ultra-reliable and low-latency VR. Furthermore, in an interactive VR gaming arcade case study, we show that a smart network design that leverages the use of mmWave communication, edge computing and proactive caching can achieve the future vision of VR over wireless.


Problem (6) can be solved efficiently by the alternating least squares matrix factorization procedure #REFR .

Since we deal with a third order tensor, this factorization has three iterative steps.

Then, it fixes two factor matrices and minimizes the error with respect to the considered factor matrix as:

Joint Sensing, Communication, And Ai: A Trifecta For Resilient Thz User Experiences

Abstract: In this paper a novel joint sensing, communication, and artificial intelligence (AI) framework is proposed so as to optimize extended reality (XR) experiences over terahertz (THz) wireless systems. The proposed framework consists of three main components. First, a tensor decomposition framework is proposed to extract unique sensing parameters for XR users and their environment by exploiting then THz channel sparsity. Essentially, THz band's quasi-opticality is exploited and the sensing parameters are extracted from the uplink communication signal, thereby allowing for the use of the same waveform, spectrum, and hardware for both communication and sensing functionalities. Then, the Cramer-Rao lower bound is derived to assess the accuracy of the estimated sensing parameters. Second, a non-autoregressive multi-resolution generative artificial intelligence (AI) framework integrated with an adversarial transformer is proposed to predict missing and future sensing information. The proposed framework offers robust and comprehensive historical sensing information and anticipatory forecasts of future environmental changes, which are generalizable to fluctuations in both known and unforeseen user behaviors and environmental conditions. Third, a multi-agent deep recurrent hysteretic Q-neural network is developed to control the handover policy of reconfigurable intelligent surface (RIS) subarrays, leveraging the informative nature of sensing information to minimize handover cost, maximize the individual quality of personal experiences (QoPEs), and improve the robustness and resilience of THz links. Simulation results show a high generalizability of the proposed unsupervised generative AI framework to fluctuations in user behavior and velocity, leading to a 61 % improvement in instantaneous reliability compared to schemes with known channel state information.

Tensor Decomposition For Signal Processing And Machine Learning

Abstract: Tensors or multi-way arrays are functions of three or more indices $(i,j,k,\cdots)$ -- similar to matrices (two-way arrays), which are functions of two indices $(r,c)$ for (row,column). Tensors have a rich history, stretching over almost a century, and touching upon numerous disciplines; but they have only recently become ubiquitous in signal and data analytics at the confluence of signal processing, statistics, data mining and machine learning. This overview article aims to provide a good starting point for researchers and practitioners interested in learning about and working with tensors. As such, it focuses on fundamentals and motivation (using various application examples), aiming to strike an appropriate balance of breadth and depth that will enable someone having taken first graduate courses in matrix algebra and probability to get started doing research and/or developing tensor algorithms and software. Some background in applied optimization is useful but not strictly required. The material covered includes tensor rank and rank decomposition; basic tensor factorization models and their relationships and properties (including fairly good coverage of identifiability); broad coverage of algorithms ranging from alternating optimization to stochastic gradient; statistical performance analysis; and applications ranging from source separation to collaborative filtering, mixture and topic modeling, classification, and multilinear subspace learning.


Miyauchi and Takeda #REFR considered the uncertainty of edge weights rather than the existence of edges.

Zou #OTHEREFR studied the densest subgraph problem on uncertain graphs.

An uncertain graph is a pair of G = (V, E) and p : E → [0, 1], where e ∈ E is present with probability p(e) whereas e ∈ E is absent with probability 1 − p(e) #OTHEREFR .

In the problem, given an uncertain graph G = (V, E) with p, we seek S ⊆ V that maximizes the expected density.

Zou #OTHEREFR showed that this problem can be reduced to DSP on edge-weighted graphs, and designed a polynomial-time exact algorithm based on the reduction.

To model that, they assumed that there is an edge-weight space I = × e∈E [l e , r e ] ⊆ × e∈E [0, ∞) that contains the unknown true edge weight w.

To evaluate the performance of S ⊆ V without any concrete edge weight, they employed a well-known measure in the field of robust optimization, called the robust ratio.

In their scenario, the robust ratio of S ⊆ V under I is defined as the multiplicative gap between the density of S in terms of edge weight w and the density of S * w in terms of edge weight w under the worst-case edge weight w ∈ I, where S * w is a densest subgraph on G with w .

Intuitively, S ⊆ V with a large robust ratio has a density close to the optimal value even on G with the edge weight selected adversarially from I.

Using the robust ratio, they introduced the robust densest subgraph problem: Given an undirected graph G = (V, E) and an edge-weight space I = × e∈E [l e , r e ] ⊆ × e∈E [0, ∞), we are asked to find S ⊆ V that maximizes the robust ratio under I.

A Survey On The Densest Subgraph Problem And Its Variants

Abstract: The Densest Subgraph Problem requires to find, in a given graph, a subset of vertices whose induced subgraph maximizes a measure of density. The problem has received a great deal of attention in the algorithmic literature over the last five decades, with many variants proposed and many applications built on top of this basic definition. Recent years have witnessed a revival of research interest on this problem with several interesting contributions, including some groundbreaking results, published in 2022 and 2023. This survey provides a deep overview of the fundamental results and an exhaustive coverage of the many variants proposed in the literature, with a special attention on the most recent results. The survey also presents a comprehensive overview of applications and discusses some interesting open problems for this evergreen research topic.

Robust Densest Subgraph Discovery

Abstract: Dense subgraph discovery is an important primitive in graph mining, which has a wide variety of applications in diverse domains. In the densest subgraph problem, given an undirected graph $G=(V,E)$ with an edge-weight vector $w=(w_e)_{e\in E}$, we aim to find $S\subseteq V$ that maximizes the density, i.e., $w(S)/|S|$, where $w(S)$ is the sum of the weights of the edges in the subgraph induced by $S$. Although the densest subgraph problem is one of the most well-studied optimization problems for dense subgraph discovery, there is an implicit strong assumption; it is assumed that the weights of all the edges are known exactly as input. In real-world applications, there are often cases where we have only uncertain information of the edge weights. In this study, we provide a framework for dense subgraph discovery under the uncertainty of edge weights. Specifically, we address such an uncertainty issue using the theory of robust optimization. First, we formulate our fundamental problem, the robust densest subgraph problem, and present a simple algorithm. We then formulate the robust densest subgraph problem with sampling oracle that models dense subgraph discovery using an edge-weight sampling oracle, and present an algorithm with a strong theoretical performance guarantee. Computational experiments using both synthetic graphs and popular real-world graphs demonstrate the effectiveness of our proposed algorithms.


This approximation of the second term on the right-hand side of (20) is significantly worse that the approximation of the transformed Schur complement S by #REFR , since it involves the product of three approximations rather than one.

Firstly, the offdiagonal blocks B α and B T α do not enjoy the same favorable sparsity as D and D T , and therefore are more expensive to compute.

Secondly, and more importantly, the Schur complement of the original system takes the form #OTHEREFR S

= W α + B α M −1 β B T α = W α + W α DM −1 β D T W α .

In order to construct block preconditioners, approximations of the inverse of S are required.

As in the construction of the approximate Schur complement S in (16), it is possible to replace W α and M β by their diagonals, to obtain a matrix S spectrally equivalent to S .

A very rough estimate would suggest that, while still asymptotically optimal, the condition number of S −1 S could be very roughly bounded by the cube of the condition number of S −1 S.

In Figure 3 , we display computed condition numbers of the transformed preconditioned Schur complement S −1 S and the untransformed version S −1 S for increasing polynomial degree.

We consider both the unit cube [0, 1] d , and a skewed single-element mesh (the same mesh used in the results shown in Figure 2 ).

Although both versions of the preconditioned Schur complement have O(1) condition number with respect to polynomial degree, the empirical results show that the transformed Schur complement results in significantly smaller condition number, in accordance with arguments described above.

These results justify the use of the transformed system (11), despite the increased cost per iteration incurred by the inverse L 2 mass matrix.

Matrix-Free Gpu-Accelerated Saddle-Point Solvers For High-Order Problems In H(Div)

Abstract: This work describes the development of matrix-free GPU-accelerated solvers for high-order finite element problems in H(div). The solvers are applicable to grad-div and Darcy problems in saddle-point formulation, and have applications in radiation diffusion and porous media flow problems, among others. Using the interpolation-histopolation basis (cf. SIAM J. Sci. Comput., 45 (2023), A675-A702), efficient matrix-free preconditioners can be constructed for the (1, 1)-block and Schur complement of the block system. With these approximations, block-preconditioned MINRES converges in a number of iterations that is independent of the mesh size and polynomial degree. The approximate Schur complement takes the form of an M-matrix graph Laplacian, and therefore can be well-preconditioned by highly scalable algebraic multigrid methods. High-performance GPU-accelerated algorithms for all components of the solution algorithm are developed, discussed, and benchmarked. Numerical results are presented on a number of challenging test cases, including the "crooked pipe" grad-div problem, the SPE10 reservoir modeling benchmark problem, and a nonlinear radiation diffusion test case.

Algebraic Hybridization And Static Condensation With Application To Scalable H(Div) Preconditioning

Abstract: We propose a unified algebraic approach for the practical application and preconditioning of static condensation and hybridization, two popular techniques in finite element discretizations. We demo...


The proof that T ′ is a tangle of order 4 is identical to the finite case in #REFR , so we only need to prove that T ′ remains a well-quasi-ordered set under .

In the context of finite graphs, #OTHEREFR proves the following lemmas that extend directly to the locally finite case:

Lemma 3.24 (Corollary 4.24 in #OTHEREFR ).

The graph G ′ resulting from the contraction of s 1 s 2 is 3-connected.

Lemma 3.25 (Lemmas 4.26 and 4.27 in #OTHEREFR ). T ′ is a region tangle of order 4.

Assume for the sake of contradiction that T ′ contains a infinite strictly decreasing sequence of separations (Y ′ n , S ′ n , Z ′ n ) n∈N .

By Lemma 3.24, G ′ is 3-connected, so these separations are tight.

By Lemma 3.1, there are finitely many integers n for which s ′ ∈ S ′ n .

Up to extracting an infinite subsequence, one can assume that for all n, either s ′ ∈ Y ′ n or s ′ ∈ Z ′ n .

If there exists N such that s ′ ∈ Y ′ N then for all n N we must have s ′ ∈ Y n by definition of .

The Structure Of Quasi-Transitive Graphs Avoiding A Minor With Applications To The Domino Problem

Abstract: An infinite graph is quasi-transitive if its vertex set has finitely many orbits under the action of its automorphism group. In this paper we obtain a structure theorem for locally finite quasi-transitive graphs avoiding a minor, which is reminiscent of the Robertson-Seymour Graph Minor Structure Theorem. We prove that every locally finite quasi-transitive graph G avoiding a minor has a treedecomposition whose torsos are finite or planar; moreover the tree-decomposition is canonical, i.e. invariant under the action of the automorphism group of G. As applications of this result, we prove the following. • Every locally finite quasi-transitive graph attains its Hadwiger number, that is, if such a graph contains arbitrarily large clique minors, then it contains an infinite clique minor. This extends a result of Thomassen (1992) who proved it in the 4-connected case and suggested that this assumption could be omitted. In particular, this shows that a Cayley graph excludes a finite minor if and only if it avoids the countable clique as a minor. • Locally finite quasi-transitive graphs avoiding a minor are accessible (in the sense of Thomassen and Woess), which extends known results on planar graphs to any proper minor-closed family. • Minor-excluded finitely generated groups are accessible (in the group-theoretic sense) and finitely presented, which extends classical results on planar groups. • The domino problem is decidable in a minor-excluded finitely generated group if and only if the group is virtually free, which proves the minorexcluded case of a conjecture of Ballier and Stein (2018).

Quasi-4-Connected Components

Abstract: We introduce a new decomposition of a graphs into quasi-4-connected components, where we call a graph quasi-4-connected if it is 3-connected and it only has separations of order 3 that remove a single vertex. Moreover, we give a cubic time algorithm computing the decomposition of a given graph. Our decomposition into quasi-4-connected components refines the well-known decompositions of graphs into biconnected and triconnected components. We relate our decomposition to Robertson and Seymour's theory of tangles by establishing a correspondence between the quasi-4-connected components of a graph and its tangles of order 4.


However, the naive pooling is independent of the image content, which can lead to suboptimal results. Therefore, Chen et al. #REFR and Yang et al.

Since the breakthrough of neural network in semantic segmentation #OTHEREFR , they were improved in various aspects such as integrating context #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR , attention mechanisms #OTHEREFR , #OTHEREFR , and Transformer backbones #OTHEREFR , #OTHEREFR .

Several architectures #OTHEREFR , #OTHEREFR , #OTHEREFR utilize intermediate features with different scales, which are generated from a single scale input, to aggregate context information.

Furthermore, multi-scale input inference, where predictions from scaled versions of an image are combined via average or max pooling, is often used to obtain better results #OTHEREFR , #OTHEREFR .

#OTHEREFR segment multi-scale inputs and learn an attention-weighted sum of the predictions. Tao et al.

#OTHEREFR further propose a hierarchical attention that is agnostic to the number of scales during inference.

For image classification, CNNs were shown to be sensitive to distribution shifts such as image corruptions #OTHEREFR or domain shifts #OTHEREFR .

Recent works #OTHEREFR , #OTHEREFR show that Transformers are more robust than CNNs with respect to these properties.

While CNNs focus on textures #OTHEREFR , Transformers put more importance on the object shape #OTHEREFR , #OTHEREFR , which is more similar to human vision #OTHEREFR .

Domain Adaptive And Generalizable Network Architectures And Training Strategies For Semantic Image Segmentation

Abstract: Unsupervised domain adaptation (UDA) and domain generalization (DG) enable machine learning models trained on a source domain to perform well on unlabeled or even unseen target domains. As previous UDA&DG semantic segmentation methods are mostly based on outdated networks, we benchmark more recent architectures, reveal the potential of Transformers, and design the DAFormer network tailored for UDA&DG. It is enabled by three training strategies to avoid overfitting to the source domain: While (1) Rare Class Sampling mitigates the bias toward common source domain classes, (2) a Thing-Class ImageNet Feature Distance and (3) a learning rate warmup promote feature transfer from ImageNet pretraining. As UDA&DG are usually GPU memory intensive, most previous methods downscale or crop images. However, low-resolution predictions often fail to preserve fine details while models trained with cropped images fall short in capturing long-range, domain-robust context information. Therefore, we propose HRDA, a multi-resolution framework for UDA&DG, that combines the strengths of small high-resolution crops to preserve fine segmentation details and large low-resolution crops to capture long-range context dependencies with a learned scale attention. DAFormer and HRDA significantly improve the state-of-the-art UDA&DG by more than 10 mIoU on 5 different benchmarks. The implementation is available at https://github.com/lhoyer/HRDA.

Attention To Scale: Scale-Aware Semantic Image Segmentation

Abstract: Incorporating multi-scale features in fully convolutional neural networks (FCNs) has been a key element to achieving state-of-the-art performance on semantic image segmentation. One common way to extract multi-scale features is to feed multiple resized input images to a shared deep network and then merge the resulting features for pixelwise classification. In this work, we propose an attention mechanism that learns to softly weight the multi-scale features at each pixel location. We adapt a state-of-the-art semantic image segmentation model, which we jointly train with multi-scale input images and the attention model. The proposed attention model not only outperforms average- and max-pooling, but allows us to diagnostically visualize the importance of features at different positions and scales. Moreover, we show that adding extra supervision to the output at each scale is essential to achieving excellent performance when merging multi-scale features. We demonstrate the effectiveness of our model with extensive experiments on three challenging datasets, including PASCAL-Person-Part, PASCAL VOC 2012 and a subset of MS-COCO 2014.


The depth accuracy (Depth) is measured as the MSE between the rendered depth and the pseudo groundtruth depth obtained from a pre-trained face reconstruction model #REFR on the face mask area.

We also report the training time when utilizing 8 Tesla V100 GPUs.

Our GMNR achieves consistent improvement in performance on all metrics and different resolutions, compared to the baseline.

Furthermore, this improvement in performance over the baseline is achieved without any significant degradation in the training time and the inference speed of the model. #OTHEREFR .

The Frechet Inception Distance (FID) #OTHEREFR and Kernel Inception Distance (KID) [4] are computed between 50K generated images rendered at different random poses and (a) 50K real images for FFHQ; (b) 5, 065 real images with flip augmentation for AFHQv2-Cats.

The multi-view facial identity consistency (ID) is computed by first generating 1, 024 MPI representations and then employing the mean Arcface #OTHEREFR cosine similarity score between pairs of rendered views at random poses for the same face.

Similarly, the 3D pose accuracy (Pose) is computed by comparing the pose input used for rendering and the yaw, pitch and roll predicted by #OTHEREFR for the rendered image.

Generative Multiplane Neural Radiance For 3D-Aware Image Generation

Abstract: We present a method to efficiently generate 3D-aware high-resolution images that are view-consistent across multiple target views. The proposed multiplane neural radiance model, named GMNR, consists of a novel α-guided view-dependent representation (α-VdR) module for learning view-dependent information. The α-VdR module, faciliated by an α-guided pixel sampling technique, computes the view-dependent representation efficiently by learning viewing direction and position coefficients. Moreover, we propose a view-consistency loss to enforce photometric similarity across multiple views. The GMNR model can generate 3D-aware high-resolution images that are viewconsistent across multiple camera poses, while maintaining the computational efficiency in terms of both training and inference time. Experiments on three datasets demonstrate the effectiveness of the proposed modules, leading to favorable results in terms of both generation quality and inference time, compared to existing approaches. Our GMNR model generates 3D-aware images of 1024 × 1024 pixels with 17.6 FPS on a single V100.

Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image To Image Set

Abstract: Recently, deep learning based 3D face reconstruction methods have shown promising results in both quality and efficiency.However, training deep neural networks typically requires a large volume of data, whereas face images with ground-truth 3D face shapes are scarce. In this paper, we propose a novel deep 3D face reconstruction approach that 1) leverages a robust, hybrid loss function for weakly-supervised learning which takes into account both low-level and perception-level information for supervision, and 2) performs multi-image face reconstruction by exploiting complementary information from different images for shape aggregation. Our method is fast, accurate, and robust to occlusion and large pose. We provide comprehensive experiments on three datasets, systematically comparing our method with fifteen recent methods and demonstrating its state-of-the-art performance.


Solving multiple problems with a single model could create efficient systems and improve recognition for general AI, such as embodied agents #REFR , Narasimhan et al. [2020] .

[2019] introduced a dynamic fusion model with adaptive weights for better contextual modeling. DDS Liu et al.

[2022a] proposed a deep supervision framework that supervises all side outputs and is currently the state-of-the-art method for SBD.

In this paper, we specify multi-task learning (MTL) as an explicit joint modeling of two or more tasks like the method introduced in #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR .

While most of the models in computer vision are task-specific, there is great interest in joint modeling.

In MTL, it is common to use a multi-head architecture with a shared backbone for memory efficiency.

The backbone is aimed to learn a shared representation between the tasks, but often times this fails due to the backbone being designed for a single-task, leading to worse results #OTHEREFR , #OTHEREFR .

The works of explores novel mechanism for obtaining features by adding task-specific attention modules in the backbone.

Our work, however, explores the use of two-head architecture, where the auxiliary semantic boundary detection task is complementary to the main segmentation task.

In semantic segmentation, edges and boundaries have been used as auxiliary tasks. #OTHEREFR

Boosting Semantic Segmentation By Conditioning The Backbone With Semantic Boundaries

Abstract: In this paper, we present the Semantic Boundary Conditioned Backbone (SBCB) framework, a simple yet effective training framework that is model-agnostic and boosts segmentation performance, especially around the boundaries. Motivated by the recent development in improving semantic segmentation by incorporating boundaries as auxiliary tasks, we propose a multi-task framework that uses semantic boundary detection (SBD) as an auxiliary task. The SBCB framework utilizes the nature of the SBD task, which is complementary to semantic segmentation, to improve the backbone of the segmentation head. We apply an SBD head that exploits the multi-scale features from the backbone, where the model learns low-level features in the earlier stages, and high-level semantic understanding in the later stages. This head perfectly complements the common semantic segmentation architectures where the features from the later stages are used for classification. We can improve semantic segmentation models without additional parameters during inference by only conditioning the backbone. Through extensive evaluations, we show the effectiveness of the SBCB framework by improving various popular segmentation heads and backbones by 0.5% ∼ 3.0% IoU on the Cityscapes dataset and gains 1.6% ∼ 4.1% in boundary Fscores. We also apply this framework on customized backbones and the emerging vision transformer models and show the effectiveness of the SBCB framework.

Gibson Env: Real-World Perception For Embodied Agents

Abstract: Developing visual perception models for active agents and sensorimotor control are cumbersome to be done in the physical world, as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly. This has given rise to learning-in-simulation which consequently casts a question on whether the results transfer to real-world. In this paper, we are concerned with the problem of developing real-world perception for active agents, propose Gibson Virtual Environment for this purpose, and showcase sample perceptual tasks learned therein. Gibson is based on virtualizing real spaces, rather than using artificially designed ones, and currently includes over 1400 floor spaces from 572 full buildings. The main characteristics of Gibson are: I. being from the real-world and reflecting its semantic complexity, II. having an internal synthesis mechanism,"Goggles", enabling deploying the trained models in real-world without needing further domain adaptation, III. embodiment of agents and making them subject to constraints of physics and space.


Imitative Models #REFR learns a density model over expert data and then uses the density model as a reward in a goal conditioned open-loop planner; they demonstrate policies robust to noise in an autonomous driving simulator environment.

For instance, the L2 reward would be exactly 0 on the dataset of demonstrations.

Model-based Imitation Learning (MBIL): Using world models in Imitation Learning has been explored in other contexts #OTHEREFR .

IMPLANT #OTHEREFR uses GAIL to learn a policy and value function using online data, which then get used in a planner policy.

While IMPLANT shows that their planner is robust in noisy deployed environments for simple locomotion environments (Hopper, HalfCheetah, and Walker2d) our method differs mainly as we use BC instead of GAIL.

This difference allows POIR to be deployed offline which is not the case of IMPLANT.

A follow-up, Robust Imitative Planning (RIP) #OTHEREFR , uses an ensemble of networks to minimize uncertainty of trajectories in the planner, and then adapts the policy online by querying an expert policy #OTHEREFR .

A similar method, RMBIL #OTHEREFR also shows improvements over BC when deployed in environments that mismatch the expert distribution.

RMBIL is a purely offline method, that uses a neural ODE controller with a CVAE to encode states.

In contrast to Imitative Models, RIP, RMBIL, and other similar works #OTHEREFR , our method uses a simpler policy: model-predictive control #OTHEREFR with an open-loop trajectory planning algorithm #OTHEREFR using BC priors; notably we show that POIR can run in both the offline and online settings without querying an expert.

Generalization in IL/RL: Generalization in IL and RL seeks to improve performance on unseen interactions during deployment #OTHEREFR .

Get Back Here: Robust Imitation By Return-To-Distribution Planning

Abstract: We consider the Imitation Learning (IL) setup where expert data are not collected on the actual deployment environment but on a different version. To address the resulting distribution shift, we combine behavior cloning (BC) with a planner that is tasked to bring the agent back to states visited by the expert whenever the agent deviates from the demonstration distribution. The resulting algorithm, POIR, can be trained offline, and leverages online interactions to efficiently fine-tune its planner to improve performance over time. We test POIR on a variety of human-generated manipulation demonstrations in a realistic robotic manipulation simulator and show robustness of the learned policy to different initial state distributions and noisy dynamics.

Deep Imitative Models For Flexible Inference, Planning, And Control

Abstract: Imitation Learning (IL) is an appealing approach to learn desirable autonomous behavior. However, directing IL to achieve arbitrary goals is difficult. In contrast, planning-based algorithms use dynamics models and reward functions to achieve goals. Yet, reward functions that evoke desirable behavior are often difficult to specify. In this paper, we propose "Imitative Models" to combine the benefits of IL and goal-directed planning. Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals. We derive families of flexible goal objectives, including constrained goal regions, unconstrained goal sets, and energy-based goals. We show that our method can use these objectives to successfully direct behavior. Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task, and is efficiently learned from expert demonstrations without online data collection. We also show our approach is robust to poorly-specified goals, such as goals on the wrong side of the road.


However, these methods are often limited by a predefined set of recognized entities and specific data types, such as proportion-related statistics #REFR .

Nonetheless, these methods share some common issues.

First, re-arXiv:2304.14630v1 [cs.AI] 28 Apr 2023 trieving an appropriate visual design that accurately aligns with the data requires a large-scale and high-quality image corpus, which is challenging to construct.

Studies in this direction often create new datasets, resulting in various datasets (e.g., #OTHEREFR ) that do not meet each other's requirements.

Second, the retrieval process requires certain guidance, which can be difficult for novice users to manipulate.

With the increasing popularity of language models, numerous works (e.g., #OTHEREFR ) have explored the automatic generation of charts from text.

Additionally, editing the image to bind it with data is an error-prone task due to the trade-off between fitting the chart into the image and adapting the chart to the image.

To alleviate these limitations, we propose ChartSpark, a novel pictorial visualization authoring tool that embed semantic context into charts based on text-to-image generative model, while preserving important visual attributes such as trend.

We firstly extract the target visualized object through text analyzer.

The generation process will then be guided by the object and its corresponding description provided by the user.

As an initial step towards utilizing generation techniques for creating visualization charts, we conducted a preliminary study that examines the different display formats of pictorial visualization.

Let The Chart Spark: Embedding Semantic Context Into Chart With Text-To-Image Generative Model

Abstract: Pictorial visualization seamlessly integrates data and semantic context into visual representation, conveying complex information in a manner that is both engaging and informative. Extensive studies have been devoted to developing authoring tools to simplify the creation of pictorial visualizations. However, mainstream works mostly follow a retrieving-and-editing pipeline that heavily relies on retrieved visual elements from a dedicated corpus, which often compromise the data integrity. Text-guided generation methods are emerging, but may have limited applicability due to its predefined recognized entities. In this work, we propose ChartSpark, a novel system that embeds semantic context into chart based on text-to-image generative model. ChartSpark generates pictorial visualizations conditioned on both semantic context conveyed in textual inputs and data information embedded in plain charts. The method is generic for both foreground and background pictorial generation, satisfying the design practices identified from an empirical research into existing pictorial visualizations. We further develop an interactive visual interface that integrates a text analyzer, editing module, and evaluation module to enable users to generate, modify, and assess pictorial visualizations. We experimentally demonstrate the usability of our tool, and conclude with a discussion of the potential of using text-to-image generative model combined with interactive interface for visualization design.

Text-To-Viz: Automatic Generation Of Infographics From Proportion-Related Natural Language Statements

Abstract: Combining data content with visual embellishments, infographics can effectively deliver messages in an engaging and memorable manner. Various authoring tools have been proposed to facilitate the creation of infographics. However, creating a professional infographic with these authoring tools is still not an easy task, requiring much time and design expertise. Therefore, these tools are generally not attractive to casual users, who are either unwilling to take time to learn the tools or lacking in proper design expertise to create a professional infographic. In this paper, we explore an alternative approach: to automatically generate infographics from natural language statements. We first conducted a preliminary study to explore the design space of infographics. Based on the preliminary study, we built a proof-of-concept system that automatically converts statements about simple proportion-related statistics to a set of infographics with pre-designed styles. Finally, we demonstrated the usability and usefulness of the system through sample results, exhibits, and expert reviews.

recognized entities
proportion-related statistics


Real images are from CelebA-HQ dataset #REFR and translated into two styles of images, "Wall painting" and "Tolkien elf". The yellow boxes show the key observation areas.

From the quality of the gener-ated images, the results of NADA have more incorrect features and noise, such as green mussy noise on hairs (Tolkien elf), ruined noses (Werewolf) and unshaped necks (Pointillism), while the results of IPL are more clear and correct.

From the mode collapse perspective of the generated images, NADA is prone to collapse to some similar facial features for different images, such as depressed emotions (Pixar character), folded ears (Cartoon) and blue noses (Cu- Figure 5 . Real-world image translation comparison results.

Baselines are NADA #OTHEREFR , Diff-CLIP #OTHEREFR and Diff-CLIP+ (an improved version of Diff-CLIP).

Recon1, Recon2 and Recon3 refer to inversion results via Restyle #OTHEREFR , DDIM and diffusion autoencoders, respectively.

GAN-IPL and Diff-IPL denote integrating IPL with NADA and Diff-CLIP+, respectively.

bism), while IPL presents consistently higher diversity and solve the mode collapse issue well.

Our advantages mainly come from the fact that the latent mapper preserves sufficient image-specific and target-domain friendly features from the source-domain images.

The produced prompt vectors provide more precise and diversified adaptation directions for the target-domain generator adaptation.

To quantify the performance improvement of IPL compared to NADA #OTHEREFR , IS, SCS, ID and SIFID are evaluated. As reported in Tab.

Zero-Shot Generative Model Adaptation Via Image-Specific Prompt Learning

Abstract: Figure 1. The mode collapse issue. For NADA [21] and our method, the same generator pre-trained on the source domain of "Photo" is adapted to the unseen target domains of "Disney", "Anime painting", "Wall painting" and "Ukiyo-e" only with the domain labels. The images above the dotted line are some examples from the internet. The generated images of NADA exhibit some similar unseen patterns (yellow box areas) which are undesired in terms of quality and diversity. This issue is largely addressed by our method.

Deep Learning Face Attributes In The Wild

Abstract: Predicting face attributes in the wild is challenging due to complex face variations. We propose a novel deep learning framework for attribute prediction in the wild. It cascades two CNNs, LNet and ANet, which are fine-tuned jointly with attribute tags, but pre-trained differently. LNet is pre-trained by massive general object categories for face localization, while ANet is pre-trained by massive face identities for attribute prediction. This framework not only outperforms the state-of-the-art with a large margin, but also reveals valuable facts on learning face representation. (1) It shows how the performances of face localization (LNet) and attribute prediction (ANet) can be improved by different pre-training strategies. (2) It reveals that although the filters of LNet are fine-tuned only with image-level attribute tags, their response maps over entire images have strong indication of face locations. This fact enables training LNet for face localization with only image-level annotations, but without face bounding boxes or landmarks, which are required by all attribute recognition works. (3) It also demonstrates that the high-level hidden neurons of ANet automatically discover semantic concepts after pre-training with massive face identities, and such concepts are significantly enriched after fine-tuning with attribute tags. Each attribute can be well explained with a sparse linear combination of these concepts.


Meanwhile, Human-in-the-loop (HITL) methods such as #REFR also attempt to utilize constraints through human interactions.

First, many constraint-based ML (CML) models are mostly designed for specific models (e.g., #OTHEREFR ).

They have shown high accuracy in the corresponding models, but it is difficult to use diverse expert inputs and they may not be robust to instances that are not known from the training data.

Expectation regularization (ER) #OTHEREFR takes the regularization function as expert input and could be applied to most supervised models that use gradient descent updates.

Similarly, weak supervision (WS) models like #OTHEREFR take supervision functions, which explain a law in physics and show performance gains in closeness.

In contrast, the ER and WS models are not able to leverage how reliable the predictions of new inputs are for the learned model.

Nevertheless, they primarily focus on collecting high-quality training data, reusing models, and managing the existing models, which still much depend on the prediction of the learned ML models.

For Out-of-distribution (OOD) models like DRC #OTHEREFR , they take the simple statistical base distribution as expert input and measure how training data is different from the new instances.

However, there are no incorporating methods to correct the predictions of the existing model.

Transfer Learning (TL) methods are also limited in modifying the predictions for exploiting the expert input.

Many Data Imputation (DI) techniques using MICE (e.g., #OTHEREFR ) can estimate missing judgment variables from the known data, but they have not been applied to correct the predictions of the current models.

Incorporating Experts' Judgment Into Machine Learning Models

Abstract: Machine learning (ML) models have been quite successful in predicting outcomes in many applications. However, in some cases, domain experts might have a judgment about the expected outcome that might conflict with the prediction of ML models. One main reason for this is that the training data might not be totally representative of the population. In this paper, we present a novel framework that aims at leveraging experts' judgment to mitigate the conflict. The underlying idea behind our framework is that we first determine, using a generative adversarial network, the degree of representation of an unlabeled data point in the training data. Then, based on such degree, we correct the machine learning model's prediction by incorporating the experts' judgment into it, where the higher that aforementioned degree of representation, the less the weight we put on the expert intuition that we add to our corrected output, and vice-versa. We perform multiple numerical experiments on synthetic data as well as two real-world case studies (one from the IT services industry and the other from the financial industry). All results show the effectiveness of our framework; it yields much higher closeness to the experts' judgment with minimal sacrifice in the prediction accuracy, when compared to multiple baseline methods. We also develop a new evaluation metric that combines prediction accuracy with the closeness to experts' judgment. Our framework yields statistically significant results when evaluated on that metric.

Towards Unified Data And Lifecycle Management For Deep Learning

Abstract: Deep learning has improved state-of-the-art results in many important fields, and has been the subject of much research in recent years, leading to the development of several systems for facilitating deep learning. Current systems, however, mainly focus on model building and training phases, while the issues of data management, model sharing, and lifecycle management are largely ignored. Deep learning modeling lifecycle generates a rich set of data artifacts, e.g., learned parameters and training logs, and it comprises of several frequently conducted tasks, e.g., to understand the model behaviors and to try out new models. Dealing with such artifacts and tasks is cumbersome and largely left to the users. This paper describes our vision and implementation of a data and lifecycle management system for deep learning. First, we generalize model exploration and model enumeration queries from commonly conducted tasks by deep learning modelers, and propose a high-level domain specific language (DSL), inspired by SQL, to raise the abstraction level and thereby accelerate the modeling process. To manage the variety of data artifacts, especially the large amount of checkpointed float parameters, we design a novel model versioning system (dlv), and a read-optimized parameter archival storage system (PAS) that minimizes storage footprint and accelerates query workloads with minimal loss of accuracy. PAS archives versioned models using deltas in a multi-resolution fashion by separately storing the less significant bits, and features a novel progressive query (inference) evaluation algorithm. Third, we develop e cient algorithms for archiving versioned models using deltas under co-retrieval constraints. We conduct extensive experiments over several real datasets from computer vision domain to show the e ciency of the proposed techniques.


Instead, we follow the interactive and hybrid cryptographic protocol #REFR and construct our Primerbase by using GC for non-polynomial operations, FHE for polynomial operations, and SS for secure communication between multiple parties.

In this work, we use this interactive hybrid method to construct Primer-base, which is a starting point for our optimization techniques. Motivation.

As Figure 2 shows, prior works like THE-X #OTHEREFR using only FHE for private inference suffer from low accuracy and enormous online latency due to polynomial approximation and expensive FHE operations.

We use prior GC-based work #OTHEREFR to implement a GCFormer (we convert the Transformer model into a circuit based on binary gates so that GC #OTHEREFR can implement it).

GCFormer achieves an accurate performance, i.e., 85.1% accuracy, but it takes a larger latency than THE-X.

Thus, the FHE-based method or GCbased method cannot achieve a low-latency and accurate private Transformer inference.

Primer-base significantly improves the accuracy of THE-X, e.g., 7.3% accuracy increase, and reduces the latency of GCFormer. However, Primer-base still suffers from enormous online latency.

This motivates us to propose techniques like the FHGS protocol, denoted by Primer-F, to offload the online computation to the offline phase where computations can be computed before inference.

Considering Primer-F still has a large total latency, we have motivations to propose techniques including computation merge, i.e., combined FHGS, and tokens-first ciphertext packing techniques.

More details about Primer and related techniques are introduced in the following section III.

Primer: Fast Private Transformer Inference On Encrypted Data

Abstract: It is increasingly important to enable privacypreserving inference for cloud services based on Transformers. Post-quantum cryptographic techniques, e.g., fully homomorphic encryption (FHE), and multi-party computation (MPC), are popular methods to support private Transformer inference. However, existing works still suffer from prohibitively computational and communicational overhead. In this work, we present, Primer, to enable a fast and accurate Transformer over encrypted data for natural language processing tasks. In particular, Primer is constructed by a hybrid cryptographic protocol optimized for attention-based Transformer models, as well as techniques including computation merge and tokens-first ciphertext packing. Comprehensive experiments on encrypted language modeling show that Primer achieves state-of-the-art accuracy and reduces the inference latency by 90.6% ∼ 97.5% over previous methods.

Gazelle: A Low Latency Framework For Secure Neural Network Inference

Abstract: The growing popularity of cloud-based machine learning raises a natural question about the privacy guarantees that can be provided in such a setting. Our work tackles this problem in the context where a client wishes to classify private images using a convolutional neural network (CNN) trained by a server. Our goal is to build efficient protocols whereby the client can acquire the classification result without revealing their input to the server, while guaranteeing the privacy of the server's neural network. To this end, we design Gazelle, a scalable and low-latency system for secure neural network inference, using an intricate combination of homomorphic encryption and traditional two-party computation techniques (such as garbled circuits). Gazelle makes three contributions. First, we design the Gazelle homomorphic encryption library which provides fast algorithms for basic homomorphic operations such as SIMD (single instruction multiple data) addition, SIMD multiplication and ciphertext permutation. Second, we implement the Gazelle homomorphic linear algebra kernels which map neural network layers to optimized homomorphic matrix-vector multiplication and convolution routines. Third, we design optimized encryption switching protocols which seamlessly convert between homomorphic and garbled circuit encodings to enable implementation of complete neural network inference. We evaluate our protocols on benchmark neural networks trained on the MNIST and CIFAR-10 datasets and show that Gazelle outperforms the best existing systems such as MiniONN (ACM CCS 2017) by 20 times and Chameleon (Crypto Eprint 2017/1164) by 30 times in online runtime. Similarly when compared with fully homomorphic approaches like CryptoNets (ICML 2016) we demonstrate three orders of magnitude faster online run-time.

secure communication
hybrid cryptographic protocol


First, if the number of views is fixed as n and the view sampling strategy is constant for all objects in a dataset, we initialize a n dimension learnable parameter. n = 12 views for ModelNet40 #REFR .

Then we extract the feature of the source view as value and synthesize the target view.

For the self-attention blocks following the cross-attention blocks, the value, key and query are the same (three black arrows shown in Fig. 4 upper part).

The numbers of cross-and self-attention blocks are the same by default. Fig.

4 (upper part) simplifies them as one cross-attention block and one self-attention block. Pose Embedding.

As described above, both key and query input of cross-attention blocks in VSA decoder are pose embeddings. Here we provide two methods to encode poses.

The shape of this learnable parameter is (n, patches, dim), where patches is the number of patches for encoder or decoder input, and dim is the dimension of extracted patch features. As shown in Fig.

4 lower part, the gray grid is the learnable parameter which has n lines (from left to right, n = 12 in this example). On the left side of Fig.

4 lower part, when the input is a single source view, the orange vector and blue vector are corresponding to the source view and target poses. On the right side of Fig.

4 lower part, when the inputs are two source views, the yellow and orange vectors are corresponding to two source poses.

The key in cross-attention blocks becomes the concatenation of two source poses embeddings (red). The blue vector is corresponding to the target pose.

Self-Supervised Learning By View Synthesis

Abstract: We present view-synthesis autoencoders (VSA) in this paper, which is a self-supervised learning framework designed for vision transformers. Different from traditional 2D pretraining methods, VSA can be pre-trained with multi-view data. In each iteration, the input to VSA is one view (or multiple views) of a 3D object and the output is a synthesized image in another target pose. The decoder of VSA has several cross-attention blocks, which use the source view as value, source pose as key, and target pose as query. They achieve cross-attention to synthesize the target view. This simple approach realizes large-angle view synthesis and learns spatial invariant representation, where the latter is decent initialization for transformers on downstream tasks, such as 3D classification on ModelNet40, ShapeNet Core55, and ScanObjectNN. VSA outperforms existing methods significantly for linear probing and is competitive for fine-tuning. The code will be made publicly available.

3D Shapenets: A Deep Representation For Volumetric Shapes

Abstract: 3D shape is a crucial but heavily underutilized cue in today's computer vision systems, mostly due to the lack of a good generic shape representation. With the recent availability of inexpensive 2.5D depth sensors (e.g. Microsoft Kinect), it is becoming increasingly important to have a powerful 3D shape representation in the loop. Apart from category recognition, recovering full 3D shapes from view-based 2.5D depth maps is also a critical part of visual understanding. To this end, we propose to represent a geometric 3D shape as a probability distribution of binary variables on a 3D voxel grid, using a Convolutional Deep Belief Network. Our model, 3D ShapeNets, learns the distribution of complex 3D shapes across different object categories and arbitrary poses from raw CAD data, and discovers hierarchical compositional part representation automatically. It naturally supports joint object recognition and shape completion from 2.5D depth maps, and it enables active object recognition through view planning. To train our 3D deep learning model, we construct ModelNet - a large-scale 3D CAD model dataset. Extensive experiments show that our 3D deep representation enables significant performance improvement over the-state-of-the-arts in a variety of tasks.


In order to decode the generated voxel-grid of features into density and radiance fields, we use a NeRF-like #REFR MLP (Multi-layer perceptron).

The MLP contains 4 layers of 256 hidden units with a skip-connection on the 3rd hidden layer.

The skip connection concatenates the input features with the intermediate hidden layer features.

Similar to NeRF, and for the reasons described in Zhang et al. #OTHEREFR , we Figure I .

Architecture of the RenderMLP used for decoding the features of the generated voxel grids into density and radiance fields.

also input the view-directions at a latter layer in the MLP.

Holodiffusion: Training A 3D Diffusion Model Using 2D Images

Abstract: Figure 1. We present HOLODIFFUSION as the first 3D-aware generative diffusion model that produces 3D-consistent images and is trained with only posed image supervision. Here we show a few different samples generated from models trained on different classes of the CO3D dataset [50].

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.


Experimental Environment And Parameter Settings

We trained all the models with AdamW as the optimizer, with the default parameters as described in #REFR and mini batches with size of 32. It should be noted that ConvNet Schirrmeister et al.

All the experiments were conducted under the Pytorch framework on a workstation with Intel(R) Xeon(R) Gold 5117 CPUs @ 2.00 GHz and Nvidia Tesla V100 GPUs.

(2017) has a shallow version and a deep version for EEG decoding, both EEGNet #OTHEREFR and ConvNet Schirrmeister et al.

(2017) mentioned that shallow-ConvNet has more advantages than deep-ConvNet, thus we only selected shallow-ConvNet (Abbreviated as ConvNet below) in the experiment.

Lmda-Net:A Lightweight Multi-Dimensional Attention Network For General Eeg-Based Brain-Computer Interface Paradigms And Interpretability

Abstract: Electroencephalography (EEG)-based brain-computer interfaces (BCIs) pose a challenge for decoding due to their low spatial resolution and signal-to-noise ratio. Typically, EEG-based recognition of activities and states involves the use of prior neuroscience knowledge to generate quantitative EEG features, which may limit BCI performance. Although neural network-based methods can effectively extract features, they often encounter issues such as poor generalization across datasets, high predicting volatility, and low model interpretability. To address these limitations, we propose a novel lightweight multi-dimensional attention network, called LMDA-Net. By incorporating two novel attention modules designed specifically for EEG signals, the channel attention module and the depth attention module, LMDA-Net is able to effectively integrate features from multiple dimensions, resulting in improved classification performance across various BCI tasks. LMDA-Net was evaluated on four high-impact public datasets, including motor imagery (MI) and P300-Speller paradigms, and was compared with other representative models. The experimental results demonstrate that LMDA-Net outperforms other representative methods in terms of classification accuracy and predicting volatility, achieving the highest accuracy in all datasets within 300 training epochs. Ablation experiments further confirm the effectiveness of the channel attention module and the depth attention module. To facilitate an in-depth understanding of the features extracted by LMDA-Net, we propose class-specific neural network feature interpretability algorithms that are suitable for event-related potentials (ERPs) and event-related desynchronization/synchronization (ERD/ERS). By mapping the output of the specific layer of MDA-Net to the time or spatial domain through class activation maps, the resulting feature visualizations can provide interpretable analysis and establish connections with EEG time-spatial analysis in neuroscience. In summary, LMDA-Net shows great potential as a general online decoding model for various EEG tasks.

Fixing Weight Decay Regularization In Adam

Abstract: We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor. We propose a simple way to resolve this issue by decoupling weight decay and the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence. Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-of-the-art results on CIFAR-10 and ImageNet32x32. Our source code is available at https://github.com/loshchil/AdamW-and-SGDW