In particular, we show thatÕ n min ε, ε 2 andÕ log n min ε, ε #REFR bits of (per-client) communication are sufficient to achieve the order-optimal error under (ε, δ)-privacy for mean and frequency estimation respectively, where n is the number of participating clients.

We consider distributed mean and frequency estimation as canonical building blocks for FL and FA.

We consider both the central DP and the multi-message shuffling models.

We characterize the order-optimal privacy-accuracy-communication trade-off for distributed mean estimation and provide an achievable scheme for frequency estimation in the central DP model.

Our results reveal that privacy and communication efficiency can be achieved simultaneously with no additional penalty on accuracy.

Without compression, each client needs O(d) bits and log d bits for the mean and frequency estimation problems respectively (where d is the number of trainable parameters in FL or the domain size in FA), which means that we can get significant savings in the regime nε 2 = o(d) (assuming ε = O(1)).

We note that this is often the relevant regime not only for cross-silo but also for cross-device FL/FA.

For instance, in practical FL, d usually ranges from 10 6 -10 9 , and n, the perepoch sample size, is usually much smaller (e.g., of the order of 10 3 -10 5 ). For distributed mean estimation,

Privacy Amplification Via Compression: Achieving The Optimal Privacy-Accuracy-Communication Trade-Off In Distributed Mean Estimation

Abstract: Privacy and communication constraints are two major bottlenecks in federated learning (FL) and analytics (FA). We study the optimal accuracy of mean and frequency estimation (canonical models for FL and FA respectively) under joint communication and (ε, δ)-differential privacy (DP) constraints. We consider both the central and the multi-message shuffling DP models. We show that in order to achieve the optimal 2 error under (ε, δ)-DP, it is sufficient for each client to send Θ n min ε, ε 2 bits for FL and Θ log n min ε, ε 2 bits for FA to the server, where n is the number of participating clients. Without compression, each client needs O(d) bits and O (log d) bits for the mean and frequency estimation problems respectively (where d corresponds to the number of trainable parameters in FL or the domain size in FA), meaning that we can get significant savings in the regime n min ε, ε 2 = o(d), which is often the relevant regime in practice. We propose two different ways to leverage compression for privacy amplification and achieve the optimal privacy-communication-accuracy trade-off. In both cases, each client communicates only partial information about its sample and we show that privacy is amplified by randomly selecting the part contributed by each client. In the first method, the random selection is revealed to the server, which results in a central DP guarantee with optimal privacy-communication-accuracy trade-off. In the second method, the random data parts at each client are privatized locally and anonymized by a secure shuffler, eliminating the need for a trusted server. This results in a multi-message shuffling scheme with the same optimal trade-off. As a result, our paper establishes the optimal three-way trade-off between privacy, communication, and accuracy for both the central DP and multi-message shuffling frameworks. 1

Inference Under Information Constraints Ii: Communication Constraints And Shared Randomness

Abstract: A central server needs to perform statistical inference based on samples that are distributed over multiple users who can each send a message of limited length to the center. We study problems of distribution learning and identity testing in this distributed inference setting and examine the role of shared randomness as a resource. We propose a general-purpose simulate-and-infer strategy that uses only private-coin communication protocols and is sample-optimal for distribution learning. This general strategy turns out to be sample-optimal even for distribution testing among private-coin protocols. Interestingly, we propose a public-coin protocol that outperforms simulate-and-infer for distribution testing and is, in fact, sample-optimal. Underlying our public-coin protocol is a random hash that when applied to the samples minimally contracts the chi-squared distance of their distribution to the uniform distribution.

frequency estimation
participating clients


Second, and consistent with prior work, we find that an even split of parameters between the encoder and decoder (6 encoder layers and 6 decoder layers, i.e., Transformer base) yields the best translation quality for the Transformer #REFR , whereas no encoder produces the best results for S4.

Encoder Scaling To explore the effect of parameter allocation on performance, we compare the translation quality of different encoder-decoder configurations with the same total number of parameters (roughly 65M).

In Figure 2a , the x axis represents the ratio of encoder layers to the total number of layers (encoder + decoder).

Starting with a decoder-only model (ratio = 0), we gradually increase the number of encoder layers, and end with a model containing only a single decoder layer.

Two results stand out: first, there is a wide gap between the best S4 and Transformer models: 20.7 and 26.4 BLEU, respectively.

Based on this finding, we focus on the S4 decoder-only variant for the next experiments.

Number of S4 Blocks per Layer Prior research set the number of S4 blocks, B, to 2 #OTHEREFR .

We found that increasing B is beneficial as S4 blocks are responsible for capturing dependencies between tokens.

In Table 1 we vary B while keeping the parameter count roughly constant.

Increasing B leads to noticeable quality improvements until B = 10.

State Spaces Aren'T Enough: Machine Translation Needs Attention

Abstract: Structured State Spaces for Sequences (S4) is a recently proposed sequence model with successful applications in various tasks, e.g. vision, language modeling, and audio. Thanks to its mathematical formulation, it compresses its input to a single hidden state, and is able to capture long range dependencies while avoiding the need for an attention mechanism. In this work, we apply S4 to Machine Translation (MT), and evaluate several encoder-decoder variants on WMT'14 and WMT'16. In contrast with the success in language modeling, we find that S4 lags behind the Transformer by approximately 4 BLEU points, and that it counter-intuitively struggles with long sentences. Finally, we show that this gap is caused by S4's inability to summarize the full source sentence in a single hidden state, and show that we can close the gap by introducing an attention mechanism.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


We model the traveling time on arc a as a linear function τ a (ξ a ) such that the traveling time increases with the increase of the traffic flow ξ a on arc a ∈ A. ; see #REFR .

The traffic network is modeled by a connected directional graph R = (V, A), where V and A denote the sets of vertexes and arcs.

where ξ 0 a , R a are the parameters related to the shape of the traveling time function.

Let C ∈ V denote the set of nodes where the charging stations are located in.

We model the nodes of charging stations as "virtual arcs" following #OTHEREFR (see Fig. 2 ).

At a node i ∈ C, the EV driver can choose the arc l c i for charging or the arc l n i for skipping charging.

Optimal Vehicle Charging In Bilevel Power-Traffic Networks Via Charging Demand Function

Abstract: Electric vehicle (EV) charging couples the operation of power and traffic networks. Specifically, the power network determines the charging price at various locations, while EVs on the traffic network optimize the charging power given the price, acting as price-takers. We model such decision-making processes by a bilevel program, with the power network at the upper-level and the traffic network at the lower-level. However, since the two networks are managed by separate entities and the charging expense term, calculated as the product of charging price and charging demand, is nonlinear. Solving the bilevel program is nontrivial. To overcome these challenges, we derive the charging demand function using multiparametric programming theory. This function establishes a piecewise linear relationship between the charging price and the optimal charging power, enabling the power network operator to manage EV charging power independently while accounting for the coupling between the two networks. With the derived function, we are also able to replace the nonlinear charging expense term with a piecewise quadratic one, thus guaranteeing solution optimality. Our numerical studies demonstrate that different traffic demands can have an impact on charging patterns and the power network can effectively incentivize charging at low-price nodes through price setting.

Optimal Pricing To Manage Electric Vehicles In Coupled Power And Transportation Networks

Abstract: We study the system-level effects of the introduction of large populations of Electric Vehicles (EVs) on the power and transportation networks. We assume that each EV owner solves a decision problem to pick a cost-minimizing charge and travel plan. This individual decision takes into account traffic congestion in the transportation network, affecting travel times, as well as congestion in the power grid, resulting in spatial variations in electricity prices for battery charging. We show that this decision problem is equivalent to finding the shortest path on an “extended” transportation graph, with virtual arcs that represent charging options. Using this extended graph, we study the collective effects of a large number of EV owners individually solving this path planning problem. We propose a scheme in which independent power and transportation system operators can collaborate to manage each network towards a socially optimum operating point while keeping the operational data of each system private. We further study the optimal reserve capacity requirements for pricing in the absence of such collaboration. We showcase numerically that a lack of attention to interdependencies between the two infrastructures can have adverse operational effects.


Note that the signal amplitude A in (6) is not the exact amplitude of the electric or magnetic field and the transform between the signal amplitude and electric field amplitude can be found in #REFR .

Finally, the complex amplitude of the EM signal reflected by the -th element on the RIS at an observation point r can be written as

where Γ = |Γ |e jφ is the complex reflection coefficient of the RIS element , φ being the phaseshift imposed by the RIS element to the EM signal.

The reflection angle (θ , ϕ ) is determined by the spatial vector (r − r ) in the local coordinates of the RIS.

In most cases, the RIS is utilized to enhance signal coverage or tackle the blockage and it is usually deployed far from the BS, hence the BS can be regarded as a point source and (6) can be applied.

If a UE equipped with an isotropic antenna is placed at the observation point, the received power at the UE reflected by a RIS with N elements can then be expressed as

where S is the effective aperture of the UE antenna. Obviously, the maximum received power can be obtained if

φ inc + k|r − r | + φ = φ c for = 1, 2, • • • , N ,

where φ c is a constant phase which can be simply set to be zero.

How Practical Phase-Shift Errors Affect Beamforming Of Reconfigurable Intelligent Surface?

Abstract: Reconfigurable intelligent surface (RIS) is a new technique that is able to manipulate the wireless environment smartly and has been exploited for assisting the wireless communications, especially at high frequency band. However, it suffers from hardware impairments (HWIs) in practical designs, which inevitably degrades its performance and thus limits its full potential. To address this practical issue, we first propose a new RIS reflection model involving phase-shift errors, which is then verified by the measurement results from field trials. With this beamforming model, various phase-shift errors caused by different HWIs can be analyzed. The phase-shift errors are classified into three categories: (1) globally independent and identically distributed errors, (2) grouped independent and identically distributed errors and (3) grouped fixed errors. The impact of typical HWIs, including frequency mismatch, PIN diode failures and panel deformation, on RIS beamforming ability are studied with the theoretical model and are compared with numerical results. The impact of frequency mismatch are discussed separately for narrow-band and wide-band beamforming. Finally, useful insights and guidelines on the RIS design and its deployment are highlighted for practical wireless sytsems.

Wireless Communications With Reconfigurable Intelligent Surface: Path Loss Modeling And Experimental Measurement

Abstract: Reconfigurable intelligent surfaces (RISs) comprised of tunable unit cells have recently drawn significant attentions due to their superior capability in manipulating electromagnetic waves. In particular, RIS-assisted wireless communications have the great potential to achieve significant performance improvement and coverage enhancement in a cost-effective and energy-efficient manner, by properly programming the reflection coefficients of the unit cells of RISs. In this paper, the free-space path loss models of RIS-assisted wireless communications are developed for different scenarios by studying the physics and electromagnetic nature of RISs. The proposed models, which are first validated through extensive simulation results, reveal the relationships between the free-space path loss of RIS-assisted wireless communications and the distances from the transmitter/receiver to the RIS, the size of the RIS, the near-field/far-field effects of the RIS, and the radiation patterns of antennas and unit cells. In addition, three fabricated RISs (metasurfaces) are utilized to further corroborate theoretical findings through experimental measurements conducted in a microwave anechoic chamber. The measurement results match well with the modeling results, thus validating the proposed free-space path loss models for RIS, which may pave the way for further theoretical studies and practical applications in this field.


This confirms the observation of #REFR that 71% of the validation set samples were re-selected in their setting.

Such information may be helpful for identifying and factoring out group-specific idiosyncrasies.

For example, worker ABC may always click near the centre of an image; we may then decide not to use her clicks as a reliable estimate of object locations.

Or we may find that the HIT DEF was done in such a rush; we would then reduce the weight for the set of annotations belonging to DEF.

Statistics. There are 1,281,167 ImageNet1K training images I imagenet . There were two annotation rounds.

In the first round, human intelligence tasks (HITs) containing all 1,281,167 original images are shown to the annotators. They have re-selected 71.8% of them.

The remaining 28.2% of I imagenet are re-packaged into a second batch of HITs and presented to the annotators.

They have additionally re-selected 14.9% of I imagenet , resulting in the final 1,110,786 (86.7%) Ima-geNet1K training images that are re-selected.

Those selected images now come with rich annotation byproducts, such as the time-series of mouse traces and clicks.

However, annotation byproducts are available even for images that are not finally selected; they are recorded even for images that annotators cancel the selection or simply hover the cursor over.

As a result, 1,272,225 (99.3%) of the ImageNet1K training set have any form of annotation byproduct available.

Neglected Free Lunch -Learning Image Classifiers Using Annotation Byproducts

Abstract: Supervised learning of image classifiers distills human knowledge into a parametric model f θ through pairs of images and corresponding labels {(X i , Y i)} N i=1. We argue that this simple and widely used representation of human knowledge neglects rich auxiliary information from the annotation procedure, such as the time-series of mouse traces and clicks left after image selection. Our insight is that such annotation byproducts Z provide approximate human attention that weakly guides the model to focus on the foreground cues, reducing spurious correlations and discouraging shortcut learning. To verify this, we create ImageNet-AB and COCO-AB. They are ImageNet and COCO training sets enriched with sample-wise annotation byproducts, collected by replicating the respective original annotation tasks. We refer to the new paradigm of training models with annotation byproducts as learning using annotation byproducts (LUAB). We show that a simple multitask loss for regressing Z together with Y already improves the generalisability and robustness of the learned models. Compared to the original supervised learning, LUAB does not require extra annotation costs. ImageNet-AB and COCO-AB are at github.com/naverai/NeglectedFreeLunch.

Do Imagenet Classifiers Generalize To Imagenet?

Abstract: We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly "harder" images than those found in the original test sets.


We use the FlyingThings3D from Scene Flow Datasets #REFR , which uses synthetic data generation to obtain all-in-focus RGB images and disparity maps.

We use the cropped 278 × 278 all-in-focus images from #OTHEREFR .

In total, we use 5077 training patches and 419 test patches.

Both the optical layer and reconstruction networks are differentiable, so the phase mask sequence and neural network can be optimized through back-propagation. Each part is implemented in PyTorch.

During training, we use the Adam #OTHEREFR optimizer with parameters β 1 = 0.99 and β 2 = 0.999.

The learning rate for the phase masks is 10 −8 and for the reconstruction network it is 10 −4 , and the batch size was 32.

Tidy-Psfs: Computational Imaging With Time-Averaged Dynamic Point-Spread-Functions

Abstract: Point-spread-function (PSF) engineering is a powerful computational imaging techniques wherein a custom phase mask is integrated into an optical system to encode additional information into captured images. Used in combination with deep learning, such systems now offer stateof-the-art performance at monocular depth estimation, extended depth-of-field imaging, lensless imaging, and other tasks. Inspired by recent advances in spatial light modulator (SLM) technology, this paper answers a natural question: Can one encode additional information and achieve superior performance by changing a phase mask dynamically over time? We first prove that the set of PSFs described by static phase masks is non-convex and that, as a result, time-averaged PSFs generated by dynamic phase masks are fundamentally more expressive. We then demonstrate, in simulation, that time-averaged dynamic (TiDy) phase masks can offer substantially improved monocular depth estimation and extended depth-of-field imaging performance.

A Large Dataset To Train Convolutional Networks For Disparity, Optical Flow, And Scene Flow Estimation

Abstract: Recent work has shown that optical flow estimation can be formulated as a supervised learning task and can be successfully solved with convolutional networks. Training of the so-called FlowNet was enabled by a large synthetically generated dataset. The present paper extends the concept of optical flow estimation via convolutional networks to disparity and scene flow estimation. To this end, we propose three synthetic stereo video datasets with sufficient realism, variation, and size to successfully train large networks. Our datasets are the first large-scale datasets to enable training and evaluation of scene flow methods. Besides the datasets, we present a convolutional network for real-time disparity estimation that provides state-of-the-art results. By combining a flow and disparity estimation network and training it jointly, we demonstrate the first scene flow estimation with a convolutional network.


We illustrate the effectiveness of the resulting model for instruction generation and also traditional dense captioning on ActivityNet Captions #REFR .

In other words, instruction generation requires at least a certain level of commonsense knowledge.

Further, we propose a video captioning model that incorporates both explicit (from COMET #OTHEREFR ) as well as implicit commonsense knowledge in the form of a language model (GPT3 #OTHEREFR ) and visio-lingual zero-shot object knowledge (through CLIP #OTHEREFR ).

Our video captioning model extracts a snippet of video, detects objects and actions in it, and uses encoding of the snippet along with object/actions, CLIP detected objects, and commonsense inferences (that are prompted with the previous sentence) to produce a new sentence.

This procedure is repeated recursively until the desired number of sentences are decoded and/or the end of the paragraph is reached.

Our focus, in this work, is on the Transformer-based language multi-sentence captioning decoder, that can be combined with any underlying dense or paragraph captioning framework; for convenience, we build on top of recent #OTHEREFR .

Our contributions are three-fold: (1) We propose a new task of video-based instruction generation, which we believe requires a certain amount of commonsense knowledge; #OTHEREFR We propose a new video multisentence captioning model that effectively leverages implicit and explicit commonsense knowledge (prompted with prior sentences) to improve current sentence prediction.

#OTHEREFR We explicitly analyze contributions these sources or knowledge make towards improved captioning quality and also illustrate our model's performance on more traditional dense video captioning tasks, showing state-of-the-art performance (without large pre-training).

Overall, ours is one of the first papers to explore the use of commonsense knowledge for video captioning tasks.

Implicit And Explicit Commonsense For Multi-Sentence Video Captioning

Abstract: Step into the tub.

Dense-Captioning Events In Videos

Abstract: Most natural videos contain numerous events. For example, in a video of a "man playing a piano", the video might also contain "another man dancing" or "a crowd clapping". We introduce the task of dense-captioning events, which involves both detecting and describing events in a video. We propose a new model that is able to identify all events in a single pass of the video while simultaneously describing the detected events with natural language. Our model introduces a variant of an existing proposal module that is designed to capture both short as well as long events that span minutes. To capture the dependencies between the events in a video, our model introduces a new captioning module that uses contextual information from past and future events to jointly describe all events. We also introduce ActivityNet Captions, a large-scale benchmark for dense-captioning events. ActivityNet Captions contains 20k videos amounting to 849 video hours with 100k total descriptions, each with it's unique start and end time. Finally, we report performances of our model for dense-captioning events, video retrieval and localization.

traditional dense captioning
ActivityNet Captions


This implies that (i) they are equivariant under spacetime isometries, such as translations and rotations, and (ii) they are well-defined for any spatial geometry with a distance metric, such as the surface of a sphere or an arbitrary spatial network #REFR .

Again, v max is typically much smaller than the speed of sound. Lagrangian lightcones capture local causality derived from Lagrangian advection.

A Lagrangian past lightcone delineates the possible reach of a particle y(t; t 0 , y 0 ) initially at point (r 0 , t 0 ) through Lagrangian advection in reverse time.

That is, all the spacetime points at earlier times from where the particle could possibly have come when evolving according to Lagrangian advection in Eq. (S1).

In practice, the value of c used can be seen as a hyperparameter that controls the spatial scale of organization captured by the approximated local causal states. A larger c captures more coarse-grained structures.

In addition to delineating causal influence for points in spacetime, note that lightcones are defined solely in terms of distances in spacetime.

Thus, they can be used in a wide variety of spatiotemporal systems, and they transform appropriately under translations, rotations, and reflections. We will return to this later point shortly.

Note that planar projections of spherical data, like those we use here for climate data, can break global rotational symmetry, and therefore will not be robust across different projections.

However, the local causal states appear to be robust to local rotations in a given projection.

For example, the local causal state signature of hurricanes does not change as the hurricane rotates locally.

Having defined lightcones, we can now define local causal states through the local causal equivalence relation.

Physics-Informed Representation Learning For Emergent Organization In Complex Dynamical Systems

Abstract: Nonlinearly interacting system components often introduce instabilities that generate phenomena with new properties and at different space-time scales than the components. This is known as spontaneous self-organization and is ubiquitous in systems far from thermodynamic equilibrium. We introduce a theoretically-grounded framework for emergent organization that, via data-driven algorithms, is constructive in practice. Its building blocks are spacetime lightcones that capture how information propagates across a system through local interactions. We show that predictive equivalence classes of lightcones-local causal states-capture organized behaviors and coherent structures in complex spatiotemporal systems. Using our unsupervised physics-informed machine learning algorithm and a high-performance computing implementation, we demonstrate the applicability of the local causal states for real-world domain science problems. We show that the local causal states capture vortices and their power-law decay behavior in two-dimensional turbulence. We then show that known (hurricanes and atmospheric rivers) and novel extreme weather events can be identified on a pixel-level basis and tracked through time in high-resolution climate data. Emergent phenomena are often of primary interest in the study of complex systems, but disentangling the web of nonlinear interactions that give rise to them presents enormous challenges to traditional paradigms of scientific inquiry. Although the broad use of numerical models has significantly improved our ability to study nonlinear systems, difficulties remain. The scale and complexity of model outputs can be such that they are now essentially as difficult to understand as the natural phenomena they approximate. That is, the challenge of uncovering the physical and causal mechanisms of emergent phenomena in a large-scale numerical model can be comparable to doing so in the natural system itself. The Earth system is a quintessential complex system [1, 2], and the emergent behaviors associated with climate change present one of the most pressing issues of our time [3]. Many of the direct impacts of climate change on society are felt through emergent organization in the form of extreme weather events [4], such as hurricanes and atmospheric rivers (AR). As the Earth warms, the character of localized extreme weather events (EWE) has and will continue to change [5, 6]. Large-scale climate models are the primary tool used by scientists to study EWEs and climate change more generally [7, 8]. While high-resolution models can produce EWEs under a variety of warming scenarios, it is not clear what underlying mechanisms drive specific shifts in intensity, duration, and spatial dynamics of individual events. Moreover, statistical changes in EWE behaviors in large-scale surveys of various models and warming scenarios [9] are difficult to assess due to the sheer quantity of data produced. For example, CMIP 6 is estimated to contain 15-30 PB of total climate data [10]. There is an immediate need in the climate community for automated discovery and tracking of EWEs to better understand and predict how they are changing in a warming world and to help mitigate their deleterious effects. Discovering EWEs in a principled, robust, and scalable manner remains an outstanding challenge. Data-driven and machine learning methods are increasingly called upon as an additional layer of analysis to parse complex phenomena, whether produced by natural systems or complicated numerical models. At first glance, EWE detection and tracking seems an ideal use case for deep learning algorithms that have brought dramatic advances in computer vision. In machine learning terminology, we seek a segmentation analysis of extreme weather events-identifying events at the single-pixel or model-grid-cell scale. Indeed, there is considerable effort devoted to extreme weather segmentation using deep learning [11-15]. Deep learning's main weakness is that neural networks have been most successful at supervised computer vision tasks. In this, the networks are trained on ground truth labeled examples-data exemplars typically labeled by human experts. As with general fluid vortices [16], no ground truth is available for extreme weather events; see, e.g., [17]. Automated heuristics [18] are used to generate "ground-truth" training examples, but the trained neural networks' high evaluation scores show that they merely learn to reproduce the output of these heuristics. Rather than rely on supervised deep learning methods

Optimal Nonlinear Prediction Of Random Fields On Networks

Abstract: It is increasingly common to encounter time-varying random fields on networks (metabolic networks, sensor arrays, distributed computing, etc.).This paper considers the problem of optimal, nonlinear prediction of these fields, showing from an information-theoretic perspective that it is formally identical to the problem of finding minimal local sufficient statistics.I derive general properties of these statistics, show that they can be composed into global predictors, and explore their recursive estimation properties.For the special case of discrete-valued fields, I describe a convergent algorithm to identify the local predictors from empirical data, with minimal prior information about the field, and no distributional assumptions.


Especially the calibration of a model's predictions can change towards full convergence, as networks often become overconfident to match the target labels #REFR .

However, this is a trade-off worth considering for many applications.

For CQKD and DML, reducing the resolution to 168 in width and height reduces the training time by a third while reducing the accuracy by only 5%.

This speed-up is especially interesting for CQKD, as it outperforms the baseline by almost two percentage points for the same resolution and scores only one percentage point lower than the baseline trained on full resolution.

The trade-off between accuracy and computational efficiency is likely to be domain-dependent, and there may be applications where it is even more effective to use downscaling.

It is possible that slight changes in the overall results would still be observed when training for a greater number of epochs.

Nevertheless, we believe that the trends and differences between the algorithms remain unchanged, albeit at a lower accuracy.

Teacher Network Calibration Improves Cross-Quality Knowledge Distillation

Abstract: We investigate cross-quality knowledge distillation (CQKD), a knowledge distillation method where knowledge from a teacher network trained with full-resolution images is transferred to a student network that takes as input low-resolution images. As image size is a deciding factor for the computational load of computer vision applications, CQKD notably reduces the requirements by only using the student network at inference time. Our experimental results show that CQKD outperforms supervised learning in large-scale image classification problems. We also highlight the importance of calibrating neural networks: we show that with higher temperature smoothing of the teacher's output distribution, the student distribution exhibits a higher entropy, which leads to both, a lower calibration error and a higher network accuracy. The implementation is available at: https://github.com/PiaCuk/distillistic.

On Calibration Of Modern Neural Networks

Abstract: Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.


NLP and sentiment analysis involve processes that could enable computer reading, understanding, and processing of spoken or written language #REFR .

Some examples of NLP-related tasks include virtual personal assistants, chatbots, speech recognition, document description, and language or machine translation.

Sentiment Analysis or Opinion Mining uses the result of NLP to mine information or trends that could translate to moods, views, and opinions from huge data collected from different social media platforms #OTHEREFR .

For instance, politicians can use sentiment analysis to ascertain the perceived views of the electorate about their candidate.

Machine Learning Research Trends In Africa: A 30 Years Overview With Bibliometric Analysis Review

Abstract: The machine learning (ML) paradigm has gained much popularity today. Its algorithmic models are employed in every field, such as natural language processing, pattern recognition, object detection, image recognition, earth observation and many other research areas. In fact, machine learning technologies and their inevitable impact suffice in many technological transformation agendas currently being propagated by many nations, for which the already yielded benefits are outstanding. From a regional perspective, several studies have shown that machine learning technology can help address some of Africa's most pervasive problems, such as poverty alleviation, improving education, delivering quality healthcare services, and addressing sustainability challenges like food security and climate change. In this state-of-the-art paper, a critical bibliometric analysis study is conducted, coupled with an extensive literature survey on recent developments and associated applications in machine learning research with a perspective on Africa. The presented bibliometric analysis study consists of 2761 machine learning-related documents, of which 89% were articles with at least 482 citations published in 903 journals during the past three decades. Furthermore, the collated documents were retrieved from the Science Citation Index EXPANDED, comprising research publications from 54 African countries between 1993 and 2021. The bibliometric study shows the visualization of the current landscape and future trends in machine learning research and its application to facilitate future collaborative research and knowledge exchange among authors from different research institutions scattered across the African continent.

A Survey Of The Usages Of Deep Learning In Natural Language Processing

Abstract: Over the last several years, the field of natural language processing has been propelled forward by an explosion in the use of deep learning models. This survey provides a brief introduction to the field and a quick overview of deep learning architectures and methods. It then sifts through the plethora of recent studies and summarizes a large assortment of relevant contributions. Analyzed research areas include several core linguistic processing issues in addition to a number of applications of computational linguistics. A discussion of the current state of the art is then provided along with recommendations for future research in the field.


Wang and Sontag #REFR showed that a singularly perturbed strongly monotone system inherits the generic convergence properties of strongly monotone systems, and demonstrated an application in systems biology.

a matrix k-cone (see (1)) in a systems and control perspective, calling them k-dominant systems, and used (2) as the definition of k-dominance with rate s.

Importantly, using the fact that (2) is an LMI, they extended the notion of a system that is monotone w.r.t.

a matrix k-cone to systems with inputs and outputs, and studied the interconnections of such systems.

Singularly perturbed systems #OTHEREFR , #OTHEREFR Ch. 11 ] appear naturally in many control applications.

For instance, see #OTHEREFR , #OTHEREFR for an application to low-gain integral control, and #OTHEREFR for an application to power systems stability.

Niu and Xie #OTHEREFR showed that singularly perturbed systems that are strongly monotone w.r.t. a 2-cone inherit the generic Poincaré-Bendixon property.

The results in #OTHEREFR , #OTHEREFR are based on geometric singular perturbations theory #OTHEREFR , #OTHEREFR .

Here, we consider k-dominance for singularly perturbed systems.

We derive a new sufficient condition for a singularly perturbed system to be k-dominant.

Our proof uses standard tools from linear algebra and dynamical systems theory, and does not require the heavy machinery of geometric singular perturbations theory.

On Singularly Perturbed Systems That Are Monotone With Respect To A Matrix Cone Of Rank K

Abstract: We derive a sufficient condition guaranteeing that a singularly perturbed linear time-varying system is strongly monotone with respect to a matrix cone C of rank k. This implies that the singularly perturbed system inherits the asymptotic properties of systems that are strongly monotone with respect to C, which include convergence to the set of equilibria when k = 1, and a Poincaré-Bendixson property when k = 2. We extend this result to singularly perturbed nonlinear systems with a compact and convex state-space. We demonstrate our theoretical results using a simple numerical example.

Singularly Perturbed Monotone Systems And An Application To Double Phosphorylation Cycles

Abstract: The theory of monotone dynamical systems has been found very useful in the modeling of some gene, protein, and signaling networks. In monotone systems, every net feedback loop is positive. On the other hand, negative feedback loops are important features of many systems, since they are required for adaptation and precision. This paper shows that, provided that these negative loops act at a comparatively fast time scale, the main dynamical property of (strongly) monotone systems, convergence to steady states, is still valid. An application is worked out to a double-phosphorylation “futile cycle” motif which plays a central role in eukaryotic cell signaling.

strongly monotone systems
monotone system


In Cui, Dai, and Nie #REFR , a semidefinite relaxation method was developed to find all real eigenvalues of symmetric tensor pairs.

For simplicity, we use • to denote the 2-norm for vectors and matrices in this paper.

Several numerical methods have been proposed in the literature for computing generalized eigenpairs for different classes of tensor pairs.

Kolda and Mayo #OTHEREFR proposed a power method for computing the generalized eigenpairs for symmetric tensor pair.

In #OTHEREFR , Chen, Han and Zhou presented the homotopy methods for computing the (generalized) tensor eigenpair.

Yu, Yu, Xu, Song, and Zhou #OTHEREFR gave an adaptive gradient method for computing generalized tensor eigenpairs.

Zhao, Yang and Liu #OTHEREFR computed the generalized eigenvalues of weakly symmetric tensors.

Che, Cichocki and Wei #OTHEREFR applied the neural dynamical network to compute a best rank-one approximation of a realvalued tensor and solve the tensor eigenvalue problems.

Mo, Wang and Wei #OTHEREFR explored time-varying generalized tensor eigenanalysis via Zhang neural networks.

Noda Iteration For Computing Generalized Tensor Eigenpairs

Abstract: In this paper, we propose the tensor Noda iteration (NI) and its inexact version for solving the eigenvalue problem of a particular class of tensor pairs called generalized M-tensor pairs. A generalized M-tensor pair consists of a weakly irreducible nonnegative tensor and a nonsingular Mtensor within a linear combination. It is shown that any generalized M-tensor pair admits a unique positive generalized eigenvalue with a positive eigenvector. A modified tensor Noda iteration(MTNI) is developed for extending the Noda iteration for nonnegative matrix eigenproblems. In addition, the inexact generalized tensor Noda iteration method (IGTNI) and the generalized Newton-Noda iteration method (GNNI) are also introduced for more efficient implementations and faster convergence. Under a mild assumption on the initial values, the convergence of these algorithms is guaranteed. The efficiency of these algorithms is illustrated by numerical experiments.

All Real Eigenvalues Of Symmetric Tensors

Abstract: This paper studies how to compute all real eigenvalues of a symmetric tensor. As is well known, the largest or smallest eigenvalue can be found by solving a polynomial optimization problem, while the other middle eigenvalues can not. We propose a new approach for computing all real eigenvalues sequentially, from the largest to the smallest. It uses Jacobian SDP relaxations in polynomial optimization. We show that each eigenvalue can be computed by solving a finite hierarchy of semidefinite relaxations. Numerical experiments are presented to show how to do this.


The gradient-based explanations (e.g., Input Gradient) are consistent with sample-based explanations (e.g., LIME) with comparable fidelity but have much lower computational cost #REFR .

A suite of techniques has been proposed to reveal the decision-making process of modern black-box ML models.

One direction is intrinsic to the model design and training, rendering an explanation along with its output, e.g., attention mechanisms #OTHEREFR and joint training #OTHEREFR .

A more popular way is to give insight into the learned associations of a model that are not readily interpretable by design, known as post-hoc methods.

Such methods usually leverage backpropagation or local approximation to offer saliency maps as explanations, e.g., Input Gradient #OTHEREFR , Grad-CAM #OTHEREFR , LIME #OTHEREFR , and SHAP #OTHEREFR .

Recent works have shed light on the downsides of post-hoc methods.

Moreover, only the Input Gradient and Grad-CAM methods passed the sanity checks in #OTHEREFR .

In our work, we incorporate gradient-based methods into optimization to calculate explanations efficiently.

To generalize machine learning models from training distributions to unseen distributions, existing methods on OOD generalization can be categorized into four branches: (1) Data augmentation.

#OTHEREFR ] enhance the generalization performance by increasing the diversity of data through augmentation. (2) Distribution alignment.

Are Data-Driven Explanations Robust Against Out-Of-Distribution Data?

Abstract: As black-box models increasingly power high-stakes applications, a variety of data-driven explanation methods have been introduced. Meanwhile, machine learning models are constantly challenged by distributional shifts. A question naturally arises: Are data-driven explanations robust against out-of-distribution data? Our empirical results show that even though predict correctly, the model might still yield unreliable explanations under distributional shifts. How to develop robust explanations against out-of-distribution data? To address this problem, we propose an end-to-end model-agnostic learning framework Distributionally Robust Explanations (DRE). The key idea is, inspired by self-supervised learning, to fully utilizes the inter-distribution information to provide supervisory signals for the learning of explanations without human annotation. Can robust explanations benefit the model's generalization capability? We conduct extensive experiments on a wide range of tasks and data types, including classification and regression on image and scientific tabular data. Our results demonstrate that the proposed method significantly improves the model's performance in terms of explanation and prediction robustness against distributional shifts.

Right For The Right Reasons: Training Differentiable Models By Constraining Their Explanations

Abstract: Neural networks are among the most accurate supervised learning methods in use today, but their opacity makes them difficult to trust in critical applications, especially when conditions in training differ from those in test. Recent work on explanations for black-box models has produced tools (e.g. LIME) to show the implicit rules behind predictions, which can help us identify when models are right for the wrong reasons. However, these methods do not scale to explaining entire datasets and cannot correct the problems they reveal. We introduce a method for efficiently explaining and regularizing differentiable models by examining and selectively penalizing their input gradients, which provide a normal to the decision boundary. We apply these penalties both based on expert annotation and in an unsupervised fashion that encourages diverse models with qualitatively different decision boundaries for the same classification problem. On multiple datasets, we show our approach generates faithful explanations and models that generalize much better when conditions differ between training and test.


The work of #REFR first introduced a baseline called MSP as the scoring function to detect OOD inputs.

OOD Detection Various methods are proposed under the setting that whether use the auxiliary data of natural outliers or not.

Later, various methods focus on designing scoring functions are proposed #OTHEREFR .

Generative models #OTHEREFR are also applied to provide reliable estimations of confidence.

Besides, OE #OTHEREFR first introduced an extra dataset consisting of outliers in training and enforced low confidence on such outliers.

To better utilize the auxiliary dataset, different methods are propose #OTHEREFR .

There are also some methods to synthesize outliers #OTHEREFR .

Average Of Pruning: Improving Performance And Stability Of Out-Of-Distribution Detection

Abstract: Detecting Out-of-distribution (OOD) inputs have been a critical issue for neural networks in the open world. However, the unstable behavior of OOD detection along the optimization trajectory during training has not been explored clearly. In this paper, we first find the performance of OOD detection suffers from overfitting and instability during training: 1) the performance could decrease when the training error is near zero, and 2) the performance would vary sharply in the final stage of training. Based on our findings, we propose Average of Pruning (AoP), consisting of model averaging and pruning, to mitigate the unstable behaviors. Specifically, model averaging can help achieve a stable performance by smoothing the landscape, and pruning is certified to eliminate the overfitting by eliminating redundant features. Comprehensive experiments on various datasets and architectures are conducted to verify the effectiveness of our method.

A Baseline For Detecting Misclassified And Out-Of-Distribution Examples In Neural Networks

Abstract: We consider the two related problems of detecting if an example is misclassified or out-of-distribution. We present a simple baseline that utilizes probabilities from softmax distributions. Correctly classified examples tend to have greater maximum softmax probabilities than erroneously classified and out-of-distribution examples, allowing for their detection. We assess performance by defining several tasks in computer vision, natural language processing, and automatic speech recognition, showing the effectiveness of this baseline across all. We then show the baseline can sometimes be surpassed, demonstrating the room for future research on these underexplored detection tasks.


Temporal Queue For Negative Pairs Exclusion

MoCo #REFR uses the pairs (q, k) as positive and all the pairs {(q, k i )} N i=1 as negative pairs by minimising and maximising their cosine distance.

Queries and keys features will be used to build the positive pairs.

To build negative pairs, we also maintain a queue Q e that stores keys features

The negative pairs are then composed by pairing features from the queue.

Since features are stored in Q e independently from the class, even features from samples sharing the same class will be pushed apart, which is in contrast with our objective. Recently, Chen et al.

#OTHEREFR propose a strategy to exclude some negative pairs from the contrastive loss.

For every negative pair, they just compare the pseudolabels of the two samples composing the pair.

If the two samples share the same pseudo-label, then the negative pair is masked out.

Otherwise, the negative pair composed of the two samples is included in the negative pairs list.

Guiding Pseudo-Labels With Uncertainty Estimation For Source-Free Unsupervised Domain Adaptation

Abstract: Standard Unsupervised Domain Adaptation (UDA) methods assume the availability of both source and target data during the adaptation. In this work, we investigate the Source-free Unsupervised Domain Adaptation (SF-UDA), a specific case of UDA where a model is adapted to a target domain without access to source data. We propose a novel approach for the SF-UDA setting based on a loss reweighting strategy that brings robustness against the noise that inevitably affects the pseudo-labels. The classification loss is reweighted based on the reliability of the pseudo-labels that is measured by estimating their uncertainty. Guided by such reweighting strategy, the pseudo-labels are progressively refined by aggregating knowledge from neighbouring samples. Furthermore, a self-supervised contrastive framework is leveraged as a target space regulariser to enhance such knowledge aggregation. A novel negative pairs exclusion strategy is proposed to identify and exclude negative pairs made of samples sharing the same class, even in presence of some noise in the pseudo-labels. Our method outperforms previous methods on three major benchmarks by a large margin. We set the new SF-UDA state-of-the-art on VisDA-C and DomainNet with a performance gain of +1.8% on both benchmarks and on PACS with +12.3% in the single-source setting and +6.6% in multi-target adaptation. Additional analyses demonstrate that the proposed approach is robust to the noise, which results in significantly more accurate pseudo-labels compared to state-of-the-art approaches.

Momentum Contrast For Unsupervised Visual Representation Learning

Abstract: We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.


Four high-resource datasets(MNLI, QQP, SST, QNLI) are used as training datasets, and four other low-resource datasets(CoLA, MRPC, STS-B, RTE) are used as testing datasets, according to #REFR .

We briefly describes the GLUE, SNLI, and SciTail datasets, as summarized in Table 1 .

GLUE The General Language Understanding Evaluation (GLUE) benchmark #OTHEREFR is a tool for evaluating and analyzing the performance of natural language understanding models across nine NLU tasks: Single-Sentence Tasks, Similarity and Paraphrase Tasks and Inference Tasks.

In our experiments we do not train or test models on the WNLI dataset because of previous work #OTHEREFR .

SNLI The Stanford Natural Language Inference dataset #OTHEREFR is a naturalistic corpus of 570k sentence pairs labeled for entailment, contradiction, and independence.

We use this dataset to examine the algorithm's fast adaptation ability in this study.

Sci-Tail This is a Textual Entailment Dataset from Science Question Answering #OTHEREFR .

Hypotheses from science questions are created while the corresponding answer candidates and premises from relevant web sentences are retrieved from a large corpus.

Model-Agnostic Meta-Learning For Natural Language Understanding Tasks In Finance

Abstract: Natural language understanding(NLU) is challenging for finance due to the lack of annotated data and the specialized language in that domain. As a result, researchers have proposed to use pre-trained language model and multi-task learning to learn robust representations. However, aggressive finetuning often causes over-fitting and multi-task learning may favor tasks with significantly larger amounts data, etc. To address these problems, in this paper, we investigate model-agnostic metalearning algorithm(MAML) in low-resource financial NLU tasks. Our contribution includes: 1. we explore the performance of MAML method with multiple types of tasks: GLUE datasets, SNLI, Sci-Tail and Financial PhraseBank; 2. we study the performance of MAML method with multiple single-type tasks: a real scenario stock price prediction problem with twitter text data. Our models achieve the state-of-the-art performance according to the experimental results, which demonstrate that our method can adapt fast and well to low-resource situations.

Investigating Meta-Learning Algorithms For Low-Resource Natural Language Understanding Tasks

Abstract: Learning general representations of text is a fundamental problem for many natural language understanding (NLU) tasks. Previously, researchers have proposed to use language model pre-training and multi-task learning to learn robust representations. However, these methods can achieve sub-optimal performance in low-resource scenarios. Inspired by the recent success of optimization-based meta-learning algorithms, in this paper, we explore the model-agnostic meta-learning algorithm (MAML) and its variants for low-resource NLU tasks. We validate our methods on the GLUE benchmark and show that our proposed models can outperform several strong baselines. We further empirically demonstrate that the learned representations can be adapted to new tasks efficiently and effectively.

training datasets
Four high-resource datasets(MNLI


The other loss for C-BERT is inspired by masked language modeling #REFR , which regards the training process as a masked token prediction task.

Thus, a margin-based loss is set to discriminate valid triples from invalid ones with the distance function in Eq.(1):

is a negative sample for (ℎ, , ) by replacing either the tail entity or the relation a random one^∈ or^∈ , where controls the safety margin between positive and negative triples.

During finetuning, each token in the input sentence has a 15% chance of being replaced with a [MASK] token, where C-BERT is required to predict the masked tokens from inputs.

It is defined as a classification task with cross-entropy loss:

where is list of masked tokens, (^) is the language model's predicted probability for token in the corresponding training sample.

The training loss is the sum of and to fuse the KG connection information and the linguistic semantics into the entity representation:

Joint Semantic And Structural Representation Learning For Enhancing User Preference Modelling

Abstract: Knowledge graphs (KGs) have become important auxiliary information for helping recommender systems obtain a good understanding of user preferences. Despite recent advances in KG-based recommender systems, existing methods are prone to suboptimal performance due to the following two drawbacks: 1) current KG-based methods overemphasize the heterogeneous structural information within a KG and overlook the underlying semantics of its connections, hindering the recommender from distilling the explicit user preferences; and 2) the inherent incompleteness of a KG (i.e., missing facts, relations and entities) will deteriorate the information extracted from KG and weaken the representation learning of recommender systems. To tackle the aforementioned problems, we investigate the potential of jointly incorporating the structural and semantic information within a KG to model user preferences in finer granularity. A new framework for KG-based recommender systems, namely Knowledge I nfomax Recommender System with Contrastive Learning (KIRS-CL) is proposed in this paper. Distinct from previous KG-based approaches, KIRS-CL utilizes structural and connectivity information with high-quality item embeddings learned by encoding KG triples with a pre-trained language model. These well-trained entity representations enable KIRS-CL to find the item to recommend via the preference connection between the user and the item. Additionally, to improve the generalizability of our framework, we introduce a contrastive warm-up learning strategy, making it capable of dealing with both warm-and cold-start recommendation scenarios. Extensive experiments on two real-world datasets demonstrate remarkable improvements over state-of-the-art baselines.

Ernie: Enhanced Representation Through Knowledge Integration

Abstract: We present a novel language representation model enhanced by knowledge called ERNIE (Enhanced Representation through kNowledge IntEgration). Inspired by the masking strategy of BERT, ERNIE is designed to learn language representation enhanced by knowledge masking strategies, which includes entity-level masking and phrase-level masking. Entity-level strategy masks entities which are usually composed of multiple words.Phrase-level strategy masks the whole phrase which is composed of several words standing together as a conceptual unit.Experimental results show that ERNIE outperforms other baseline methods, achieving new state-of-the-art results on five Chinese natural language processing tasks including natural language inference, semantic similarity, named entity recognition, sentiment analysis and question answering. We also demonstrate that ERNIE has more powerful knowledge inference capacity on a cloze test.


Unlike #REFR , we explore using the encoded positional information to capture high-frequency signals, which echoes our theoretical findings in Appendix A.

where v is the logit vector of that pixel such that the coarse segmentation is given as Softmax(v).

The rest (1 − ρ)N p pixels are sampled uniformly from all the remaining pixels.

This mechanism ensures the selected points contain a large portion of points with uncertain segmentation which require refinement.

Positional Encoding It is well-known that neural networks can be cast as universal function approximators, but they are inferior to high-frequency signals due to their limited learning power #OTHEREFR .

Specifically, for a coordinate-based point (x, y) ∈ [H] × [W ], the positional encoding function is given as:

are trainable parameters with Gaussian random initialization, where we set L = 128 #OTHEREFR .

For each selected point, its position encoding will then be concatenated with the coarse features of that point (i.e., x out defined in Sec. 2.1), to output the fine-grained features.

Implicit Anatomical Rendering For Medical Image Segmentation With Stochastic Experts

Abstract: Integrating high-level semantically correlated contents and low-level anatomical features is of central importance in medical image segmentation. Towards this end, recent deep learning-based medical segmentation methods have shown great promise in better modeling such information. However, convolution operators for medical segmentation typically operate on regular grids, which inherently blur the high-frequency regions, i.e., boundary regions. In this work, we propose MORSE, a generic implicit neural rendering framework designed at an anatomical level to assist learning in medical image segmentation. Our method is motivated by the fact that implicit neural representation has been shown to be more effective in fitting complex signals and solving computer graphics problems than discrete grid-based representation. The core of our approach is to formulate medical image segmentation as a rendering problem in an end-to-end manner. Specifically, we continuously align the coarse segmentation prediction with the ambiguous coordinate-based point representations and aggregate these features to adaptively refine the boundary region. To parallelly optimize multi-scale pixel-level features, we leverage the idea from Mixture-of-Expert (MoE) to design and train our MORSE with a stochastic gating mechanism. Our experiments demonstrate that MORSE can work well with different medical segmentation backbones, consistently achieving competitive performance improvements in both 2D and 3D supervised medical segmentation methods. We also theoretically analyze the superiority of MORSE.

Pointrend: Image Segmentation As Rendering

Abstract: We present a new method for efficient high-quality image segmentation of objects and scenes. By analogizing classical computer graphics methods for efficient rendering with over- and undersampling challenges faced in pixel labeling tasks, we develop a unique perspective of image segmentation as a rendering problem. From this vantage, we present the PointRend (Point-based Rendering) neural network module: a module that performs point-based segmentation predictions at adaptively selected locations based on an iterative subdivision algorithm. PointRend can be flexibly applied to both instance and semantic segmentation tasks by building on top of existing state-of-the-art models. While many concrete implementations of the general idea are possible, we show that a simple design already achieves excellent results. Qualitatively, PointRend outputs crisp object boundaries in regions that are over-smoothed by previous methods. Quantitatively, PointRend yields significant gains on COCO and Cityscapes, for both instance and semantic segmentation. PointRend's efficiency enables output resolutions that are otherwise impractical in terms of memory or computation compared to existing approaches.


C. Approximate True Policy By Composing Primitives

The approximation in #REFR assumes the maximum SF of each sub-task is most likely to be the SF of the primitive trained from the respective sub-task.

We provide analytical view on how the task policy can be reconstructed by a set of task primitives given their relation in value space introduced in Sec. IV-A.

1) MSF Composition: Following MSF in value space (16) we derive the correspondent policy composition,

Then the MSF composition can be achieved by MCP, i.e.π(•|s) = M CP (Π, g M SF ), where g M SF = w |w| is the normalized gating vector. return a

2) SFV Composition: Following SFV in value space #OTHEREFR we derive the correspondent policy composition,

= d i=0 e 1 d wiψ π i i (s,•) d i=0 d j =i e 1 d wj ψ π i j (s,•) ∝ d i=0 π i (•|s) w i d • (22)

Multi-Task Reinforcement Learning In Continuous Control With Successor Feature-Based Concurrent Composition

Abstract: Deep reinforcement learning (DRL) frameworks are increasingly used to solve high-dimensional continuouscontrol tasks in robotics. However, due to the lack of sample efficiency, applying DRL for online learning is still practically infeasible in the robotics domain. One reason is that DRL agents do not leverage the solution of previous tasks for new tasks. Recent work on multi-tasking DRL agents based on successor features has proven to be quite promising in increasing sample efficiency. In this work, we present a new approach that unifies two prior multi-task RL frameworks, SF-GPI and value composition, for the continuous control domain. We exploit compositional properties of successor features to compose a policy distribution from a set of primitives without training any new policy. Lastly, to demonstrate the multi-tasking mechanism, we present a new benchmark for multi-task continuous control environment based on Raisim. This also facilitates large-scale parallelization to accelerate the experiments. Our experimental results in the Pointmass environment show that our multi-task agent has single task performance on par with soft actor critic (SAC) and the agent can successfully transfer to new unseen tasks where SAC fails. We provide our code as open-source at https://github.com/robot-perception-group/ concurrent_composition for the benefit of the community.

Kickstarting Deep Reinforcement Learning

Abstract: We present a method for using previously-trained 'teacher' agents to kickstart the training of a new 'student' agent. To this end, we leverage ideas from policy distillation and population based training. Our method places no constraints on the architecture of the teacher or student agents, and it regulates itself to allow the students to surpass their teachers in performance. We show that, on a challenging and computationally-intensive multi-task benchmark (DMLab-30), kickstarted training improves the data efficiency of new agents, making it significantly easier to iterate on their design. We also show that the same kickstarting pipeline can allow a single student agent to leverage multiple 'expert' teachers which specialize on individual tasks. In this setting kickstarting yields surprisingly large gains, with the kickstarted agent matching the performance of an agent trained from scratch in almost 10x fewer steps, and surpassing its final performance by 42 percent. Kickstarting is conceptually simple and can easily be incorporated into reinforcement learning experiments.


This leads to a more powerful network and can generate better human face images #REFR but ∇ log q θ (x) is not Lipschitz. This matches to the experiments in Sec. IV-A.

Note that the light tail of q θ (x) does not necessarily imply the Lipschitz of ∇ log q θ (x), but the condition they assume is heuristic: if ϕ j is bounded above and η j is Lipschitz, then q θ (x) is light-tailed.

In Glow #OTHEREFR these conditions on ϕ and η are satisfied and even stricter.

Therefore we are able to prove the Lipschitz of ∇ log q θ (x) in the proposition below, by enforcing a stricter condition on ϕ and η: Proposition 2. Assume that q z (z) is standard normal distribution.

Denote G(x) = T −1 (x) and assume the affine coupling meanwhile ϕ = 1 is used in additive coupling layer.

Note that in Glow there is another option using affine coupling layer where ϕ is sigmoid function.

The conditions on ϕ and η might be relaxed if q z (z) is not Gaussian, but this requires re-training the net since most of the popular normalizing flows has standard Gaussian base distribution.

Hence we leave these possible studies on the Lipschitz-continuity of ∇ log q θ (x) for future work.

layer G j (x j , x <j ) = ϕ j (x <j )x j + η j (x <j ), j = 1, • • • , d.

If ϕ j is a constant function, η j is Lipschitz, ∀i < j,

In what follows, we introduce the associated stochastic kernel R δ : #OTHEREFR and the drift b λ (x):

Nf-Ula: Langevin Monte Carlo With Normalizing Flow Prior For Imaging Inverse Problems

Abstract: Bayesian methods for solving inverse problems are a powerful alternative to classical methods since the Bayesian approach gives a probabilistic description of the problems and offers the ability to quantify the uncertainty in the solution. Meanwhile, solving inverse problems by data-driven techniques also proves to be successful, due to the increasing representation ability of data-based models. In this work, we try to incorporate the data-based models into a class of Langevin-based sampling algorithms in Bayesian inference. Loosely speaking, we introduce NF-ULA (Unadjusted Langevin algorithms by Normalizing Flows), which involves learning a normalizing flow as the prior. In particular, our algorithm only requires a pre-trained normalizing flow, which is independent of the considered inverse problem and the forward operator. We perform theoretical analysis by investigating the well-posedness of the Bayesian solution and the non-asymptotic convergence of the NF-ULA algorithm. The efficacy of the proposed NF-ULA algorithm is demonstrated in various imaging problems, including image deblurring, image inpainting, and limited-angle X-ray computed tomography (CT) reconstruction.

Glow: Generative Flow With Invertible 1X1 Convolutions

Abstract: Flow-based generative models (Dinh et al., 2014) are conceptually attractive due to tractability of the exact log-likelihood, tractability of exact latent-variable inference, and parallelizability of both training and synthesis. In this paper we propose Glow, a simple type of generative flow using an invertible 1x1 convolution. Using our method we demonstrate a significant improvement in log-likelihood on standard benchmarks. Perhaps most strikingly, we demonstrate that a generative model optimized towards the plain log-likelihood objective is capable of efficient realistic-looking synthesis and manipulation of large images. The code for our model is available at https://github.com/openai/glow


In #REFR a method to use a learned deep inverse dynamics model to decide which real-world action is most suitable to achieve the same state as the simulator is proposed.

In #OTHEREFR , the authors separate the learned policy from the raw inputs and outputs to ease the transfer from simulation.

Mutual alignment transfer learning approaches employ auxiliary rewards for transfer learning under discrepancies in system dynamics for simulation to robot transfer ( #OTHEREFR ).

Approaches such as #OTHEREFR compensate for the difference in dynamics by modifying the reward function such that the modified reward function penalizes the agent for visiting states and taking actions in the source domain which are not possible in the target domain.

In #OTHEREFR , the authors study robust adversarial reinforcement learning.

Inspired with the H ∞ control idea, they assume the destabilizing adversaries like the gap between simulation and environment as uncertainties and devise a learning algorithm which considers the worst case adversary and is robust against it.

Transfer learning has also been investigated in the multi-agent setting #OTHEREFR , where the problem of training agents with continuous actions is studied to ensure that the trained agents can still generalize when their opponent's policies alter.

Convex Optimization-Based Policy Adaptation To Compensate For Distributional Shifts

Abstract: Many real-world systems often involve physical components or operating environments with highly nonlinear and uncertain dynamics. A number of different control algorithms can be used to design optimal controllers for such systems, assuming a reasonably high-fidelity model of the actual system. However, the assumptions made on the stochastic dynamics of the model when designing the optimal controller may no longer be valid when the system is deployed in the real-world. The problem addressed by this paper is the following: Suppose we obtain an optimal trajectory by solving a control problem in the training environment, how do we ensure that the real-world system trajectory tracks this optimal trajectory with minimal amount of error in a deployment environment. In other words, we want to learn how we can adapt an optimal trained policy to distribution shifts in the environment. Distribution shifts are problematic in safety-critical systems, where a trained policy may lead to unsafe outcomes during deployment. We show that this problem can be cast as a nonlinear optimization problem that could be solved using heuristic method such as particle swarm optimization (PSO). However, if we instead consider a convex relaxation of this problem, we can learn policies that track the optimal trajectory with much better error performance, and faster computation times. We demonstrate the efficacy of our approach on tracking an optimal path using a Dubin's car model, and collision avoidance using both a linear and nonlinear model for adaptive cruise control.

Transfer From Simulation To Real World Through Learning Deep Inverse Dynamics Model

Abstract: Developing control policies in simulation is often more practical and safer than directly running experiments in the real world. This applies to policies obtained from planning and optimization, and even more so to policies obtained from reinforcement learning, which is often very data demanding. However, a policy that succeeds in simulation often doesn't work when deployed on a real robot. Nevertheless, often the overall gist of what the policy does in simulation remains valid in the real world. In this paper we investigate such settings, where the sequence of states traversed in simulation remains reasonable for the real world, even if the details of the controls are not, as could be the case when the key differences lie in detailed friction, contact, mass and geometry properties. During execution, at each time step our approach computes what the simulation-based control policy would do, but then, rather than executing these controls on the real robot, our approach computes what the simulation expects the resulting next state(s) will be, and then relies on a learned deep inverse dynamics model to decide which real-world action is most suitable to achieve those next states. Deep models are only as good as their training data, and we also propose an approach for data collection to (incrementally) learn the deep inverse dynamics model. Our experiments shows our approach compares favorably with various baselines that have been developed for dealing with simulation to real world model discrepancy, including output error control and Gaussian dynamics adaptation.


In #REFR , convolutional kernels are decomposed into shared bases and coefficient volumes to reduce parameters of each layer.

It is composed of fixed spatial kernels (blue) and switchable angular kernels (green and orange) for learning spatial information and angular information respectively.

To further compact the network, angular kernels are allocated to the horizontal (green) and vertical (orage) directions, and both spatial and angular kernels are decomposed into the global bases (yellow cylinder) and corresponding coefficients (blue, green and orange cylinders).

After training process, a light field can be implicitly represented by global bases {B}, spatial kernel coefficients {W s i } and angular kernel coefficients and biases

{K a i u , K a i v , b a i u , b a i v }.

mulating it as a tensor rank optimization problem, and decompose tensors into Tensor Ring (TR) or Tensor Train (TT) format within Alternating Direction Method of Multipliers (ADMM) framework.

Pruning is another compression approach that enforces sparsity of the network. Zhang et al.

#OTHEREFR tackle weights pruning as a nonconvex optimization problem with combinatorial constraints specifying the sparsity requirements, and iteratively solve this optimization within ADMM framework.

The authors in #OTHEREFR propose using a proxy network to predict the accuracy of the pruned network and guide the pruning process to achieve maximum performance.

In #OTHEREFR , network pruning is reformulated as a structural redundancy identification problem, pruning the layers with the most redundancy instead of the least important filters gives better performance.

Quantizing the network weights is another method to reduce the model size.

Learning-Based Spatial And Angular Information Separation For Light Field Compression

Abstract: Light fields are a type of image data that capture both spatial and angular scene information by recording light rays emitted by a scene from different orientations. In this context, spatial information is defined as features that remain static regardless of perspectives, while angular information refers to features that vary between viewpoints. We propose a novel neural network that, by design, can separate angular and spatial information of a light field. The network represents spatial information using spatial kernels shared among all Sub-Aperture Images (SAIs), and angular information using sets of angular kernels for each SAI. To further improve the representation capability of the network without increasing parameter number, we also introduce angular kernel allocation and kernel tensor decomposition mechanisms. Extensive experiments demonstrate the benefits of information separation: when applied to the compression task, our network outperforms other state-ofthe-art methods by a large margin. And angular information can be easily transferred to other scenes for rendering dense views, showing the successful separation and the potential use case for the view synthesis task. We plan to release the code upon acceptance of the paper to encourage further research on this topic.

Dcfnet: Deep Neural Network With Decomposed Convolutional Filters

Abstract: Filters in a Convolutional Neural Network (CNN) contain model parameters learned from enormous amounts of data. In this paper, we suggest to decompose convolutional filters in CNN as a truncated expansion with pre-fixed bases, namely the Decomposed Convolutional Filters network (DCFNet), where the expansion coefficients remain learned from data. Such a structure not only reduces the number of trainable parameters and computation, but also imposes filter regularity by bases truncation. Through extensive experiments, we consistently observe that DCFNet maintains accuracy for image classification tasks with a significant reduction of model parameters, particularly with Fourier-Bessel (FB) bases, and even with random bases. Theoretically, we analyze the representation stability of DCFNet with respect to input variations, and prove representation stability under generic assumptions on the expansion coefficients. The analysis is consistent with the empirical observations.


Using this constraint, the ζ T,max v for a critical VNF instance can be set at a lower value, and hence it will not be shared which will protect the critical VNF. This is represented in constraint #REFR .

• Constraint #OTHEREFR certifies that the latency introduced by nodes' processing and network propagation can't exceed the maximum tolerated latency of a particular procedure.

Two security constraints are formulated in the model; VNF's maximum traffic and VNF exposure constraints. These constraints are explained as follows:

• The VNF's maximum traffic: This constraint ensures that the traffic processing capacity ζ T,n vi of a VNF instance v i should not exceed the predefined maximum traffic processing capacity ζ T,max v .

ζ n,T vi ≤ ζ T,max v , ∀v ∈ V, i ∈ I v , n ∈ N (16)

• The VNF exposure: The VNF that is exposed to the outside network cannot be assigned to more than one slice.

A VNF is exposed to the outside network if it the first VNF in the VNFs chain serving a procedure that is initiated by the UE or the Radio Access Network (RAN).

First, let Ω n,s vi denotes to that VNF instance v i deployed in physical node n is exposed to the outside by the slice s.

The Ω n,s vi is calculated by Equations #OTHEREFR and #OTHEREFR

A Security-Aware Network Function Sharing Model For 5G Slicing Authors' Draft For Soliciting Feedback, March 6, 2023

Abstract: Sharing Virtualized Network Functions (VNFs) among different slices in Fifth Generation (5G) is a potential strategy to simplify the system implementation and utilize 5G resources efficiently. In this paper, we propose a security-aware VNF sharing model for 5G networks. The proposed optimization model satisfies the service requirements of various slices, enhances slice security by isolating their critical VNFs, and enhances resource utilization of the underlying physical infrastructure. The model tries to systematically decide on sharing a particular VNF based on two groups of constraints; the first group of constraints is common assignment constraints used in the existing literature. The second group is the novel security constraints that we propose in this work; the maximum traffic allowed to be processed by the VNF and the exposure of the VNF to procedures sourced via untrusted users or access networks. This sharing problem is formalized to allow for procedure-level modeling that satisfies the requirements of slice requests in 5G systems. The model is tested using standard VNFs and procedures of the 5G system rather than generic ones. The numerical results of the model show the benefits and costs of applying the security constraints along with the network performance in terms of different metrics.

Optimal Slice Allocation In 5G Core Networks

Abstract: 5G network slicing is essential to providing flexible, scalable, and on-demand solutions for the vast array of applications in 5G networks. Two key challenges of 5G network slicing are function isolation (intra-slice) and guaranteeing end-to-end delay for a slice. In this letter, we address the question of optimal allocation of a slice in 5G core networks by tackling these two challenges. We adopt and extend the work by Dietrich et al. (see introduction) to create a model that satisfies constraints on end-to-end delay as well as isolation between components of a slice for reliability.


We identified that the VoxCeleb2 #REFR dataset is ideal for our needs because it comprises video snippets from a wide range of speakers.

To learn the styles from different speakers, we require an audio-visual dataset with a broad selection of speakers to learn the speaker-aware dynamics variations of head motion and facial expressions.

Since our purpose is to capture speaker dynamics for talking head synthesis, we picked a subset of 67 speakers from VoxCeleb2 with a total of 1, 232 video clips.

We have about 5 − 10 minutes of footage for each speaker.

For each video, we perform image retrieval to find key motion frames and use them as the Style Reference Images.

During retrieval, we use each motion template from 4 pre-defined key motion templates (mentioned in Section 4.1) to retrieve from a series of motion maps in the reference clip.

Then, we collect indexes and obtain corresponding images and audio signals, which are expected to have the style information.

Style Transfer For 2D Talking Head Animation

Abstract: Audio-driven talking head animation is a challenging research topic with many real-world applications. Recent works have focused on creating photo-realistic 2D animation, while learning different talking or singing styles remains an open problem. In this paper, we present a new method to generate talking head animation with learnable style references. Given a set of style reference frames, our framework can reconstruct 2D talking head animation based on a single input image and an audio stream. Our method first produces facial landmarks motion from the audio stream and constructs the intermediate style patterns from the style reference images. We then feed both outputs into a style-aware image generator to generate the photo-realistic and fidelity 2D animation. In practice, our framework can extract the style information of a specific character and transfer it to any new static image for talking head animation. The intensive experimental results show that our method achieves better results than recent state-of-the-art approaches qualitatively and quantitatively. Our source code can be found at: https://github.com/aioz-ai/AudioDrivenStyleTransfer.

Voxceleb2: Deep Speaker Recognition

Abstract: The objective of this paper is speaker recognition under noisy and unconstrained conditions. ::: We make two key contributions. First, we introduce a very large-scale audio-visual speaker recognition dataset collected from open-source media. Using a fully automated pipeline, we curate VoxCeleb2 which contains over a million utterances from over 6,000 speakers. This is several times larger than any publicly available speaker recognition dataset. ::: Second, we develop and compare Convolutional Neural Network (CNN) models and training strategies that can effectively recognise identities from voice under various conditions. The models trained on the VoxCeleb2 dataset surpass the performance of previous works on a benchmark dataset by a significant margin.


A region-of-interest VAE framework is proposed in #REFR to generate video sequences that are consistent with the given simple text captions.

The methods in #OTHEREFR use semantic masks to inject motion information since objects from different categories have visually different motion patterns (e.g. cars, buildings).

Existing Imageto-Video methods can only achieve low-level control of the generated videos, thus they are not suitable to be used to generate videos with complex motions.

Text-to-Video methods aim at generating video sequences from text captions.

However, the appearance and motion information in the text caption is highly ambiguous leading to unavoidable uncertainties in generated videos.

Sync-Draw #OTHEREFR is the first framework proposed to solve Text-to-Video tasks.

Recently, GO-DIVA #OTHEREFR was proposed to generate open-domain videos from given text captions in an auto-regressive way.

A VQ-VAE #OTHEREFR is used in #OTHEREFR to represent the video content and generate visual tokens.

Then, three attention models across temporal, column, and row dimensions, are used to generate temporally consistent videos conditioned by the caption.

Given the ambiguous nature of the text, Text-to-Video methods can only achieve a low level of control over the generated videos.

As a result, the appearance and motions within generated videos are mostly determined by the training dataset.

Controllable Video Generation By Learning The Underlying Dynamical System With Neural Ode

Abstract: Videos depict the change of complex dynamical systems over time in the form of discrete image sequences. Generating controllable videos by learning the dynamical system is an important yet underexplored topic in the computer vision community. This paper presents a novel framework, TiV-ODE, to generate highly controllable videos from a static image and a text caption. Specifically, our framework leverages the ability of Neural Ordinary Differential Equations (Neural ODEs) to represent complex dynamical systems as a set of nonlinear ordinary differential equations. The resulting framework is capable of generating videos with both desired dynamics and content. Experiments demonstrate the ability of the proposed method in generating highly controllable and visually consistent videos, and its capability of modeling dynamical systems. Overall, this work is a significant step towards developing advanced controllable video generation models that can handle complex and dynamic scenes.

Sync-Draw: Automatic Video Generation Using Deep Recurrent Attentive Architectures

Abstract: This paper introduces a novel approach for generating videos called Synchronized Deep Recurrent Attentive Writer (Sync-DRAW). Sync-DRAW can also perform text-to-video generation which, to the best of our knowledge, makes it the first approach of its kind. It combines a Variational Autoencoder(VAE) with a Recurrent Attention Mechanism in a novel manner to create a temporally dependent sequence of frames that are gradually formed over time. The recurrent attention mechanism in Sync-DRAW attends to each individual frame of the video in sychronization, while the VAE learns a latent distribution for the entire video at the global level. Our experiments with Bouncing MNIST, KTH and UCF-101 suggest that Sync-DRAW is efficient in learning the spatial and temporal information of the videos and generates frames with high structural integrity, and can generate videos from simple captions on these datasets.


MotionStyleGAN We propose the image pair generator MotionStyleGAN built on top of StyleGAN2 #REFR to generate image pairs sharing the same contents and producing various motions.

We modify the official Pytorch implementation of StyleGAN2 from three output color channels (RGB) for a single image to six color channels for image pairs.

Although StyleGAN3 #OTHEREFR has been proposed to generate images that are fully equivariant to translation and rotation, we still choose StyleGAN2 as the baseline model of MotionStyleGAN since we find it difficult for StyleGAN3 to achieve equivariance of translation and rotation on the employed video datasets.

For future work, we plan to use StyleGAN3 for large-resolution human face video datasets like MEAD 1024 2 #OTHEREFR .

We train the proposed MotionStyleGAN on UCF101 256 2 #OTHEREFR , FaceForensics 256 2 #OTHEREFR , and SkyTimelapse 256 2 #OTHEREFR with gamma (R1 regularization weight, used in StyleGAN models) value of 1 and ADA augmentation method #OTHEREFR .

The training was done on ×4 24 GB NVidia GeForce RTX 3090 GPUs.

Motionvideogan: A Novel Video Generator Based On The Motion Space Learned From Image Pairs

Abstract: Video generation has achieved rapid progress benefiting from high-quality renderings provided by powerful image generators. We regard the video synthesis task as generating a sequence of images sharing the same contents but varying in motions. However, most previous video synthesis frameworks based on pre-trained image generators treat content and motion generation separately, leading to unrealistic generated videos. Therefore, we design a novel framework to build the motion space, aiming to achieve content consistency and fast convergence for video generation. We present MotionVideoGAN, a novel video generator synthesizing videos based on the motion space learned by pre-trained image pair generators. Firstly, we propose an image pair generator named MotionStyleGAN to generate image pairs sharing the same contents and producing various motions. Then we manage to acquire motion codes to edit one image in the generated image pairs and keep the other unchanged. The motion codes help us edit images within the motion space since the edited image shares the same contents with the other unchanged one in image pairs. Finally, we introduce a latent code generator to produce latent code sequences using motion codes for video generation. Our approach achieves state-of-the-art performance on the most complex video dataset ever used for unconditional video generation evaluation, UCF101. The source code is available on https://github.com/bbzhu-jy16/MotionVideoGAN.

Analyzing And Improving The Image Quality Of Stylegan

Abstract: The style-based GAN architecture (StyleGAN) yields state-of-the-art results in data-driven unconditional generative image modeling. We expose and analyze several of its characteristic artifacts, and propose changes in both model architecture and training methods to address them. In particular, we redesign the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images. In addition to improving image quality, this path length regularizer yields the additional benefit that the generator becomes significantly easier to invert. This makes it possible to reliably attribute a generated image to a particular network. We furthermore visualize how well the generator utilizes its output resolution, and identify a capacity problem, motivating us to train larger models for additional quality improvements. Overall, our improved model redefines the state of the art in unconditional image modeling, both in terms of existing distribution quality metrics as well as perceived image quality.


Our contribution complements many popular offline RL methods focused on distribution shift (e.g., BCQ #REFR ) and goes beyond pessimism-only methods by leveraging domain knowledge.

Lastly, we show that when sub-actions exhibit certain structures (e.g., when two sub-actions "reinforce" their independent effects), the linear approximation, though biased, can still lead to the optimal policy.

We test our approach in offline RL domains using a simulator #OTHEREFR and a real clinical dataset #OTHEREFR , where domain knowledge about the relationship among actions suggests our proposed factorization approach is applicable.

Empirically, our approach outperforms a non-factored baseline when the sample size is limited, even when the theoretical assumptions (around the validity of a linear decomposition) are not perfectly satisfied.

Qualitatively, in the real-data experiment, our approach learns policies that better capture the effect of less frequently observed treatment combinations.

Our work provides both theoretical insights and empirical evidence for RL practitioners to consider this simple linear decomposition for value-based RL approaches.

Compatible with any algorithm that has a Q-function component, we expect our approach will lead to gains for offline RL problems with combinatorial action spaces where data are limited and when domain knowledge can be used to check the validity of theoretical assumptions.

Leveraging Factored Action Spaces For Efficient Offline Reinforcement Learning In Healthcare

Abstract: Many reinforcement learning (RL) applications have combinatorial action spaces, where each action is a composition of sub-actions. A standard RL approach ignores this inherent factorization structure, resulting in a potential failure to make meaningful inferences about rarely observed sub-action combinations; this is particularly problematic for offline settings, where data may be limited. In this work, we propose a form of linear Q-function decomposition induced by factored action spaces. We study the theoretical properties of our approach, identifying scenarios where it is guaranteed to lead to zero bias when used to approximate the Q-function. Outside the regimes with theoretical guarantees, we show that our approach can still be useful because it leads to better sample efficiency without necessarily sacrificing policy optimality, allowing us to achieve a better bias-variance trade-off. Across several offline RL problems using simulators and real-world datasets motivated by healthcare, we demonstrate that incorporating factored action spaces into valuebased RL can result in better-performing policies. Our approach can help an agent make more accurate inferences within underexplored regions of the state-action space when applying RL to observational datasets.

Off-Policy Deep Reinforcement Learning Without Exploration

Abstract: Reinforcement learning traditionally considers the task of balancing exploration and exploitation. This work examines batch reinforcement learning--the task of maximally exploiting a given batch of off-policy data, without further data collection. We demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are only capable of learning with data correlated to their current policy, making them ineffective for most off-policy applications. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space to force the agent towards behaving on-policy with respect to a subset of the given data. We extend this notion to deep reinforcement learning, and to the best of our knowledge, present the first continuous control deep reinforcement learning algorithm which can learn effectively from uncorrelated off-policy data.