Herein, GRU is a type of recurrent neural networks (RNNs) with a gating mechanism #REFR .

4 , the actor network (proxy model) is established as a GRU-equipped DNN, which is designed to output proper actions (Var compensation) for inverters based on the historical trajectory (actor network's input).

Such refinement of RNN includes an update gate and a reset gate, which determine what information is allowed through to the output, and it can be trained to retain information over time.

With this simulated ability to remember information, GRU is used in this paper to process the time-series coupled variable-length historical trajectory as:

where denotes the GRU mapping rule, and represents its network parameter.

represents the GRU hidden state at time instant , which will be updated at each time step.

A Soft Coordination Method Of Heterogeneous Devices In Distribution System Voltage Control

Abstract: With the continuous increase of photovoltaic (PV) penetration, the voltage control interactions between newly installed PV inverters and previously deployed on-load tapchanger (OLTC) transformers become ever more significant. To achieve coordinated voltage regulation, current methods often rely on a decision-making algorithm to fully take over the control of all devices, requiring OLTC to give up its existing tap switching logic and execute corresponding upgrades. Aiming at bridging this gap, a soft coordination framework is proposed in this paper. Specifically, the decision-making commands are only applied on inverters, and OLTC that retains its own operation rule will be indirectly controlled by the changed system voltage, which is a result of appropriately adjusting inverters' Var output. The proposed method achieves the soft coordination by establishing a modified actor-critic algorithm to train a proxy model of inverters. The well-trained proxy model can properly adjust inverters' Var output to "softly" coordinate OLTC's tap operations, which finally attains coordinated voltage regulation and line loss minimization. Simulation results verify the superiority of our proposed method over traditional ones in coordinating heterogeneous devices for voltage control.

On The Properties Of Neural Machine Translation: Encoder-Decoder Approaches

Abstract: Neural machine translation is a relatively new approach to statistical machine translation based purely on neural networks. The neural machine translation models often consist of an encoder and a decoder. The encoder extracts a fixed-length representation from a variable-length input sentence, and the decoder generates a correct translation from this representation. In this paper, we focus on analyzing the properties of the neural machine translation using two models; RNN Encoder--Decoder and a newly proposed gated recursive convolutional neural network. We show that the neural machine translation performs relatively well on short sentences without unknown words, but its performance degrades rapidly as the length of the sentence and the number of unknown words increase. Furthermore, we find that the proposed gated recursive convolutional network learns a grammatical structure of a sentence automatically.


Contrary to that, the weights of the model trained with range-regularization follows a more uniform distribution as expected in #REFR .

To understand behavior of range-regularization, we analyse weight distribution change during the pre-training stage of ResNet-18 with and without range-regularization.

Figure 2 shows the weight distribution change of the first layer (conv1), first convolution of second residual block (block2.conv1), first downsampling layer in second residual block and the final linear layer of ResNet-18.

For the no range regularization case (original) we observe that all the demonstrated weights follow a normal distribution with outliers distributed wide apart.

We observe the same trend as shown in Table 6 .

The weight range for each layer with range-regularization is less than the model trained without range-regularization while both models achieve similar accuracy, 70.15% and 69.76% with and without range-regularization respectively, in full-precision evaluation.

Therefore, we demonstrate that range-regularization effectively removes the outliers in weights without notice- Table 6 it can also be observed that even the standard deviation of the weights for most of the layers are the same with and without range-regularization.

This indicates that the proposed approach doesn't cause a major shift in most of the weights, therefore the ensuring minimal regression to the accuracy of the pre-trained full precision model.

In addition, for ResNet-18, range regularization shows some effect in addressing over-fitting as seen from the pre-train full precision metrics in Table 1 .

R 2 : Range Regularization For Model Compression And Quantization

Abstract: Model parameter regularization is a widely used technique to improve generalization, but also can be used to shape the weight distributions for various purposes. In this work, we shed light on how weight regularization can assist model quantization and compression techniques, and then propose range regularization (R 2) to further boost the quality of model optimization by focusing on the outlier prevention. By effectively regulating the minimum and maximum weight values from a distribution, we mold the overall distribution into a tight shape so that model compression and quantization techniques can better utilize their limited numeric representation powers. We introduce L ∞ regularization, its extension margin regularization and a new soft-min-max regularization to be used as a regularization loss during full-precision model training. Coupled with state-of-the-art quantization and compression techniques, models trained with R 2 perform better on an average, specifically at lower bit weights with 16x compression ratio. We also demonstrate that R 2 helps parameter constrained models like MobileNetV1 achieve significant improvement of around 8% for 2 bit quantization and 7% for 1 bit compression.

Robust Quantization: One Model To Rule Them All

Abstract: Neural network quantization methods often involve simulating the quantization process during training. This makes the trained model highly dependent on the precise way quantization is performed. Since low-precision accelerators differ in their quantization policies and their supported mix of data-types, a model trained for one accelerator may not be suitable for another. To address this issue, we propose KURE, a method that provides intrinsic robustness to the model against a broad range of quantization implementations. We show that KURE yields a generic model that may be deployed on numerous inference accelerators without a significant loss in accuracy.


In #REFR , it is shown a real-world application of the chance-constrained MPC for dynamic obstacle avoidance.

MPC has been used for perching on power lines #OTHEREFR and agile flights #OTHEREFR .

The first work introducing perception awareness in MPC is #OTHEREFR where the authors propose to include a perception objective in the MPC to keep a point of interest in the camera field of view. Our MPC controller is inspired by #OTHEREFR .

However, we deal with two different perception objectives, line tracking, and collision avoidance, which conflict with each other when the drone approaches the power masts.

To enable collision avidance capabilities, in #OTHEREFR , the authors utilized a chanceconstrained MPC formulation.

This probabilistic collision constraint allows to account for the perceptual uncertainty and consequently enhances the obstacle avoidance robustness.

In #OTHEREFR , an MPC-based reactive planner for visual target tracking and obstacle avoidance is presented.

Different from our method, this MPC does not directly generate control commands but planned trajectories, which are tracked by another low-level controller.

Autonomous Power Line Inspection With Drones Via Perception-Aware Mpc

Abstract: The original rough reference trajectory is depicted in yellow and the drone deviates from it while avoiding obstacles (the power masts) and keeping the power line visible in the field of view of the onboard camera.

Robust Vision-Based Obstacle Avoidance For Micro Aerial Vehicles In Dynamic Environments

Abstract: In this paper, we present an on-board vision-based approach for avoidance of moving obstacles in dynamic environments. Our approach relies on an efficient obstacle detection and tracking algorithm based on depth image pairs, which provides the estimated position, velocity and size of the obstacles. Robust collision avoidance is achieved by formulating a chance-constrained model predictive controller (CC-MPC) to ensure that the collision probability between the micro aerial vehicle (MAV) and each moving obstacle is below a specified threshold. The method takes into account MAV dynamics, state estimation and obstacle sensing uncertainties. The proposed approach is implemented on a quadrotor equipped with a stereo camera and is tested in a variety of environments, showing effective on-line collision avoidance of moving obstacles.


We complement the finding of #REFR by showing that random crops increase not only translation invariance but also translation equivariance in the intermediate layers.

We train for 200 epochs using SGD with a batch size of 128 and a learning rate of 0.1 and a momentum of 0.9, L2 regularization at 0.0001 and weight decay at epochs 100 and 150 with a factor of 10.0.

In principle we expect the in-and equivariance to be the same since translation equivariance should be provided by the convolution.

However, we include equivariance in our experiments since there are works showing that the information about location can be encoded in different channels #OTHEREFR .

In Figure 6a we show learned translation equivariance for the tested data augmentations.

Random crops and Ran-dAugment increase the equivariance of learned features in the Middle, Late and Final parts, while the other data augmentation methods do not have any significant effect, with CutMix even having less equivariance than the baseline in the Middle part.

Also, we do not see any difference between invariance and equivariance for any data augmentation, which means that any equivariance learned is just invariance.

What Affects Learned Equivariance In Deep Image Recognition Models?

Abstract: Equivariance w.r.t. geometric transformations in neural networks improves data efficiency, parameter efficiency and robustness to out-of-domain perspective shifts. When equivariance is not designed into a neural network, the network can still learn equivariant functions from the data. We quantify this learned equivariance, by proposing an improved measure for equivariance. We find evidence for a correlation between learned translation equivariance and validation accuracy on ImageNet. We therefore investigate what can increase the learned equivariance in neural networks, and find that data augmentation, reduced model capacity and inductive bias in the form of convolutions induce higher learned equivariance in neural networks.

Quantifying Translation-Invariance In Convolutional Neural Networks

Abstract: A fundamental problem in object recognition is the development of image representations that are invariant to common transformations such as translation, rotation, and small deformations. There are multiple hypotheses regarding the source of translation invariance in CNNs. One idea is that translation invariance is due to the increasing receptive field size of neurons in successive convolution layers. Another possibility is that invariance is due to the pooling operation. We develop a simple a tool, the translation-sensitivity map, which we use to visualize and quantify the translation-invariance of various architectures. We obtain the surprising result that architectural choices such as the number of pooling layers and the convolution filter size have only a secondary effect on the translation-invariance of a network. Our analysis identifies training data augmentation as the most important factor in obtaining translation-invariant representations of images using convolutional neural networks.


On the contrary, despite tracing insufficient generalization to overfitting to the training data, occurrences of benign overfitting have been shown and characterized #REFR .

Whilst important for supervised-and unsupervised-learning tasks, said generalization is especially important for the safe application * Accepted for publication at IJCAI 2023 of reinforcement learning (RL), where unexpected observations may cause unintended and potentially unsafe behavior #OTHEREFR .

For instance, a shift might be present when transferring a robotics model from the training simulation to the real-world .

Generally, increased generalization from a few examples, also referred to as frew-shot learning, is induced by altering the training data to increase its diversity and impede overfitting .

Those data augmentation techniques have been successfully applied to both supervised and reinforcement learning by increasing the information used for training the model [Laskin et al., 2020] .

However, while the enlargement of the data may prevent overfitting and thus improve robustness to OOD samples, the increased complexity also increases training difficulty and hinders optimal convergence.

Also, the states observed by the agent might not resemble an optimal representation, an "issues that are so often critical to successful applications" of reinforcement learning #OTHEREFR .

Extending on these insights, we propose to apply Compact Reshaped Observation Processing (CROP) to reduce the observation such that presumably irrelevant details are removed and the remaining information is sufficient for learning a robust policy that generalizes to unseen shifted observations. Overall, we provide the following contribution:

• We formulate three concrete CROP methods applicable to fully observable state and action spaces, reducing the information with regards to the agent's position within the environment, the agent's action space and affect, and the objects within the environment (Figure 1 ).

• We provide proof-of-concept and show the strengths of each CROP over the full observation in a safety gridworld training environment and a shifted test environment.

• We evaluate the proposed CROPs and compare their effect in zero-shot generalization to the state-of the art of data augmentation in randomly generated mazes.

Crop: Towards Distributional-Shift Robust Reinforcement Learning Using Compact Reshaped Observation Processing *

Abstract: The safe application of reinforcement learning (RL) requires generalization from limited training data to unseen scenarios. Yet, fulfilling tasks under changing circumstances is a key challenge in RL. Current state-of-the-art approaches for generalization apply data augmentation techniques to increase the diversity of training data. Even though this prevents overfitting to the training environment(s), it hinders policy optimization. Crafting a suitable observation, only containing crucial information, has been shown to be a challenging task itself. To improve data efficiency and generalization capabilities, we propose Compact Reshaped Observation Processing (CROP) to reduce the state information used for policy optimization. By providing only relevant information, overfitting to a specific training layout is precluded and generalization to unseen environments is improved. We formulate three CROPs that can be applied to fully observable observation-and action-spaces and provide methodical foundation. We empirically show the improvements of CROP in a distributionally shifted safety gridworld. We furthermore provide benchmark comparisons to full observability and data-augmentation in two different-sized procedurally generated mazes.

Benign Overfitting In Linear Regression

Abstract: The phenomenon of benign overfitting is one of the key mysteries uncovered by deep learning methodology: deep neural networks seem to predict well, even with a perfect fit to noisy training data. Motivated by this phenomenon, we consider when a perfect fit to training data in linear regression is compatible with accurate prediction. We give a characterization of gaussian linear regression problems for which the minimum norm interpolating prediction rule has near-optimal prediction accuracy. The characterization is in terms of two notions of the effective rank of the data covariance. It shows that overparameterization is essential for benign overfitting in this setting: the number of directions in parameter space that are unimportant for prediction must significantly exceed the sample size.


In the field of action localization, P-GCN #REFR constructs a graph of proposals by establishing the edges and then applying GCNs to do message aggregation among proposals. For 3D object detection, Feng et al.

Modeling proposal relations has emerged in many fields, including 2D #OTHEREFR and 3D object detection #OTHEREFR .

#OTHEREFR extract uniform appearance features for each 3D object proposal and construct a relation graph that exploits the 3D object-object relationships.

In order to explore and utilize object-to-object relations, RDN #OTHEREFR assembles and propagates object relations to enhance object features for video object detection.

Despite the rapid development of modeling the proposal relations of 2D or 3D bounding boxes, the relations between lane proposals have not been well exploited yet.

In our work, we aim to mine the location and score relations between proposals to generate diverse and discriminative proposals.

Dense Hybrid Proposal Modulation For Lane Detection

Abstract: In this paper, we present a dense hybrid proposal modulation (DHPM) method for lane detection. Most existing methods perform sparse supervision on a subset of high-scoring proposals, while other proposals fail to obtain effective shape and location guidance, resulting in poor overall quality. To address this, we densely modulate all proposals to generate topologically and spatially high-quality lane predictions with discriminative representations. Specifically, we first ensure that lane proposals are physically meaningful by applying single-lane shape and location constraints. Benefitting from the proposed proposal-to-label matching algorithm, we assign each proposal a target ground truth lane to efficiently learn from spatial layout priors. To enhance the generalization and model the inter-proposal relations, we diversify the shape difference of proposals matching the same ground-truth lane. In addition to the shape and location constraints, we design a quality-aware classification loss to adaptively supervise each positive proposal so that the discriminative power can be further boosted. Our DHPM achieves very competitive performances on four popular benchmark datasets. Moreover, we consistently outperform the baseline model on most metrics without introducing new parameters and reducing inference speed. The codes of our method are available at https://github.com/wuyuej/DHPM.

Graph Convolutional Networks For Temporal Action Localization

Abstract: Most state-of-the-art action localization systems process each action proposal individually, without explicitly exploiting their relations during learning. However, the relations between proposals actually play an important role in action localization, since a meaningful action always consists of multiple proposals in a video. In this paper, we propose to exploit the proposal-proposal relations using Graph Convolutional Networks (GCNs). First, we construct an action proposal graph, where each proposal is represented as a node and their relations between two proposals as an edge. Here, we use two types of relations, one for capturing the context information for each proposal and the other one for characterizing the correlations between distinct actions. Then we apply the GCNs over the graph to model the relations among different proposals and learn powerful representations for the action classification and localization. Experimental results show that our approach significantly outperforms the state-of-the-art on THUMOS14 (49.1% versus 42.8%). Moreover, augmentation experiments on ActivityNet also verify the efficacy of modeling action proposal relationships. Codes are available at this https URL.


As an efficient alternative to full fine-tuning, Adapter-Tuning #REFR was first proposed to transfer large pre-trained Language models to downstream tasks.

Inspired by textual prompting, Prompt-Tuning #OTHEREFR and Prefix-Tuning #OTHEREFR insert learnable tokens to adapt to the new task.

More advanced methods #OTHEREFR achieved comparable or superior performance to full finetuning, and keep the same inference cost by merging the additional learnable parameters to the original pre-trained model.

Following the step of successful Vision Transformers #OTHEREFR , VPT #OTHEREFR and AdapterFormer #OTHEREFR have been proposed to solve visual transfer learning problems.

Prompt-Tuning and Prefix-Tuning depend on the transformer architecture because they modify input or hidden tokens.

Adapter and its variants are network architecture generalizable since they are new modules that can be implemented in forms compatible with pre-trained models.

A Unified Continual Learning Framework With General Parameter-Efficient Tuning

Abstract: The "pre-training → downstream adaptation" presents both new opportunities and challenges for Continual Learning (CL). Although the recent state-of-the-art in CL is achieved through Parameter-Efficient-Tuning (PET) adaptation paradigm, only prompt has been explored, limiting its application to Transformers only. In this paper, we position prompting as one instantiation of PET, and propose a unified CL framework with general PET, dubbed as Learning-Accumulation-Ensemble (LAE). PET, e.g., using Adapter, LoRA, or Prefix, can adapt a pre-trained model to downstream tasks with fewer parameters and resources. Given a PET method, our LAE framework incorporates it for CL with three novel designs. 1) Learning: the pre-trained model adapts to the new task by tuning an online PET module, along with our adaptation speed calibration to align different PET modules, 2) Accumulation: the task-specific knowledge learned by the online PET module is accumulated into an offline PET module through momentum update, 3) Ensemble: During inference, we respectively construct two experts with online/offline PET modules (which are favored by the novel/historical tasks) for prediction ensemble. We show that LAE is compatible with a battery of PET methods and gains strong CL capability. For example, LAE with Adaptor PET surpasses the prior state-ofthe-art by 1.3% and 3.6% in last-incremental accuracy on CIFAR100 and ImageNet-R datasets, respectively.

Parameter-Efficient Transfer Learning For Nlp

Abstract: Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.


Besides, #REFR proposed to directly manipulate the neuron values to ameliorate the model performance, and similar strategy has been adopted by us to verify the functionality of neurons.

One of the most commonly used strategy utilizes visualization to observe the dynamics of hidden states for each time-step #OTHEREFR .

In #OTHEREFR , it shows that counting behavior can be achieved much easier in LSTM, while in other variants such as GRU, the dynamics of most neuron values seem to be irregular.

Our findings can be considered as offering an explanation how GRU can perform counting even though each individual neuron's value does not seem to have strong correlation with the counts.

There have been some works utilizing diagnostic classifier #OTHEREFR , Pearson Correlation #OTHEREFR , and ablation test to probe the functions of neurons, which shares similar spirits to the first strategy we used (the only difference is that we further select a subset of dominant features).

Nevertheless, as we have pointed out, simply performing diagnostic classification or correlation analysis cannot directly confirm the hypothesis, and the third stage of our solution (verification by manipulating neuron values) is the key factor to verify the functions of neurons.

Nevertheless, we believe the proposed 3-stage strategy can be a general mechanism that researchers can adopt to confirm the hypothesis of functions of neurons.

Exposing The Functionalities Of Neurons For Gated Recurrent Unit Based Sequence-To-Sequence Model

Abstract: The goal of this paper is to report certain scientific discoveries about a Seq2Seq model. It is known that analyzing the behavior of RNN-based models at the neuron level is considered a more challenging task than analyzing a DNN or CNN models due to their recursive mechanism in nature. This paper aims to provide neuron-level analysis to explain why a vanilla GRU-based Seq2Seq model without attention can achieve token-positioning. We found four different types of neurons: storing, counting, triggering, and outputting and further uncover the mechanism for these neurons to work together in order to produce the right token in the right position. Preprint. Under review.

Identifying And Controlling Important Neurons In Neural Machine Translation

Abstract: Neural machine translation (NMT) models learn representations containing substantial linguistic information. However, it is not clear if such information is fully distributed or if some of it can be attributed to individual neurons. We develop unsupervised methods for discovering important neurons in NMT models. Our methods rely on the intuition that different models learn similar properties, and do not require any costly external supervision. We show experimentally that translation quality depends on the discovered neurons, and find that many of them capture common linguistic phenomena. Finally, we show how to control NMT translations in predictable ways, by modifying activations of individual neurons.


Factoid extractive QA datasets like SQuAD #REFR , come up with a challenge to measure a QA model competency in identifying the span of an answer from a context for factoid questions.

In the latter approach, question-answer pairs are formulated using language generation models, machine translation, or manual/learned templates.

The main challenge in automatic generation is gold answer extraction. Mostly accomplished using existing QA models.

But getting a dependable model, as perfect as a human, that can produce a correct answer is challenging.

So, to minimize the generation of trivial and un-grammatical question-1 Amharic is written using Ge'ez script known as ፊደል (Fidel) answer pairs, aside from improving the performance of the generation models, experts paraphrase the generated question-answer pairs #OTHEREFR .

The distinction between the existing datasets lies in the question types (factoid vs non-factoid) and answer formulation sub-task (extractive vs abstractive).

Factoid questions like 'What is the capital city of Ethiopia?' (Answer: Addis Ababa) seeks a factual answer that appears as a named entity such as date, location, proper noun, other short noun phrases, or short sentence.

Unlike that, abstractive QA datasets contain questions whose answer is a comprehension of a context, not a direct copy #OTHEREFR .

Recently, the QA field of study is getting too many datasets in mono, cross, and multi-lingual settings #OTHEREFR .

However, Amharic 1 is not included yet in the map of the QA datasets.

Specific to Amharic there are attempts to develop datasets for other Natural Language Processing (NLP) tasks like sentiment analysis #OTHEREFR , morphologically annotated corpus #OTHEREFR , contemporary Amharic corpus #OTHEREFR , and parallel corpora for machine translation #OTHEREFR .

Amqa: Amharic Question Answering Dataset

Abstract: Question Answering (QA) returns concise answers or answer lists from natural language text given a context document. To advance robust models' development, large amounts of resources go into curating QA datasets. There is a surge of QA datasets for languages like English, however, this is not the case for Amharic. Amharic, the official language of Ethiopia, is the second most spoken Semitic language in the world. There is no published or publicly available Amharic QA dataset. Hence, to foster the research in Amharic QA, we present the first Amharic QA (AmQA) dataset. We crowdsourced 2628 question-answer pairs over 378 Wikipedia articles. Additionally, we run an XLMRLarge-based baseline model to spark opendomain QA research interest. The best-performing baseline achieves an F-score of 69.58 and 71.74 in reader-retriever QA and reading comprehension settings respectively.

Squad: 100,000+ Questions For Machine Comprehension Of Text

Abstract: We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. ::: The dataset is freely available at this https URL


Since the information of all APs and users are required in cell-free massive MIMO, it is suggested in literature to serve each user by a subset of APs to limit data sharing and computational complexity via two networkcentric and user-centric approaches #REFR .

In #OTHEREFR , three methods for user scheduling has been presented to optimize the ergodic sum-rate.

A joint user scheduling and transceiver design scheme for cross-link interference suppression that works on per-frame basis in interfering massive MIMO multicell scenarios has been reported in #OTHEREFR .

In #OTHEREFR , a userselection method based on simple zero-forcing beamforming with selection (ZFS) has been proposed which attains a sig-nificant fraction of sum capacity and throughput of optimal method.

Unlike multicell massive MIMO systems where users in each cell are served by a BS, a network with the same area of cells including randomly located single-antenna access points (APs) serving all users in the same time-frequency resource was introduced in #OTHEREFR , #OTHEREFR , #OTHEREFR and called cell-free massive MIMO which presents higher throughput compared with small cells.

Cell-free massive MIMO is compared with cellular massive MIMO systems in #OTHEREFR , and it is shown that cell-free systems can provide significantly higher spectral efficiency for all users.

The first approach divides the APs into several clusters, where each includes different APs from other clusters.

The latter, considers the AP subset providing the best channel conditions for each user which could be dynamic.

In this paper, we investigate the sum-rate performance of the multicell and cell-free massive MIMO systems using greedy user scheduling based on ZFS algorithm in the downlink #OTHEREFR .

We consider the use of a network-centric clustering approach to reduce the computational complexity of the techniques applied to the cell-free systems #OTHEREFR .

Study Of Multiuser Scheduling With Enhanced Greedy Techniques For Multicell And Cell-Free Massive Mimo Networks

Abstract: In this work, we investigate the sum-rate performance of multicell and cell-free massive MIMO systems using linear precoding and multiuser scheduling algorithms. We consider the use of a network-centric clustering approach to reduce the computational complexity of the techniques applied to the cellfree system. We then develop a greedy algorithm that considers multiple candidates for the subset of users to be scheduled and that approaches the performance of the optimal exhaustive search. We assess the proposed and existing scheduling algorithms in both multicell and cell-free networks with the same coverage area. Numerical results illustrate the sum-rate performance of the proposed scheduling algorithm against existing approaches.

Scalable Cell-Free Massive Mimo Systems

Abstract: Imagine a coverage area with many wireless access points that cooperate to jointly serve the users, instead of creating autonomous cells. Such a cell-free network operation can potentially resolve many of the interference issues that appear in current cellular networks. This ambition was previously called Network MIMO (multiple-input multiple-output) and has recently reappeared under the name Cell-Free Massive MIMO. The main challenge is to achieve the benefits of cell-free operation in a practically feasible way, with computational complexity and fronthaul requirements that are scalable to large networks with many users. We propose a new framework for scalable Cell-Free Massive MIMO systems by exploiting the dynamic cooperation cluster concept from the Network MIMO literature. We provide algorithms for initial access, pilot assignment, cluster formation, precoding, and combining that are proved to be scalable. Interestingly, the proposed scalable precoding and combining outperform conventional maximum ratio processing and also performs closely to the best unscalable alternatives.


In our implementation, the Pytorch #REFR auto-differentiation package is used to calculate the gradients.

A key insight is that with real-world traffic, one may expect certain patterns in this mapping; for example, if two histories of traffic conditions are very similar, their corresponding optimal TE configurations should also be similar. However, measuring and explicitly quantifying such similarities is nontrivial.

Our approach is to exploit deep neural networks, which have demonstrated remarkable success in identifying complex patterns in high dimensional data, for this task.

L(D i t , π θ (D i t−1 , . . . , D i t−H )

) can be calculated in closed form via backpropagation.

Dote: Rethinking (Predictive) Wan Traffic Engineering

Abstract: We explore a new design point for traffic engineering on wide-area networks (WANs): directly optimizing traffic flow on the WAN using only historical data about traffic demands. Doing so obviates the need to explicitly estimate, or predict, future demands. Our method, which utilizes stochastic optimization, provably converges to the global optimum in well-studied theoretical models. We employ deep learning to scale to large WANs and real-world traffic. Our extensive empirical evaluation on real-world traffic and network topologies establishes that our approach's TE quality almost matches that of an (infeasible) omniscient oracle, outperforming previously proposed approaches, and also substantially lowers runtimes.

Pytorch: An Imperative Style, High-Performance Deep Learning Library

Abstract: Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it provides an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. ::: In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. ::: We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several common benchmarks.


We conducted our experiments on two visual model architectures that currently perform best on RVL-CDIP with less than a hundred million of parameters, as far as we know #REFR : an EfficientNetB0 and a ResNet50. According to the method of Ferrando et al.

#OTHEREFR , who observed the best accuracy of both models on our classification task, we initialized these models with weights that have been pre-trained on the ImageNet classification task.

We implemented preprocessing steps, that precede the model backbone, with the aim of improving their robustness.

The first of these reactive defenses we implemented is the JPEG preprocessing.

We compress the document image into the JPEG format, then decompress each image that is given as input of the model backbone (JPEG defense). This transformation has been identified by Dsiugaite et al. #OTHEREFR as a factor of robustness against adversarial attacks.

Evaluating Adversarial Robustness On Document Image Classification

Abstract: Adversarial attacks and defenses have gained increasing interest on computer vision systems in recent years, but as of today, most investigations are limited to natural images. However, many artificial intelligence models actually handle documentary data, which is very different from real world images. Hence, in this work, we try to apply the adversarial attack philosophy on documentary data and to protect models against such attacks. Our methodology is to implement untargeted gradient-based, transfer-based and score-based attacks and evaluate the impact of defenses such as adversarial training, JPEG input compression and grey-scale input transformation on the robustness of ResNet50 and EfficientNetB0 model architectures. To the best of our knowledge, no such work has been conducted by the community in order to study the impact of these attacks on the document image classification task.

On Evaluating Adversarial Robustness

Abstract: Correctly evaluating defenses against adversarial examples has proven to be extremely difficult. Despite the significant amount of recent work attempting to design defenses that withstand adaptive attacks, few have succeeded; most papers that propose defenses are quickly shown to be incorrect. ::: We believe a large contributing factor is the difficulty of performing security evaluations. In this paper, we discuss the methodological foundations, review commonly accepted best practices, and suggest new methods for evaluating defenses to adversarial examples. We hope that both researchers developing defenses as well as readers and reviewers who wish to understand the completeness of an evaluation consider our advice in order to avoid common pitfalls.


Fixed Cost Functions And Time-Varying Demand

Finally, we note that the regret guarantee obtained in Theorem 3 compares favorably to classical O( √ T ) regret guarantees in the OCO or MAB literature #REFR .

Finally, choosing γ = 1 √ T achieves an optimal balance (up to logarithmic terms) between the two quantities above, i.e., O(γ −1 log log T ) and O(γT ), which establishes the O( √ T log log T ) regret bound.

For a complete proof of Theorem 3, see Appendix B.

We reiterate that Theorem 3 applies to strongly convex cost functions as with Theorem 1 and that extending this result to general convex cost functions, e.g., linear functions, is, in general, not possible (see Example 1 in Section 4.2).

Furthermore, compared to the regret guarantee obtained in Theorem 1, Theorem 3 establishes that the time-varying nature of the customer demand incurs an additional factor of O( √ T ) in the regret guarantee as compared to the setting with fixed demands.

However, we do note that if the set of demand realizations D is known a priori to be o( √ T ), then the regret guarantee in Theorem 2 can be improved to O(|D| log log T ) by running the direct extension of Algorithm 1, wherein a feasible price interval is maintained for each realized demand.

Online Learning For Equilibrium Pricing In Markets Under Incomplete Information

Abstract: The study of market equilibria is central to economic theory, particularly in efficiently allocating scarce resources. However, the computation of equilibrium prices at which the supply of goods matches their demand typically relies on having access to complete information on private attributes of agents, e.g., suppliers' cost functions, which are often unavailable in practice. Motivated by this practical consideration, we consider the problem of setting equilibrium prices in the incomplete information setting wherein a market operator seeks to satisfy the customer demand for a commodity by purchasing the required amount from competing suppliers with privately known cost functions unknown to the market operator. In this incomplete information setting, we consider the online learning problem of learning equilibrium prices over time while jointly optimizing three performance metricsunmet demand, cost regret, and payment regret-pertinent in the context of equilibrium pricing over a horizon of T periods. We first consider the setting when suppliers' cost functions are fixed and develop algorithms that achieve a regret of O(log log T) when the customer demand is constant over time, or O(√ T log log T) when the demand is variable over time. Next, we consider the setting when the suppliers' cost functions can vary over time and illustrate that no online algorithm can achieve sublinear regret on all three metrics when the market operator has no information about how the cost functions change over time. Thus, we consider an augmented setting wherein the operator has access to hints/contexts that, without revealing the complete specification of the cost functions, reflect the variation in the cost functions over time and propose an algorithm with sublinear regret in this augmented setting.

Introduction To Online Convex Optimization

Abstract: This monograph portrays optimization as a process. In many practical applications the environment is so complex that it is infeasible to lay out a comprehensive theoretical model and use classical algorithmic theory and mathematical optimization. It is necessary as well as beneficial to take a robust approach, by applying an optimization method that learns as one goes along, learning from experience as more aspects of the problem are observed. This view of optimization as a process has become prominent in varied fields and has led to some spectacular success in modeling and systems that are now part of our daily lives.


We choose adaptive instance normalization #REFR to build the AIN module due to its efficiency and compact representation. Fig. 3 shows the architecture of the AIN module.

For a given input hyperspectral image, the proposed model first downsamples the input image into 1/2 and 1/4 scales by using Gaussian kernels.

Shallow features are extracted by multiple parallel convolutions, as illustrated in Fig. 2 (the initial layer).

Next, the coarse-fusion network extracts deep features and fuses the multiscale information through several parallel adaptive instance (AIN) modules.

The motivations for designing the coarse-fusion network are twofold: 1) The multiscale structure presents a solution to increase the receptive field to capture more content.

2) The AIN module can transfer the basic structures from the low-resolution feature maps to the high-resolution ones.

The adaptive instance normalization affine transforms the normalized feature map h ∈ R H×W ×C by taking an input

Here H and W denote the height and width of the feature map, respectively. C is the number of channels.

It should be noted that h is the feature from the downscale.

Multi-Scale Adaptive Fusion Network For Hyperspectral Image Denoising

Abstract: Removing the noise and improving the visual quality of hyperspectral images (HSIs) is challenging in academia and industry. Great efforts have been made to leverage local, global or spectral context information for HSI denoising. However, existing methods still have limitations in feature interaction exploitation among multiple scales and rich spectral structure preservation. In view of this, we propose a novel solution to investigate the HSI denoising using a Multi-scale Adaptive Fusion Network (MAFNet), which can learn the complex nonlinear mapping between clean and noisy HSI. Two key components contribute to improving the hyperspectral image denoising: A progressively multiscale information aggregation network and a co-attention fusion module. Specifically, we first generate a set of multiscale images and feed them into a coarse-fusion network to exploit the contextual texture correlation. Thereafter, a fine fusion network is followed to exchange the information across the parallel multiscale subnetworks. Furthermore, we design a co-attention fusion module to adaptively emphasize informative features from different scales, and thereby enhance the discriminative learning capability for denoising. Extensive experiments on synthetic and real HSI datasets demonstrate that the proposed MAFNet has achieved better denoising performance than other state-of-the-art techniques. Our codes are available at https://github.com/summitgao/MAFNet.

Arbitrary Style Transfer In Real-Time With Adaptive Instance Normalization

Abstract: Gatys et al. recently introduced a neural algorithm that renders a content image in the style of another image, achieving so-called style transfer. However, their framework requires a slow iterative optimization process, which limits its practical application. Fast approximations with feed-forward neural networks have been proposed to speed up neural style transfer. Unfortunately, the speed improvement comes at a cost: the network is usually tied to a fixed set of styles and cannot adapt to arbitrary new styles. In this paper, we present a simple yet effective approach that for the first time enables arbitrary style transfer in real-time. At the heart of our method is a novel adaptive instance normalization (AdaIN) layer that aligns the mean and variance of the content features with those of the style features. Our method achieves speed comparable to the fastest existing approach, without the restriction to a pre-defined set of styles. In addition, our approach allows flexible user controls such as content-style trade-off, style interpolation, color & spatial controls, all using a single feed-forward neural network.


MixUp #REFR is a mixing approach that linearly combines two data samples as well as the labels.

Gaussian patch augmentation provides better robustness to corruptions while keeping the accuracy unaffected.

SaliencyMix #OTHEREFR utilizes salient information to select the dominant patch in a data sample.

The patch is then used to replace a patch in a different random data sample.

ResizeMix #OTHEREFR is a similar approach to CutMix, however, instead of replacing the source image patch with a random patch, it uses a whole random, resized data sample.

The authors make the argument that salient information used by some augmentations for guidance is redundant and outperformed by their re-scaling -pasting into a random position algorithm.

The ratio between mixed samples is controlled by the λ parameter sampled from Beta distribution.

Mixup reduces generalization error, memorization of corrupted labels, and improves GAN training #OTHEREFR .

Puzzle Mix #OTHEREFR combines two images and their labels to create a new data sample.

The method preserves a local image statistic and at the same time maximizes the amount of saliency information exposed in the resulting data sample.

Super-Mix #OTHEREFR generalizes mixing up to k images using supervised learning to preserve saliency information.

Unproportional Mosaicing

Abstract: Data shift is a gap between data distribution used for training and data distribution encountered in the real-world. Data augmentations help narrow the gap by generating new data samples, increasing data variability, and data space coverage. We present a new data augmentation: Unproportional mosaicing (Unprop). Our augmentation randomly splits an image into various-sized blocks and swaps its content (pixels) while maintaining block sizes. Our method achieves a lower error rate when combined with other state-of-the-art augmentations. Keywords image augmentation • convolutional neural network • image processing

Mixup: Beyond Empirical Risk Minimization

Abstract: Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.


LiDAR-Flow #REFR fuses the LiDAR with a stereo camera to boost the cost computation in the matching process for both stereo and optical flow.

However, #OTHEREFR heavily rely on a rigid motion prior for moving vehicles, which does not apply to debris flows. #OTHEREFR suffer from overfitting to specific camera intrinsics.

Their networks are trained on KITTI #OTHEREFR with known camera intrinsics, but encounter problems when applied to other datasets, e.g., nuScenes #OTHEREFR or our debris-flow data.

An intuitive way to disambiguate depth scaling is to incorporate direct 3D measurements into image-based pipelines.

Many works have thus focused on sensor fusion for scene flow estimation.

Subsequent work like DeepLiDARFlow #OTHEREFR encapsulates the solution into an end-to-end deep neural network that regresses scene flow from consecutive LiDAR scans and monocular images.

CamLiFlow #OTHEREFR mitigates the sen-sor fusion problem with a multi-stage fusion framework and estimates point-wise motion for sparse LiDAR point clouds.

Two-frame methods #OTHEREFR do not account for the temporal continuity across multiple consecutive frames.

In contrast, #OTHEREFR enforces consistency for both spatial and temporal neighbors in a sliding temporal window, #OTHEREFR leverages multi-frame consistency to detect moving object regions, and #OTHEREFR introduces a convolutional LSTM on top of #OTHEREFR to propagate the hidden states via forward warping, but requires stereo images as supervision. Huang et al.

Deflow: Self-Supervised 3D Motion Estimation Of Debris Flow

Abstract: Existing work on scene flow estimation focuses on autonomous driving and mobile robotics, while automated solutions are lacking for motion in nature, such as that exhibited by debris flows. We propose DEFLOW, a model for 3D motion estimation of debris flows, together with a newly captured dataset. We adopt a novel multi-level sensor fusion architecture and self-supervision to incorporate the inductive biases of the scene. We further adopt a multi-frame temporal processing module to enable flow speed estimation over time. Our model achieves state-of-the-art optical flow and depth estimation on our dataset, and fully automates the motion estimation for debris flows. Source code and dataset are available at project page.

Lidar-Flow: Dense Scene Flow Estimation From Sparse Lidar And Stereo Images

Abstract: We propose a new approach called LiDAR-Flow to robustly estimate a dense scene flow by fusing a sparse LiDAR with stereo images. We take the advantage of the high accuracy of LiDAR to resolve the lack of information in some regions of stereo images due to textureless objects, shadows, ill-conditioned light environment and many more. Additionally, this fusion can overcome the difficulty of matching unstructured 3D points between LiDAR-only scans. Our LiDAR-Flow approach consists of three main steps; each of them exploits LiDAR measurements. First, we build strong seeds from LiDAR to enhance the robustness of matches between stereo images. The imagery part seeks the motion matches and increases the density of scene flow estimation. Then, a consistency check employs LiDAR seeds to remove the possible mismatches. Finally, LiDAR measurements constraint the edge-preserving interpolation method to fill the remaining gaps. In our evaluation we investigate the individual processing steps of our LiDAR-Flow approach and demonstrate the superior performance compared to image-only approach.


We utilize Proximal Policy Optimization (PPO) #REFR to find the optimal control policy π PACER .

To enable full-body pedestrian simulation, we design the Pedestrian Animation ControllER (PACER) to execute the 2D trajectories generated by TRACE in a physics simulator. Background: Goal-Conditioned RL. Our framework ( Fig.

3) follows the general goal-conditioned reinforcement learning framework, where a goal-conditioned policy π PACER is trained to follow 2D target trajectories specified by τ s .

The task is formulated as a Markov Decision Process (MDP) defined by a tuple M = S, A, T , R, γ of states, actions, transition dynamics, reward function, and discount factor.

The state S, transition dynamics T , and reward R are calculated by the environment based on the current simulation and goal, while the action A is computed by the policy π PACER .

The policy's objective is to maximize the discounted return E T t=1 γ t−1 r t where r t is the reward per timestep.

To create a controller that can simulate crowds in realistic 3D scenes (e.g. scans, neural reconstructions, or artist-created meshes (Fig.

1) ), our humanoid must be terrain aware, socially aware of other agents, and support diverse body types.

We use a humanoid model that conforms to the kinematic structure of SMPL #OTHEREFR , and is automatically generated using a procedure similar to #OTHEREFR .

Our control policy π PACER (a t |h t , o t , β, τ s ) is conditioned on the state of the simulated character h t , environmental features o t , body type β, and goal trajectory τ s .

Trace And Pace: Controllable Pedestrian Animation Via Guided Trajectory Diffusion

Abstract: Figure 1. (Left) We propose TRACE, a trajectory diffusion model that enables user control through test-time guidance. (Right) Generated trajectories are passed to a novel physics-based humanoid controller (PACER), forming a closed-loop pedestrian animation system.

Proximal Policy Optimization Algorithms

Abstract: We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.

optimal control policy
Proximal Policy Optimization


Specifically, we used Bert-base #REFR and Roberta-base (Liu et al.) , for SNLI and ANLI, respectively.

The data consists of 162,865/3,200/3,200 sentence pairs for train/val/test set respectively.

The max length of the input is set to 128.

For the Image Classification datasets, we used Convolutional Neural Networks (CNNs).

Specifically, we used LeNet5 #OTHEREFR , ResNet50, ResNet34, and ResNet18 #OTHEREFR , for MNIST, CIFAR10, CIFAR100, and ImageNet100, respectively.

For the NLI datasets, we finetuned transformer based Pre-trained Language Models (PLMs).

Esd: Expected Squared Difference As A Tuning-Free Trainable Calibration Measure

Abstract: Studies have shown that modern neural networks tend to be poorly calibrated due to over-confident predictions. Traditionally, post-processing methods have been used to calibrate the model after training. In recent years, various trainable calibration measures have been proposed to incorporate them directly into the training process. However, these methods all incorporate internal hyperparameters, and the performance of these calibration objectives relies on tuning these hyperparameters, incurring more computational costs as the size of neural networks and datasets become larger. As such, we present Expected Squared Difference (ESD), a tuning-free (i.e., hyperparameter-free) trainable calibration objective loss, where we view the calibration error from the perspective of the squared difference between the two expectations. With extensive experiments on several architectures (CNNs, Transformers) and datasets, we demonstrate that (1) incorporating ESD into the training improves model calibration in various batch size settings without the need for internal hyperparameter tuning, (2) ESD yields the best-calibrated results compared with previous approaches, and (3) ESD drastically improves the computational costs required for calibration during training due to the absence of internal hyperparameter. The code is publicly accessible at https://github.com/heesuk-yoon/ESD.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


Calibration #REFR : A classifier is said to satisfy calibration if both protected and unprotected groups have almost similar positive predictive values (PPV).

If the difference SP m,k − SP m ,k is greater than 0, then the protected group is benefited.

On the other hand, if the difference is less than 0, the unprotected group is benefited.

Note that the statistical parity of any system can be directly measured from the system's outcome labels, without any need for an alternative classifier.

However, this is not the case with other statistical fairness metrics.

The PPV represents the probability of an individual with a positive prediction actually experiencing a positive outcome.

Formally, if C m,k = P(z = k | y = k, x ∈ X m ) denote the positive predictive rate for the group X m , the calibration of the system g with respect to the classifier f is computed as

Accuracy Equality #OTHEREFR : This statistical measure computes the probability that both the classifiers g and f yielding the same label.

Towards Inclusive Fairness Evaluation Via Eliciting Disagreement Feedback From Non-Expert Stakeholders

Abstract: Traditional algorithmic fairness notions rely on label feedback, which can only be elicited from expert critics. However, in most practical applications, several non-expert stakeholders also play a major role in the system and can have distinctive opinions about the decision making philosophy. For example, in kidney placement programs, transplant surgeons are very wary about accepting kidney offers for black patients due to genetic reasons. However, non-expert stakeholders in kidney placement programs (e.g. patients, donors and their family members) may misinterpret such decisions from the perspective of social discrimination. This paper evaluates group fairness notions from the viewpoint of non-expert stakeholders, who can only provide binary agreement/disagreement feedback regarding the decision in context. Specifically, two types of group fairness notions have been identified: (i) definite notions (e.g. calibration), which can be evaluated exactly using disagreement feedback, and (ii) indefinite notions (e.g. equal opportunity) which suffer from uncertainty due to lack of label feedback. In the case of indefinite notions, bounds are presented based on disagreement rates, and an estimate is constructed based on established bounds. The efficacy of all our findings are validated empirically on real human feedback dataset. This work is supported by an NSF grant...

Fair Prediction With Disparate Impact: A Study Of Bias In Recidivism Prediction Instruments

Abstract: Recidivism prediction instruments (RPI's) provide decision makers with an assessment of the likelihood that a criminal defendant will reoffend at a future point in time. While such instruments are gaining increasing popularity across the country, their use is attracting tremendous controversy. Much of the controversy concerns potential discriminatory bias in the risk assessments that are produced. This paper discusses several fairness criteria that have recently been applied to assess the fairness of recidivism prediction instruments. We demonstrate that the criteria cannot all be simultaneously satisfied when recidivism prevalence differs across groups. We then show how disparate impact can arise when a recidivism prediction instrument fails to satisfy the criterion of error rate balance.


B. Semi-Decentralized Equilibrium Seeking

In particular, we adopt the Inertial Forward-Reflected-Backward (I-FoRB) algorithm #REFR , for its convergence speed and low computational complexity.

To solve the game in (16), we focus on the computation of a variational GNE (v-GNE) #OTHEREFR Def.

3.10] , that is, the subset of GNEs which satisfy the KKT conditions

ω λ ∈ zer × i∈I N Ωi (ω i ) + F (ω) + col(A i λ) i N R |E| ≥0 (λ) − Aω + b (23) where λ ∈ R |E| ≥0

is the dual variable associated to the shared constraints in #OTHEREFR .

The v-GNEs have desirable characteristics of fairness between agents and there exist several efficient algorithms for their computation.

The I-FoRB algorithm guarantees convergence in the general class of (non-strictly) monotone games.

On the contrary, the algorithm proposed in #OTHEREFR converges only if the game is strongly Algorithm 1 I-FoRB GNE seeking for traffic routing (k) and computes:

Set (ω (0) i ) i∈I , (ω (1) i ) i∈I ∈ Ω j , λ (1) ∈ R |E| ≥0 . Set σ (1) = avg(ω (1) ).

Probabilistic Game-Theoretic Traffic Routing

Abstract: We examine the routing problem for self-interested vehicles using stochastic decision strategies. By approximating the road latency functions and a non-linear variable transformation, we frame the problem as an aggregative game. We characterize the approximation error and we derive a new monotonicity condition for a broad category of games that encompasses the problem under consideration. Next, we propose a semidecentralized algorithm to calculate the routing as a variational generalized Nash equilibrium and demonstrate the solution's benefits with numerical simulations. We also explore a recursive receding-horizon formulation of the routing problem for potential games, showing asymptotic convergence to destinations and analysing closed-loop performance dependence on horizon length through numerical simulations.

Semi-Decentralized Generalized Nash Equilibrium Seeking In Monotone Aggregative Games

Abstract: We address the generalized Nash equilibrium seeking problem for a population of agents playing aggregative games with affine coupling constraints. We focus on semi-decentralized communication architectures, where there is a central coordinator able to gather and broadcast signals of aggregative nature to the agents. By exploiting the framework of monotone operator theory and operator splitting, we first critically review the most relevant available algorithms and then design two novel schemes: (i) a single-layer, fixed-step algorithm with convergence guarantee for general (non cocoercive, non-strictly) monotone aggregative games and (ii) a single-layer proximal-type algorithm for a class of monotone aggregative games with linearly coupled cost functions. We also design novel accelerated variants of two algorithms via (alternating) inertial steps. Finally, we show via numerical simulations that the proposed algorithms outperform those in the literature in terms of convergence speed.

low computational complexity
convergence speed


Given this difficulty, work has been done in learning CBFs from expert data of safe and unsafe interactions #REFR .

Model predictive control (MPC) #OTHEREFR and trajectory optimization (TO) based methods #OTHEREFR have been widely used for safetycritical robot applications, e.g., obstacle avoidance.

However, the computation time of MPC and TO based methods has made them difficult to be directly deployed on systems that require fast response time.

Recently, a new control paradigm, CBF-based control #OTHEREFR , has become increasingly popular for safe robotic control, since it provides a simple yet computationally efficient way for safe control synthesis.

One of the main hurdles to applying CBF-based methods to robotic systems is the construction of a valid CBF.

Although work has been done in synthesizing CBFs for robots, the constructed CBF is often not intuitive #OTHEREFR .

Also, CBFs for robotics systems can be learned online using onboard sensor data #OTHEREFR .

Additionally, work has been done in learning CBFs starting from a handcrafted CBF #OTHEREFR .

Although the learning-based methods for CBF synthesis are appealing, Fig. 1 .

The structure of safe robotic control using differentiableoptimization-based CBFs.

acquiring a dataset may be costly in many cases, e.g., selfdriving cars.

Safe Navigation And Obstacle Avoidance Using Differentiable Optimization Based Control Barrier Functions

Abstract: Control barrier functions (CBFs) have been widely applied to safety-critical robotic applications. However, the construction of control barrier functions for robotic systems remains a challenging task. Recently, collision detection using differentiable optimization has provided a way to compute the minimum uniform scaling factor that results in an intersection between two convex shapes and to also compute the Jacobian of the scaling factor. In this paper, we propose a framework that uses this scaling factor, with an offset, to systematically define a CBF for obstacle avoidance tasks. We provide a theoretical analysis that proves the continuity of the proposed CBF. Empirically, we show that the proposed CBF is continuously differentiable, and the resulting optimal control problem is computationally efficient, which makes it applicable for realtime robotic control. We validate our approach, first using a 2D mobile robot example, then on the Franka-Emika Research 3 (FR3) robot manipulator both in simulation and experiment.

Learning Control Barrier Functions From Expert Demonstrations

Abstract: Inspired by the success of imitation and inverse reinforcement learning in replicating expert behavior through optimal control, we propose a learning based approach to safe controller synthesis based on control barrier functions (CBFs). We consider the setting of a known nonlinear control affine dynamical system and assume that we have access to safe trajectories generated by an expert - a practical example of such a setting would be a kinematic model of a self-driving vehicle with safe trajectories (e.g. trajectories that avoid collisions with obstacles in the environment) generated by a human driver. We then propose and analyze an optimization-based approach to learning a CBF that enjoys provable safety guarantees under suitable Lipschitz smoothness assumptions on the underlying dynamical system. A strength of our approach is that it is agnostic to the parameterization used to represent the CBF, assuming only that the Lipschitz constant of such functions can be efficiently bounded. Furthermore, if the CBF parameterization is convex, then under mild assumptions, so is our learning process. We end with extensive numerical evaluations of our results on both planar and realistic examples, using both random feature and deep neural network parameterizations of the CBF. To the best of our knowledge, these are the first results that learn provably safe control barrier functions from data.


SRM module is composed of a modified ASPP block #REFR and three spatial-aware feature refinement blocks, as shown in Figure 5 .

Semantic segmentation is a pixel-level scene understanding task, which relies on fine-grained semantic features for pixel-wise classification.

However, detailed information loss caused by downsampling is inevitable during encoding.

To compensate information loss and refine the finegrained features, we propose a SRM module that leverages the fused multimodal features in a recursive manner with spatial awareness.

For RGB modality, the encoded feature map f rgb4 is first embedded with more global features via atrous spatial pyramid pooling.

We use three dilation rates (d = 2, 4, 8), and the number of channels is reduced to 256 after this block.

To refine the fine-grained semantic features, the features of different receptive fields are recursively introduced via skip-connection.

Instead of simple concatenation of features, we perform spatial-aware feature refinement to proactively mesh upsampled features with multimodal features f mi which contain rich detailed semantic information from both RGB-T signals.

Considering the complexity, channel reduction is applied to f mi via convolutional operation.

Spidermesh: Spatial-Aware Demand-Guided Recursive Meshing For Rgb-T Semantic Segmentation

Abstract: For semantic segmentation in urban scene understanding, RGB cameras alone often fail to capture a clear holistic topology, especially in challenging lighting conditions. Thermal signal is an informative additional channel that can bring to light the contour and fine-grained texture of blurred regions in low-quality RGB image. Aiming at RGB-T (thermal) segmentation, existing methods either use simple passive channel/spatial-wise fusion for cross-modal interaction, or rely on heavy labeling of ambiguous boundaries for fine-grained supervision. We propose a Spatialaware Demand-guided Recursive Meshing (SpiderMesh) framework that: 1) proactively compensates inadequate contextual semantics in optically-impaired regions via a demand-guided target masking algorithm; 2) refines multimodal semantic features with recursive meshing to improve pixel-level semantic analysis performance. We further introduce an asymmetric data augmentation technique M-CutOut, and enable semi-supervised learning to fully utilize RGB-T labels only sparsely available in practical use. Extensive experiments on MFNet and PST900 datasets demonstrate that SpiderMesh achieves state-of-the-art performance on standard RGB-T segmentation benchmarks.

Rethinking Atrous Convolution For Semantic Image Segmentation

Abstract: In this work, we revisit atrous convolution, a powerful tool to explicitly adjust filter's field-of-view as well as control the resolution of feature responses computed by Deep Convolutional Neural Networks, in the application of semantic image segmentation. To handle the problem of segmenting objects at multiple scales, we design modules which employ atrous convolution in cascade or in parallel to capture multi-scale context by adopting multiple atrous rates. Furthermore, we propose to augment our previously proposed Atrous Spatial Pyramid Pooling module, which probes convolutional features at multiple scales, with image-level features encoding global context and further boost performance. We also elaborate on implementation details and share our experience on training our system. The proposed `DeepLabv3' system significantly improves over our previous DeepLab versions without DenseCRF post-processing and attains comparable performance with other state-of-art models on the PASCAL VOC 2012 semantic image segmentation benchmark.


Task Graph Type Graph Approach Task Level Paper

The aggregation and burst phenomenon observed by #REFR suggested that a large number of fraudsters logging the same device in a short time tend to be high-risk.

Financial fraud Transactions Attributed (statistical feature) Subgraph #OTHEREFR Node #OTHEREFR Att+Dee+Hom (graph embedding) #OTHEREFR Att+Dee+Het (graph embedding) #OTHEREFR Account-device #OTHEREFR Underground market

Underground forum Str+Sha+Het (graph embedding) #OTHEREFR Att+Dee+Het (graph embedding)

Node #OTHEREFR Word co-occurrence Att+Sha+Hom (graph embedding) Node/graph #OTHEREFR Transaction Structural (statistical feature) Node #OTHEREFR Host Attributed (statistical feature) Subgraph #OTHEREFR systems (e.g., Alipay 2 ) #OTHEREFR .

#OTHEREFR extracted high-risk dense subgraphs and obtain the risk scores based on homogeneous graph embeddings for further screening.

It initiated user node vectors by temporal activity statistics and utilized GNNs to aggregate local neighbors.

However, normal accounts may also log in to the same device so that the above GNN-based models raise a high false positive rate. #OTHEREFR solved this problem by introducing adaptive receptive paths.

Besides these general solutions, graph mining techniques have also been successfully used in many specific tasks of financial risk management recently. We introduce these tasks and corresponding solutions as follows.

Money laundering is a financial industry behavior, which makes illegal income legalization.

Graph Mining For Cybersecurity: A Survey

Abstract: The explosive growth of cyber attacks nowadays, such as malware, spam, and intrusions, caused severe consequences on society. Securing cyberspace has become an utmost concern for organizations and governments. Traditional Machine Learning (ML) based methods are extensively used in detecting cyber threats, but they hardly model the correlations between real-world cyber entities. In recent years, with the proliferation of graph mining techniques, many researchers investigated these techniques for capturing correlations between cyber entities and achieving high performance. It is imperative to summarize existing graph-based cybersecurity solutions to provide a guide for future studies. Therefore, as a key contribution of this paper, we provide a comprehensive review of graph mining for cybersecurity, including an overview of cybersecurity tasks, the typical graph mining techniques, and the general process of applying them to cybersecurity, as well as various solutions for different cybersecurity tasks. For each task, we probe into relevant methods and highlight the graph types, graph approaches, and task levels in their modeling. Furthermore, we collect open datasets and toolkits for graph-based cybersecurity. Finally, we outlook the potential directions of this field for future research.

Heterogeneous Graph Neural Networks For Malicious Account Detection

Abstract: We present, GEM, the first heterogeneous graph neural network approach for detecting malicious accounts at Alipay, one of the world's leading mobile cashless payment platform. Our approach, inspired from a connected subgraph approach, adaptively learns discriminative embeddings from heterogeneous account-device graphs based on two fundamental weaknesses of attackers, i.e. device aggregation and activity aggregation. For the heterogeneous graph consists of various types of nodes, we propose an attention mechanism to learn the importance of different types of nodes, while using the sum operator for modeling the aggregation patterns of nodes in each type. Experiments show that our approaches consistently perform promising results compared with competitive methods over time.


We provide an extension of their work, by introducing a learning method to estimate the bias model parameters using raw sensory measurements. Sterzentsenko et al. #REFR presented self-supervised depth denoising method for RGBD sensors.

A laser beam width is typically modeled as a Gaussian function of the distance along the beam #OTHEREFR . Using non-zero-mean Gaussian beam model, Laconte et al.

#OTHEREFR were able to measure the bias of several commercial lidar sensors as a function of depth and incidence angle.

The authors additionally present an experimental methodology on estimation of the bias model parameters for a lidar sensor.

Similar to our approach, the authors support the idea of utilizing spatial consistency and surface smoothness as optimization objectives.

We use the lidar depth correction model, introduced in #OTHEREFR .

It estimates the bias as a polynomial function of laser beam incidence angle, Section III-A.

The authors utilize the model for lidars and cameras calibration scenario.

However, the application of the model in localization and mapping scenarios was not studied in their paper.

Self-Supervised Depth Correction Of Lidar Measurements From Map Consistency Loss

Abstract: Depth perception is considered an invaluable source of information in the context of 3D mapping and various robotics applications. However, point cloud maps acquired using consumer-level light detection and ranging sensors (lidars) still suffer from bias related to local surface properties such as measuring beam-to-surface incidence angle, distance, texture, reflectance, or illumination conditions. This fact has recently motivated researchers to exploit traditional filters, as well as the deep learning paradigm, in order to suppress the aforementioned depth sensors error while preserving geometric and map consistency details. Despite the effort, depth correction of lidar measurements is still an open challenge mainly due to the lack of clean 3D data that could be used as ground truth. In this paper, we introduce two novel point cloud map consistency losses, which facilitate self-supervised learning on real data of lidar depth correction models. Specifically, the models exploit multiple point cloud measurements of the same scene from different viewpoints in order to learn to reduce the bias based on the constructed map consistency signal. Complementary to the removal of the bias from the measurements, we demonstrate that the depth correction models help to reduce localization drift. Additionally, we release a data set that contains point cloud data captured in an indoor corridor environment with precise localization and ground truth mapping information 1 .

Self-Supervised Deep Depth Denoising

Abstract: Depth perception is considered an invaluable source of information for various vision tasks. However, depth maps acquired using consumer-level sensors still suffer from non-negligible noise. This fact has recently motivated researchers to exploit traditional filters, as well as the deep learning paradigm, in order to suppress the aforementioned non-uniform noise, while preserving geometric details. Despite the effort, deep depth denoising is still an open challenge mainly due to the lack of clean data that could be used as ground truth. In this paper, we propose a fully convolutional deep autoencoder that learns to denoise depth maps, surpassing the lack of ground truth data. Specifically, the proposed autoencoder exploits multiple views of the same scene from different points of view in order to learn to suppress noise in a self-supervised end-to-end manner using depth and color information during training, yet only depth during inference. To enforce selfsupervision, we leverage a differentiable rendering technique to exploit photometric supervision, which is further regularized using geometric and surface priors. As the proposed approach relies on raw data acquisition, a large RGB-D corpus is collected using Intel RealSense sensors. Complementary to a quantitative evaluation, we demonstrate the effectiveness of the proposed self-supervised denoising approach on established 3D reconstruction applications. Code is avalable at this https URL


GTGAN has 11 baselines, as shown in Table 3 : (1) B1 is our baseline combining HouseGAN G and HouseGAN D (i.e., the original HouseGAN #REFR ).

We conduct extensive ablation studies on house layout generation ("10-12" subset) to evaluate the effectiveness of each component of the proposed GTGAN. Baselines Models.

(2) B2 adopts the combination of GTGAN G and HouseGAN D. (3) B3 combines HouseGAN G and GTGAN D. (4) B4 employs both GTGAN G and GTGAN D. (5) B5 is our baseline without using NNA. (6) B6 is our baseline without using CNA. (7) B7 is our baseline without using GMB.

(8) B8 is our baseline using Transformer layers instead of Conv-MPN layers. (9) B9 is the variation using Eq. (3) instead of Eq. (2). (10) B10 is the variation using Eq. (4) instead of Eq. (2).

(11) B11 is our full model, using the proposed graph-based cycle-consistency loss upon B4. Ablation Analysis.

The results of the ablation study, shown in Table 3 , prove that our graph Transformer generator G and node classification-based discriminator D improve the generation performance over the baseline models, validating the effectiveness of the proposed framework.

Specifically, when using our generator, B2 yields further improvements over B1, meaning that our generator learns local and global relations across connected and non-connected nodes more effectively, confirming our design motivation.

Graph Transformer Gans For Graph-Constrained House Generation

Abstract: We present a novel graph Transformer generative adversarial network (GTGAN) to learn effective graph node relations in an end-to-end fashion for the challenging graphconstrained house generation task. The proposed graph-Transformer-based generator includes a novel graph Transformer encoder that combines graph convolutions and selfattentions in a Transformer to model both local and global interactions across connected and non-connected graph nodes. Specifically, the proposed connected node attention (CNA) and non-connected node attention (NNA) aim to capture the global relations across connected nodes and non-connected nodes in the input graph, respectively. The proposed graph modeling block (GMB) aims to exploit local vertex interactions based on a house layout topology. Moreover, we propose a new node classification-based discriminator to preserve the high-level semantic and discriminative node features for different house components. Finally, we propose a novel graph-based cycle-consistency loss that aims at maintaining the relative spatial relationships between ground truth and predicted graphs. Experiments on two challenging graph-constrained house generation tasks (i.e., house layout and roof generation) with two public datasets demonstrate the effectiveness of GTGAN in terms of objective quantitative scores and subjective visual realism. New state-of-the-art results are established by large margins on both tasks.

House-Gan: Relational Generative Adversarial Networks For Graph-Constrained House Layout Generation

Abstract: This paper proposes a novel graph-constrained generative adversarial network, whose generator and discriminator are built upon relational architecture. The main idea is to encode the constraint into the graph structure of its relational networks. We have demonstrated the proposed architecture for a new house layout generation problem, whose task is to take an architectural constraint as a graph (i.e., the number and types of rooms with their spatial adjacency) and produce a set of axis-aligned bounding boxes of rooms. We measure the quality of generated house layouts with the three metrics: the realism, the diversity, and the compatibility with the input graph constraint. Our qualitative and quantitative evaluations over 117,000 real floorplan images demonstrate that the proposed approach outperforms existing methods and baselines. We will publicly share all our code and data.


Alternatively, attention mechanisms have been also proposed to capture the importance of different relationships between users and items #REFR .

proposed a Neural Graph Collaborative Filtering (NGCF) framework that captured collaborative signals in high-hop neighbors and integrates them into the embedding learning process #OTHEREFR .

However, further studies showed that NGCF demonstrates higher training loss and worse generalization performance with nonlinear activation and feature transformation #OTHEREFR .

As a result, the authors proposed a simplified model named Light Graph Convolution Network (LightGCN).

Other works have leveraged LightGCN by aggregating information from different aspect-level graphs #OTHEREFR (e.g., adding a user-director graph on a user-movie recommendation to guide the embedding learning process).

These RS models typically aggregate information by averaging data from neighbors.

In our experiments, we also test various aggregation strategies (see Section 5.4)

Stance Inference In Twitter Through Graph Convolutional Collaborative Filtering Networks With Minimal Supervision

Abstract: Social Media (SM) has become a stage for people to share thoughts, emotions, opinions, and almost every other aspect of their daily lives. This abundance of human interaction makes SM particularly attractive for social sensing. Especially during polarizing events such as political elections or referendums, users post information and encourage others to support their side, using symbols such as hashtags to represent their attitudes. However, many users choose not to attach hashtags to their messages, use a different language, or show their position only indirectly. Thus, automatically identifying their opinions becomes a more challenging task. To uncover these implicit perspectives, we propose a collaborative filtering model based on Graph Convolutional Networks that exploits the textual content in messages and the rich connections between users and topics. Moreover, our approach only requires a small annotation effort compared to state-of-the-art solutions. Nevertheless, the proposed model achieves competitive performance in predicting individuals' stances. We analyze users' attitudes ahead of two constitutional referendums in Chile in 2020 and 2022. Using two large Twitter datasets, our model achieves improvements of 3.4% in recall and 3.6% in accuracy over the baselines.

Graph Neural Networks For Social Recommendation

Abstract: In recent years, Graph Neural Networks (GNNs), which can naturally integrate node information and topological structure, have been demonstrated to be powerful in learning on graph data. These advantages of GNNs provide great potential to advance social recommendation since data in social recommender systems can be represented as user-user social graph and user-item graph; and learning latent factors of users and items is the key. However, building social recommender systems based on GNNs faces challenges. For example, the user-item graph encodes both interactions and their associated opinions; social relations have heterogeneous strengths; users involve in two graphs (e.g., the user-user social graph and the user-item graph). To address the three aforementioned challenges simultaneously, in this paper, we present a novel graph neural network framework (GraphRec) for social recommendations. In particular, we provide a principled approach to jointly capture interactions and opinions in the user-item graph and propose the framework GraphRec, which coherently models two graphs and heterogeneous strengths. Extensive experiments on two real-world datasets demonstrate the effectiveness of the proposed framework GraphRec. Our code is available at \url{this https URL}


We wrote all code in Julia 1.6 #REFR , using the JuMP package (Dunning et al., 2017) to write optimization functions and Gurobi (Gurobi Optimization, LLC, 2022) as the solver.

We performed each individual experiment reported in this paper using 4 Intel Xeon Platinum 8260 CPU cores from the Supercloud cluster #OTHEREFR .

Ensemble Modeling For Time Series Forecasting: An Adaptive Robust Optimization Approach

Abstract: Accurate time series forecasting is critical for a wide range of problems with temporal data. Ensemble modeling is a well-established technique for leveraging multiple predictive models to increase accuracy and robustness, as the performance of a single predictor can be highly variable due to shifts in the underlying data distribution. This paper proposes a new methodology for building robust ensembles of time series forecasting models. Our approach utilizes Adaptive Robust Optimization (ARO) to construct a linear regression ensemble in which the models' weights can adapt over time. We demonstrate the effectiveness of our method through a series of synthetic experiments and real-world applications, including air pollution management, energy consumption forecasting, and tropical cyclone intensity forecasting. Our results show that our adaptive ensembles outperform the best ensemble member in hindsight by 16-26% in root mean square error and 14-28% in conditional value at risk and improve over competitive ensemble techniques.

Julia: A Fresh Approach To Numerical Computing

Abstract: The Julia programming language is gaining enormous popularity. Julia was designed to be easy and fast. Most importantly, Julia shatters deeply established notions widely held in the applied community: $\begin{array}{l} \mbox{1. High-level, dynamic code has to be slow by some sort of law of nature. } \\ \mbox{2. It is sensible to prototype in one language and then recode in another language.} \\ \mbox {3. There are parts of a system for the programmer, and other parts best left untouched as they are built by the experts.} \end{array}$ Julia began with a deep understanding of the needs of the scientific programmer and the needs of the computer in mind. Bridging cultures that have often been distant, Julia combines expertise from computer science and computational science creating a new approach to scientific computing. This note introduces the programmer to the language and the underlying design theory. It invites the reader to rethink the fundamental foundations of numerical computing systems. In particular, there is the fascinating dance between specialization and abstraction. Specialization allows for custom treatment. We can pick just the right algorithm for the right circumstance and this can happen at runtime based on argument types (code selection via multiple dispatch). Abstraction recognizes what remains the same after differences are stripped away and ignored as irrelevant. The recognition of abstraction allows for code reuse (generic programming). A simple idea that yields incredible power. The Julia design facilitates this interplay in many explicit and subtle ways for machine performance and, most importantly, human convenience.


The parameter scaling exponent α N is measured to be ≈ 0.083 from the first six points along the curve (the seventh model appears to break the trend), roughly similar to the parameter scaling exponent measured in #REFR .

This is roughly in line with the parameter scaling exponent of 0.076 measured in Kaplan et al. 2020 1 .

The 6.4b model does not fit the scaling curve well, so we excluded its loss when measuring the scaling exponent.

Next, we plot the probability distribution over per-token losses p(L).

We find that losses close to zero are by far the most common, and that scaling increases the portion of approximately-zero losses.

We also plot Figure 3 : Top left: Scaling of mean test loss w.r.t. non-embedding parameters for the Eleuther Pythia models.

Top center: the distribution p(L) over losses on individual tokens for models of different size.

Token losses ≈ 0 are by far the most common, and larger models achieve ≈ 0 loss on an increasing fraction of tokens.

Top right: the expected loss integrand L • p(L) for models of different sizes.

Despite their very high prevalence, low-loss tokens contribute minimal mass to the mean loss, which is instead dominated by tokens with much higher loss of 5-10 bits (depending on scale). Bottom left: Training curves (scaling w.r.t. steps S) of mean test loss for Pythia models. We measure exponents α S between 0.037 and 0.06. Bottom center: the distribution p(L) over time.

Over time, models achieve ≈ 0 loss on an increasing fraction of tokens, similar to scaling in model size. Bottom right:

The Quantization Model Of Neural Scaling

Abstract: We propose the Quantization Model of neural scaling laws, explaining both the observed power law dropoff of loss with model and data size, and also the sudden emergence of new capabilities with scale. We derive this model from what we call the Quantization Hypothesis, where learned network capabilities are quantized into discrete chunks (quanta). We show that when quanta are learned in order of decreasing use frequency, then a power law in use frequencies explains observed power law scaling of loss. We validate this prediction on toy datasets, then study how scaling curves decompose for large language models. Using language model internals, we auto-discover diverse model capabilities (quanta) and find tentative evidence that the distribution over corresponding subproblems in the prediction of natural text is compatible with the power law predicted from the neural scaling exponent as predicted from our theory.

Scaling Laws For Neural Language Models

Abstract: We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.