However classical and quantum error correction are fundamentally different #REFR , and these results do not directly translate to the quantum case.

#OTHEREFR ) obtains the largest system size and threshold among neural decoders by combining a convolutional neural network backbone with a union find decoder.

In our work we show that our model, which does not rely on a handcrafted decoder, achieves higher accuracy.

From the perspective of equivariant architectures #OTHEREFR , our work studies a generalized form of equivariance, where the output representation depends on the values of the inputs to the neural network.

To the best of our knowledge, this type of symmetry properties for a neural network have not been considered before.

Finally, neural decoders for classical error correction were discussed as a form of generalized belief propagation in #OTHEREFR .

See #OTHEREFR for an attempt which however does not achieve good accuracy for the toric code.

The End: An Equivariant Neural Decoder For Quantum Error Correction

Abstract: Quantum error correction is a critical component for scaling up quantum computing. Given a quantum code, an optimal decoder maps the measured code violations to the most likely error that occurred, but its cost scales exponentially with the system size. Neural network decoders are an appealing solution since they can learn from data an efficient approximation to such a mapping and can automatically adapt to the noise distribution. In this work, we introduce a data efficient neural decoder that exploits the symmetries of the problem. To this end, we characterize the symmetries of the optimal decoder for the toric code and propose a novel equivariant architecture that achieves state of the art reconstruction accuracy compared to previous neural decoders.

Hardness Of Decoding Quantum Stabilizer Codes

Abstract: In this paper, we address the computational hardness of optimally decoding a quantum stabilizer code. Much like classical linear codes, errors are detected by measuring certain check operators which yield an error syndrome, and the decoding problem consists of determining the most likely recovery given the syndrome. The corresponding classical problem is known to be NP-complete, and are appropriate a similar decoding problem for quantum codes is also known to be NP-complete. However, this decoding strategy is not optimal in the quantum setting as it does not consider error degeneracy, which causes distinct errors to have the same effect on the code. Here, we show that optimal decoding of stabilizer codes (previously known to be NP-hard) is in fact computationally much harder than optimal decoding of classical linear codes, it is #P-complete.


We then provide a simple fix, σReparam, which reparameterizes all weight matrices by sequentially applying Spectral Normalization #REFR and a learned multiplicative scalar.

We see that interventions inducing the largest drop in attention entropy result in the sharpness exceeding the stability threshold, whereas the later interventions do not cause the threshold to be crossed, explaining how they can recover.

For details on the empirical setup and additional results see Appendix B.

The empirical correlation of entropy collapse and training instability leads to the following questions: 1) How do we prevent entropy collapse? 2) Can we improve training stability by doing so? We answer these by showing that entropy collapse can be effectively prevented by controlling the spectral norms of the query and key projections.

In particular, we prove a tight lower bound on the attention entropy, which decreases exponentially fast with the growth of the spectral norm of the attention matrix logits.

This bound suggests that entropy collapse can occur swiftly when letting the spectral norm of the weights increase uncontrollably.

Intuitively, σReparam decouples the update of the spectral norms of weights from their dimensionality, which allows them to update smoothly and in a controlled way.

Also note that σReparam does not change the model space, which allows one to learn an equally expressive model.

We evaluate five tasks: image classification, self-supervised learning (SSL), machine translation, automatic speech recognition (Appendix E), and language modeling (Appendix G). We highlight the empirical results as follows:

We show that entropy collapse is commonly observed in the baseline models of various benchmarks.

Stabilizing Transformer Training By Preventing Attention Entropy Collapse

Abstract: Training stability is of great importance to Transformers. In this work, we investigate the training dynamics of Transformers by examining the evolution of the attention layers. In particular, we track the attention entropy for each attention head during the course of training, which is a proxy for model sharpness. We identify a common pattern across different architectures and tasks, where low attention entropy is accompanied by high training instability, which can take the form of oscillating loss or divergence. We denote the pathologically low attention entropy, corresponding to highly concentrated attention scores, as entropy collapse. As a remedy, we propose σReparam, a simple and efficient solution where we reparametrize all linear layers with spectral normalization and an additional learned scalar. We demonstrate that the proposed reparameterization successfully prevents entropy collapse in the attention layers, promoting more stable training. Additionally, we prove a tight lower bound of the attention entropy, which decreases exponentially fast with the spectral norm of the attention logits, providing additional motivation for our approach. We conduct experiments with σReparam on image classification, image self-supervised learning, machine translation, automatic speech recognition, and language modeling tasks, across Transformer architectures. We show that σReparam provides stability and robustness with respect to the choice of hyperparameters, going so far as enabling training (a) a Vision Transformer to competitive performance without warmup, weight decay, layer normalization or adaptive optimizers; (b) deep architectures in machine translation and (c) speech recognition to competitive performance without warmup and adaptive optimizers.

Spectral Normalization For Generative Adversarial Networks

Abstract: One of the challenges in the study of generative adversarial networks is the instability of its training. In this paper, we propose a novel weight normalization technique called spectral normalization to stabilize the training of the discriminator. Our new normalization technique is computationally light and easy to incorporate into existing implementations. We tested the efficacy of spectral normalization on CIFAR10, STL-10, and ILSVRC2012 dataset, and we experimentally confirmed that spectrally normalized GANs (SN-GANs) is capable of generating images of better or equal quality relative to the previous training stabilization techniques.


As shown in Table 2 , the proposed DSC algorithm outperforms all the baseline methods except that the DEC algorithm #REFR has a lower RM SE mean score.

To compute these evaluation results we trained each model several times and picked the best result from them.

The results of DEC and DTC methods are generated by running their published codes.

The hyperparameters of these models like batch size, learning rate, maximum iteration, etc.

are finetuned to get better results on the applied dataset.

The k-means and hierarchical clustering method are taken from sklearn python library and applied to preprocessed data without any dimension reduction.

Note that the DEC method does not consider spatial features and uses a greedy pretraining process.

In contrast, our proposed method achieved RM SE mean close to DEC without any preprocessing step and significant improvement for the other four evaluation metrics.

In Figure 2 , we visualize the clusters generated by the DEC and the proposed DSC method.

As the latent feature space is not suitable for visualization we use t-SNE #OTHEREFR to reduce the dimension of the feature space.

The cluster plot of the DEC method shows some clear overlap between the three clusters represented by red, orange, and cyan, even though DEC method achieved a better RM SE mean score.

Deep Spatiotemporal Clustering: A Temporal Clustering Approach For Multi-Dimensional Climate Data

Abstract: Clustering high-dimensional spatiotemporal data using an unsupervised approach is a challenging problem for many data-driven applications. Existing state-of-the-art methods for unsupervised clustering use different similarity and distance functions but focus on either spatial or temporal features of the data. Concentrating on joint deep representation learning of spatial and temporal features, we propose Deep Spatiotemporal Clustering (DSC), a novel algorithm for the temporal clustering of high-dimensional spatiotemporal data using an unsupervised deep learning method. Inspired by the U-net architecture, DSC utilizes an autoencoder integrating CNN-RNN layers to learn latent representations of the spatiotemporal data. DSC also includes a unique layer for cluster assignment on latent representations that uses the Student's t-distribution. By optimizing the clustering loss and data reconstruction loss simultaneously, the algorithm gradually improves clustering assignments and the nonlinear mapping between low-dimensional latent feature space and high-dimensional original data space. A multivariate spatiotemporal climate dataset is used to evaluate the efficacy of the proposed method. Our extensive experiments show our approach outperforms both conventional and deep learning-based unsupervised clustering algorithms. Additionally, we compared the proposed model with its various variants (CNN encoder, CNN autoencoder, CNN-RNN encoder, CNN-RNN autoencoder, etc.) to get insight into using both the CNN and RNN layers in the autoencoder, and our proposed technique outperforms these variants in terms of clustering results.

Unsupervised Deep Embedding For Clustering Analysis

Abstract: Clustering is central to many data-driven application domains and has been studied extensively in terms of distance functions and grouping algorithms. Relatively little work has focused on learning representations for clustering. In this paper, we propose Deep Embedded Clustering (DEC), a method that simultaneously learns feature representations and cluster assignments using deep neural networks. DEC learns a mapping from the data space to a lower-dimensional feature space in which it iteratively optimizes a clustering objective. Our experimental evaluations on image and text corpora show significant improvement over state-of-the-art methods.


Language Adapters, Task Adapters And Adapterhub

Adapters also have the benefit of requiring as little as 1MB of storage space per task #REFR .

Fine-tuning all or a majority of a pre-trained model needs a lot of computational resources and also it depends on the availability of the dataset for the specific task which is the problem in the African context. #OTHEREFR

(2020) introduced AdapterHub, which is a central repository for pretrained adapter modules.

Adapters refers to a set of newly introduced weights, typically within the layers of a transformer model.

Adapters provide an alternative to fully fine-tuning the model for each downstream task while maintaining performance.

Rather than pre-training a large language model, a language adapter can be pre-trained for each language.

Adapter modules are parameter-efficient, sustainable, and achieve near SOTA results on low-resource and cross-lingual tasks #OTHEREFR .

In this work, we leverage Adapter-based tuning for African languages because it has been shown to mitigate forgetting issues better than fine-tuning, as it yields representations with less deviation from those generated by the original pre-training #OTHEREFR .

Moreover, Adapter fine-tuning does not take a lot of time because of its lightweight nature: we believe this will be one of the main advantages for people who are highly limited with computational resources.

Adapting To The Low-Resource Double-Bind: Investigating Low-Compute Methods On Low-Resource African Languages

Abstract: Many natural language processing (NLP) tasks make use of massively pre-trained language models, which are computationally expensive. However, access to high computational resources added to the issue of data scarcity of African languages constitutes a real barrier to research experiments on these languages. In this work, we explore the applicability of low-compute approaches such as language adapters in the context of this low-resource double-bind. We intend to answer the following question: do language adapters allow those who are doubly bound by data and compute to practically build useful models? Through fine-tuning experiments on African languages, we evaluate their effectiveness as cost-effective approaches to low-resource African NLP. Using solely free compute resources, our results show that language adapters achieve comparable performances to massive pretrained language models which are heavy on computational resources. This opens the door to further experimentation and exploration on full-extent of language adapters capacities.

Parameter-Efficient Transfer Learning For Nlp

Abstract: Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.


Additionally, this GIF supplements our findings by illustrating how increasing n out contributes to improved stability, a logical yet interesting result #REFR .

When examining the MAE, M4 frequently performs the worst, followed by either EveryNth or LTTB.

Regarding the MEAP, EveryNth demonstrates to be the most unstable, followed by M4 and then LTTB.

Notably, when focusing solely on the peaks (MEAP), the margin between MinMax and the other algorithms became even more pronounced.

For the generated noise in the last column, Min-Max once again emerges as the most stable algorithm, while LTTB, in contrast to previous templates, is the least stable.

Another note- worthy observation is the minimal variation in the results across the three different cases.

A plausible explanation for MinMax's superior stability lies in its high locality, referring to the extent that data point selection depends on individual bin values without influence from neighboring bins.

High locality results in greater stability and reduced alignment issues susceptibility.

MinMax has the highest locality among all considered algorithms, as it selects the minimum and maximum values for each bin with no interplay of neighboring bins.

In contrast, LTTB exhibits #OTHEREFR Further details regarding visual stability, showing that at least 30% of the data points are included for the MEAP metric, can be found in the GitHub repository.

lower locality, initializing its sequential data point selection process with the first data point and then using the previously selected data point from the preceding bin (together with the average point of the next bin) as input for the local algorithm.

Data Point Selection For Line Chart Visualization: Methodological Assessment And Evidence-Based Guidelines

Abstract: Figure 1: Visual analysis of four time series data point selection algorithms. Each row corresponds to a unique template, with the rightmost column presenting a reference image, constructed from 200k data points. The four adjacent columns demonstrate the varying visual representations, each generated by a data point selection algorithm, all employing an equal number of output data points (n out = 300). Visualization properties used include: Plotly as toolkit, line-width set to 2, and anti-aliasing enabled. A GIF further demonstrates the above visualization with varying values for n out .

Interactive (Statistical) Visualisation And Exploration Of A Billion Objects With Vaex

Abstract: With new catalogues arriving such as the Gaia DR1, containing more than a billion objects, new methods of handling and visualizing these data volumes are needed. In visualization, one problem is that the number of datapoints can become so large, that a scatter plot becomes cluttered. Another problem is that with over a billion objects, only a few cpu cycles are available per object if one wants to process them within a second, making traditional methods by rendering glyphs not viable. Instead, we show that by calculating statistics on a regular (N-dimensional) grid, visualizations of a billion objects can be done within a second on a modern desktop computer. This is achieved using memory mapping of hdf5 files together with a simple binning algorithm, which are part of a Python library called vaex. This enables efficient exploration or large datasets interactively, making science exploration of large catalogues feasible. Vaex is a Python library, which also integrates well in the Jupyter/Numpy/Astropy/matplotlib stack. Build on top of this is the vaex application, which allows for interactive exploration and visualization. The motivation for developing vaex is the catalogue of the Gaia satellite, however, vaex can also be used on SPH or N-body simulations, any other (future) catalogues such as SDSS, Pan-STARRS, LSST, WISE, 2MASS, etc. or other tabular data. The homepage for vaex is http://vaex.astro.rug.nl.


The CNN models have been demonstrated to efficiently perform a lane following task on an unmarked road #REFR .

As of 2020 in the United States, about 30% of the roads are still unpaved #OTHEREFR .

Recent technologies like the end-to-end learning approach enable AVs to drive safely even under challenging circumstances, such as unpaved and unmarked roads.

The prediction of steering wheel angle is one of the critical aspects of an autonomous vehicle (Society of Automotive Engineers (SAE) Level 4 and Level 5 #OTHEREFR ) that controls the lateral position of the vehicle.

The steering wheel angle is predicted from the RGB images collected from the front-facing camera as input.

The dataset consisting of camera images and the steering wheel angle is trained on a CNN model #OTHEREFR .

Related literature has demonstrated that FL can collaboratively learn the neural network model at a significantly lower communication cost while also preserving privacy #OTHEREFR - #OTHEREFR .

A Survey Of Federated Learning For Connected And Automated Vehicles

Abstract: Connected and Automated Vehicles (CAVs) are one of the emerging technologies in the automotive domain that has the potential to alleviate the issues of accidents, traffic congestion, and pollutant emissions, leading to a safe, efficient, and sustainable transportation system. Machine learningbased methods are widely used in CAVs for crucial tasks like perception, motion planning, and motion control, where machine learning models in CAVs are solely trained using the local vehicle data, and the performance is not certain when exposed to new environments or unseen conditions. Federated learning (FL) is an effective solution for CAVs that enables a collaborative model development with multiple vehicles in a distributed learning framework. FL enables CAVs to learn from a wide range of driving environments and improve their overall performance while ensuring the privacy and security of local vehicle data. In this paper, we review the progress accomplished by researchers in applying FL to CAVs. A broader view of the various data modalities and algorithms that have been implemented on CAVs is provided. Specific applications of FL are reviewed in detail, and an analysis of the challenges and future scope of research are presented.

End To End Learning For Self-Driving Cars

Abstract: We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands. This end-to-end approach proved surprisingly powerful. With minimum training data from humans the system learns to drive in traffic on local roads with or without lane markings and on highways. It also operates in areas with unclear visual guidance such as in parking lots and on unpaved roads. ::: The system automatically learns internal representations of the necessary processing steps such as detecting useful road features with only the human steering angle as the training signal. We never explicitly trained it to detect, for example, the outline of roads. ::: Compared to explicit decomposition of the problem, such as lane marking detection, path planning, and control, our end-to-end system optimizes all processing steps simultaneously. We argue that this will eventually lead to better performance and smaller systems. Better performance will result because the internal components self-optimize to maximize overall system performance, instead of optimizing human-selected intermediate criteria, e.g., lane detection. Such criteria understandably are selected for ease of human interpretation which doesn't automatically guarantee maximum system performance. Smaller networks are possible because the system learns to solve the problem with the minimal number of processing steps. ::: We used an NVIDIA DevBox and Torch 7 for training and an NVIDIA DRIVE(TM) PX self-driving car computer also running Torch 7 for determining where to drive. The system operates at 30 frames per second (FPS).


Results similar to Theorem 3.2 have been proved for the stochastic gradient method in the unconstrained setting #REFR .

, which is the first desired conclusion.

The second desired conclusion follow from the fact that if r exists as stated, then

The last desired conclusion follows using the same argument as in the proof of Theorem 3.1.

In fact, for the stochastic gradient method in an unconstrained setting, one can prove under conditions that are similar to ours that the gradient of the objective function convergences to zero almost surely; this is stronger than a lim inf result of the type in Theorem 3.2.

However, such results rely on the gradient of the objective function being Lipschitz continuous.

In our setting, we have the Lipschitzcontinuity-type property in Lemma 3.2, but the gradient of the (shifted) barrier function is not globally Lipschitz over B, meaning that such a result in the unconstrained setting does not carry over to our present setting.

A Stochastic-Gradient-Based Interior-Point Algorithm For Solving Smooth Bound-Constrained Optimization Problems

Abstract: A stochastic-gradient-based interior-point algorithm for minimizing a continuously differentiable objective function (that may be nonconvex) subject to bound constraints is presented, analyzed, and demonstrated through experimental results. The algorithm is unique from other interior-point methods for solving smooth (nonconvex) optimization problems since the search directions are computed using stochastic gradient estimates. It is also unique in its use of inner neighborhoods of the feasible region-defined by a positive and vanishing neighborhood-parameter sequence-in which the iterates are forced to remain. It is shown that with a careful balance between the barrier, step-size, and neighborhood sequences, the proposed algorithm satisfies convergence guarantees in both deterministic and stochastic settings. The results of numerical experiments show that in both settings the algorithm can outperform a projected-(stochastic)-gradient method.

Optimization Methods For Large-Scale Machine Learning

Abstract: This paper provides a review and commentary on the past, present, and future of numerical optimization algorithms in the context of machine learning applications. Through case studies on text classification and the training of deep neural networks, we discuss how optimization problems arise in machine learning and what makes them challenging. A major theme of our study is that large-scale machine learning represents a distinctive setting in which the stochastic gradient (SG) method has traditionally played a central role while conventional gradient-based nonlinear optimization techniques typically falter. Based on this viewpoint, we present a comprehensive theory of a straightforward, yet versatile SG algorithm, discuss its practical behavior, and highlight opportunities for designing algorithms with improved performance. This leads to a discussion about the next generation of optimization methods for large-scale machine learning, including an investigation of two main streams of research on techniques that diminish noise in the stochastic directions and methods that make use of second-order derivative approximations.


Ringer et al., study the effects of texture bias illustrated by Convolutional Neural Networks (CNN) in FSL #REFR and address the same by modifying the training data with a combination of non-texture and texture-based images.

IER incorporates both invariance and equivariance inductive biases of certain geometric transformations.

Unlike RFS, IER employs a multi-head distillation over standard supervised distillation. Liu et al.

#OTHEREFR learn a base embedding model by unifying the two-stage training in RFS-Distill using online self-distillation.

We incorporate shape bias by distilling features in latent space and model decisions between two networks trained synchronously.

Overcoming Biases in FSL: The biases affecting the generalization of conventional DNNs create a larger impact on the FSL setup.

The image background also serves as a shortcut in the FSL scenario and impacts classification performance. Luo et al.

#OTHEREFR mitigates the problem by sampling the foreground and original images in the training and testing phases.

Finally, even though the feature distribution of the base classes used in pretraining FSL models is well defined, the distribution is affected by the domain shift incurred by the novel classes. Tao et al.

#OTHEREFR propose Distribution Calibration Module (DCM) and Selected Sampling (SS) to mitigate class-agnostic and class-specific bias, respectively, in the meta-testing phase.

However, LSFSL utilizes the implicit shape information in the training data with alignment objectives that aid in improving the generalization of few-shot models by learning global features.

Lsfsl: Leveraging Shape Information In Few-Shot Learning

Abstract: Few-shot learning (FSL) techniques seek to learn the underlying patterns in data using fewer samples, analogous to how humans learn from limited experience. In this limiteddata scenario, the challenges associated with deep neural networks, such as shortcut learning and texture bias behaviors, are further exacerbated. Moreover, the significance of addressing shortcut learning is not yet fully explored in the few-shot setup. To address these issues, we propose LS-FSL, which enforces the model to learn more generalizable features utilizing the implicit prior information present in the data. Through comprehensive analyses, we demonstrate that LSFSL-trained models are less vulnerable to alteration in color schemes, statistical correlations, and adversarial perturbations leveraging the global semantics in the data. Our findings highlight the potential of incorporating relevant priors in few-shot approaches to increase robustness and generalization.

Texture Bias Of Cnns Limits Few-Shot Classification Performance

Abstract: Accurate image classification given small amounts of labelled data (few-shot classification) remains an open problem in computer vision. In this work we examine how the known texture bias of Convolutional Neural Networks (CNNs) affects few-shot classification performance. Although texture bias can help in standard image classification, in this work we show it significantly harms few-shot classification performance. After correcting this bias we demonstrate state-of-the-art performance on the competitive miniImageNet task using a method far simpler than the current best performing few-shot learning approaches.

Convolutional Neural Networks
texture bias


State-of-the-art high-throughput low-latency decoder architectures are based on the Fast-SSC decoding algorithm #REFR .

Since Fast-SSC decoding operates on higher stages in the PFT, the LLR values of the frozen leaf nodes are not calculated.

Therefore, the PM calculation presented previously is not directly applicable.

Thus, we derive PM calculation methods for the specialized nodes analogous to Fast Simplified SCL (SSCL) decoding #OTHEREFR .

The value of PM ( ) is accumulated sequentially, where denotes the index of the leaf node in the pruned PFT and is initialized with PM (0) = 0.

Automorphism Ensemble Polar Code Decoders For 6G Urllc

Abstract: The URLLC scenario in the upcoming 6G standard requires low latency and ultra reliable transmission, i.e., error correction towards ML performance. Achieving near-ML performance is very challenging especially for short block lengths. Polar codes are a promising candidate and already part of the 5G standard. The Successive Cancellation List (SCL) decoding algorithm provides very good error correction performance but at the cost of high computational decoding complexity resulting in large latency and low area and energy efficiency. Recently, Automorphism Ensemble Decoding (AED) gained a lot of attention to improve the error correction capability. In contrast to SCL, AED performs several low-complexity (e.g., SC) decoding in parallel. However, it is an open question whether AED can compete with sophisticated SCL decoders, especially from an implementation perspective in state of the art silicon technologies. In this paper we present an elaborated AED architecture that uses an advanced path metric based candidate selection to reduce the implementation complexity and compare it to state of the art SCL decoders in a 12nm FinFET technology. Our AED implementation outperform state of the art SCL decoders by up to 4.4x in latency, 8.9x in area efficiency and 4.6x in energy efficiency, while providing the same or even better error correction performance.

Polarbear: A 28-Nm Fd-Soi Asic For Decoding Of Polar Codes

Abstract: Polar codes are a recently proposed class of block codes that provably achieve the capacity of various communication channels. They received a lot of attention as they can do so with low-complexity encoding and decoding algorithms, and they have an explicit construction. Their recent inclusion in a 5G communication standard will only spur more research. However, only a couple of ASICs featuring decoders for polar codes were fabricated, and none of them implements a list-based decoding algorithm. In this paper, we present ASIC measurement results for a fabricated 28-nm CMOS chip that implements two different decoders: the first decoder is tailored toward error-correction performance and flexibility. It supports any code rate as well as three different decoding algorithms: successive cancellation (SC), SC flip, and SC list (SCL). The flexible decoder can also decode both non-systematic and systematic polar codes. The second decoder targets speed and energy efficiency. We present measurement results for the first silicon-proven SCL decoder, where its coded throughput is shown to be of 306.8 Mbps with a latency of 3.34 us and an energy per bit of 418.3 pJ/b at a clock frequency of 721 MHz for a supply of 1.3 V. The energy per bit drops down to 178.1 pJ/b with a more modest clock frequency of 308 MHz, lower throughput of 130.9 Mbps and a reduced supply voltage of 0.9 V. For the other two operating modes, the energy per bit is shown to be of approximately 95 pJ/b. The less flexible high-throughput unrolled decoder can achieve a coded throughput of 9.2 Gbps and a latency of 628 ns for a measured energy per bit of 1.15 pJ/b at 451 MHz.


We use the evaluation methodology followed by prior works #REFR Yang et al., 2019) . Two protocols are used for evaluation.

Protocol 1 (LFH1): This protocol uses the BIWI #OTHEREFR Evaluation metrics: We report the Mean Absolute Error (MAE) between the targets (y i ) and predictions (ŷ i ).

Let N represent the number of samples, and P represent the number of labels (three in head pose estimation). The MAE is defined as:

Network architecture and training parameters: Table 12 summarizes the hyperparameters used for RLEL .

The learning rate of the decoding matrix D is kept 10× higher than the learning rate of the feature extractor.

Learning Label Encodings For Deep Regres-Sion

Abstract: Deep regression networks are widely used to tackle the problem of predicting a continuous value for a given input. Task-specialized approaches for training regression networks have shown significant improvement over generic approaches, such as direct regression. More recently, a generic approach based on regression by binary classification using binary-encoded labels has shown significant improvement over direct regression. The space of label encodings for regression is large. Lacking heretofore have been automated approaches to find a good label encoding for a given application. This paper introduces Regularized Label Encoding Learning (RLEL) for end-to-end training of an entire network and its label encoding. RLEL provides a generic approach for tackling regression. Underlying RLEL is our observation that the search space of label encodings can be constrained and efficiently explored by using a continuous search space of real-valued label encodings combined with a regularization function designed to encourage encodings with certain properties. These properties balance the probability of classification error in individual bits against error correction capability. Label encodings found by RLEL result in lower or comparable errors to manually designed label encodings. Applying RLEL results in 10.9% and 12.4% improvement in Mean Absolute Error (MAE) over direct regression and multiclass classification, respectively. Our evaluation demonstrates that RLEL can be combined with off-the-shelf feature extractors and is suitable across different architectures, datasets, and tasks. Code is available at https://github.com/ubc-aamodt-group/RLEL_regression.

Fine-Grained Head Pose Estimation Without Keypoints

Abstract: Estimating the head pose of a person is a crucial problem that has a large amount of applications such as aiding in gaze estimation, modeling attention, fitting 3D models to video and performing face alignment. Traditionally head pose is computed by estimating some keypoints from the target face and solving the 2D to 3D correspondence problem with a mean human head model. We argue that this is a fragile method because it relies entirely on landmark detection performance, the extraneous head model and an ad-hoc fitting step. We present an elegant and robust way to determine pose by training a multi-loss convolutional neural network on 300W-LP, a large synthetically expanded dataset, to predict intrinsic Euler angles (yaw, pitch and roll) directly from image intensities through joint binned pose classification and regression. We present empirical tests on common in-the-wild pose benchmark datasets which show state-of-the-art results. Additionally we test our method on a dataset usually used for pose estimation using depth and start to close the gap with state-of-the-art depth pose methods. We open-source our training and testing code as well as release our pre-trained models.


This suggests that METEOR is biased towards measuring surface similarity rather than underlying semantics, consistent with the findings of #REFR who explicitly compare METEOR and BLEURT.

We additionally provide scores for the METEOR met- ric #OTHEREFR , which uses unigram alignment statistics and incorporates both precision and recall.

It also uses stemming and synonym matching to provide some robustness relative to changes in the surface forms of semantically similar texts.

Although METEOR does not ignore most predictions as does BLEU, we find that it underperforms BLEURT in capturing semantic similarity in our setting.

The base-lines in Table 10 are calculated by (1) using the constant text "[NAME] meeting with [NAME]", and (2) randomizing the order of the predictions of the model fine-tuned on our pseudo-labels.

Both baselines achieve a relatively high METEOR score, while BLEURT decreases significantly and approaches the BLEURT score of the plain captioning model.

This can also be seen in the qualitative examples in Table 11 of prediction (CLIPCap CC+pHHI) and ground truth pairs from Waldo and Wenda where METEOR and BLEURT differ strongly in magnitude.

Note we replace [NAME] with the text "person" as needed to calculate BLEU and METEOR scores, just as we do for calculating BLEURT scores (see Section B.9).

Learning Human-Human Interactions In Images From Weak Textual Supervision

Abstract: Interactions between humans are diverse and contextdependent, but previous works have treated them as categorical, disregarding the heavy tail of possible interactions. We propose a new paradigm of learning human-human interactions as free text from a single still image, allowing for flexibility in modeling the unlimited space of situations and relationships between people. To overcome the absence of data labelled specifically for this task, we use knowledge distillation applied to synthetic caption data produced by a large language model without explicit supervision. We show that the pseudo-labels produced by this procedure can be used to train a captioning model to effectively understand human-human interactions in images, as measured by a variety of metrics that measure textual and semantic faithfulness and factual groundedness of our predictions. We further show that our approach outperforms SOTA image captioning and situation recognition models on this task. We will release 1 our code and pseudo-labels along with Waldo and Wenda, a manually-curated test set for still image human-human interaction understanding.

Bleurt: Learning Robust Metrics For Text Generation

Abstract: Text generation has made significant advances in the last few years. Yet, evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU and ROUGE) may correlate poorly with human judgments. We propose BLEURT, a learned evaluation metric based on BERT that can model human judgments with a few thousand possibly biased training examples. A key aspect of our approach is a novel pre-training scheme that uses millions of synthetic examples to help the model generalize. BLEURT provides state-of-the-art results on the last three years of the WMT Metrics shared task and the WebNLG Competition dataset. In contrast to a vanilla BERT-based approach, it yields superior results even when the training data is scarce and out-of-distribution.


Projection-Free Online Learning and Optimization Hazan and Kale #REFR developed the first online projectionfree algorithm, based on the Frank-Wolfe method [5] , that guarantees O(T 3/4 ) regret for non-smooth loss functions.

However, for the general setting, it is known that it is impossible to simultaneously bound the regret and constraint violation by a sublinear function in T #OTHEREFR . Mahdavi et al.

#OTHEREFR then considered the special case where g t = g for some fixed function g for all t ∈ [T ], and they provided an augmented-Lagrangian-based algorithm that achieves O( √ T ) regret and O(T 3/4 ) constraint violation. Jenatton et al.

#OTHEREFR modified the algorithm to obtain O(T max{β,1−β} ) regret and O(T 1−β/2 ) constraint violation where β ∈ (0, 1) is an algorithm parameter.

#OTHEREFR considered the case where g t 's are time-varying but are i.

Garber and Hazan [6] provided an improved algorithm for smooth and strongly convex loss functions.

After these works, there has been many results on projection-free algorithms for online convex optimization with no long-term

Projection-Free Online Convex Optimization With Stochastic Constraints

Abstract: This paper develops projection-free algorithms for online convex optimization with stochastic constraints. We design an online primal-dual projection-free framework that can take any projection-free algorithms developed for online convex optimization with no long-term constraint. With this general template, we deduce sublinear regret and constraint violation bounds for various settings. Moreover, for the case where the loss and constraint functions are smooth, we develop a primal-dual conditional gradient method that achieves O(√ T) regret and O(T 3/4) constraint violations. Furthermore, for the setting where the loss and constraint functions are stochastic and strong duality holds for the associated offline stochastic optimization problem, we prove that the constraint violation can be reduced to have the same asymptotic growth as the regret.

Projection-Free Online Learning

Abstract: The computational bottleneck in applying online learning to massive data sets is usually the projection step. We present efficient online learning algorithms that eschew projections in favor of much more efficient linear optimization steps using the Frank-Wolfe technique. We obtain a range of regret bounds for online convex optimization, with better bounds for specific cases such as stochastic online smooth convex optimization. ::: Besides the computational advantage, other desirable features of our algorithms are that they are parameter-free in the stochastic case and produce sparse decisions. We apply our algorithms to computationally intensive applications of collaborative filtering, and show the theoretical improvements to be clearly visible on standard datasets.


To render the color of a given pixel, the original volume rendering formulation in NeRF #REFR is augmented to include the transient colors and densities, and the color of a given image-pixel (Ĉ(r)) is computed as a composite of the static and transient components.

We propose a Neural Feature Synthesizer (NeFeS) model that directly renders dense feature maps of a given viewpoint to refine the predictions of an underlying APR network.

Similar to NeRF-W #OTHEREFR , our NeFeS architecture uses a base MLP module with static and transient heads that predict the static and transient density (σ (s) and σ (τ ) ) and view-dependent color (c (s) and c (τ ) ) respectively, given an input 3D position (x) and viewing direction (d).

We use the frequency encoding #OTHEREFR to encode all 3D positions and view directions.

The transient head models the colors of the 3D points using an isotropic normal distribution and predicts a view-dependent variance value (β 2 ) for the transient color distribution.

Here, r denotes the ray (corresponding to the pixel) on which points are sampled to compute the volume rendering quadrature approximation #OTHEREFR .

The variances of sampled points along the corresponding ray are also rendered using only the tran- i .

In addition, the color map adopts exposure-adaptive ACT to compensate for exposure differences between images.

The final feature map F f usion is the concatenation of rendered RGB and feature map processed by the convolutional fusing layer.

sient densities (and not the static densities) to obtain a perpixel color variance β(r) 2 .

Refinement For Absolute Pose Regression With Neural Feature Synthesis

Abstract: Absolute Pose Regression (APR) methods use deep neural networks to directly regress camera poses from RGB images. Despite their advantages in inference speed and simplicity, these methods still fall short of the accuracy achieved by geometry-based techniques. To address this issue, we propose a new model called the Neural Feature Synthesizer (NeFeS). Our approach encodes 3D geometric features during training and renders dense novel view features at test time to refine estimated camera poses from arbitrary APR methods. Unlike previous APR works that require additional unlabeled training data, our method leverages implicit geometric constraints during test time using a robust feature field. To enhance the robustness of our NeFeS network, we introduce a feature fusion module and a progressive training strategy. Our proposed method improves the state-of-the-art single-image APR accuracy by as much as 54.9% on indoor and outdoor benchmark datasets without additional time-consuming unlabeled data training.

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.


Get π i (k) by #REFR The numerical simulation is implemented on MATLAB R2020b, and Fig. 10 is one example of the simulation scenarios.

And we also perform the hardwarein-the-loop simulations, which are described in Section 7.3.

Assuming that each target can be tracked by a maximum of 2 UAVs simultaneously, set the parameters related to grid visiting requirements as w p1 = 1, w p2 = 1.2, w p3 = 0.8 and the related parameters of sensor detection performance as α = 1.1, β = 0.8.

Considering the randomness of the experiment, including the initial positions of the target and the UAV, and the random movement of the target, we performed 50 repetitions of each set of experiments with 1500 simulation steps to get average results.

Integrated Design Of Cooperative Area Coverage And Target Tracking With Multi-Uav System

Abstract: This paper systematically studies the cooperative area coverage and target tracking problem of multiple-unmanned aerial vehicles (multi-UAVs). The problem is solved by decomposing into three sub-problems: information fusion, task assignment, and multi-UAV behavior decision-making. Specifically, in the information fusion process, we use the maximum consistency protocol to update the joint estimation states of multi-targets (JESMT) and the area detection information. The area detection information is represented by the equivalent visiting time map (EVTM), which is built based on the detection probability and the actual visiting time of the area. Then, we model the task assignment problem of multi-UAV searching and tracking multi-targets as a network flow model with upper and lower flow bounds. An algorithm named task assignment minimum-cost maximum-flow (TAMM) is proposed. Cooperative behavior decision-making uses Fisher information as the mission reward to obtain the optimal tracking action of the UAV. Furthermore, a coverage behavior decision-making algorithm based on the anti-flocking method is designed for those UAVs assigned the coverage task. Finally, a distributed multi-UAV cooperative area coverage and target tracking algorithm is designed, which integrates information fusion, task assignment, and behavioral decision-making. Numerical and hardware-in-the-loop simulation results show that the proposed method can achieve persistent area coverage and cooperative target tracking.

Coordinated Path Following Control Of Fixed-Wing Unmanned Aerial Vehicles

Abstract: In this paper, we investigate the problem of coordinated path following for fixed-wing UAVs with speed constraints in 2D plane. The objective is to steer a fleet of UAVs along the path(s) while achieving the desired sequenced inter-UAV arc distance. In contrast to the previous coordinated path following studies, we are able through our proposed hybrid control law to deal with the forward speed and the angular speed constraints of fixed-wing UAVs. More specifically, the hybrid control law makes all the UAVs work at two different levels: those UAVs whose path following errors are within an invariant set (i.e., the designed coordination set) work at the coordination level; and the other UAVs work at the single-agent level. At the coordination level, we prove that even with speed constraints, the proposed control law can make sure the path following errors reduce to zero, while the desired arc distances converge to the desired value. At the single-agent level, the convergence analysis for the path following error entering the coordination set is provided. We develop a hardware-in-the-loop simulation testbed of the multi-UAV system by using actual autopilots and the X-Plane simulator. The effectiveness of the proposed approach is corroborated with both MATLAB and the testbed.


Limitations First, similar to other optimization-based attribute discovery approaches #REFR , [43] , there is no guarantee that all prevalent factors are captured, though our controlled empirical studies clearly demonstrate the efficacy of xGA over existing approaches.

We introduced the first cross-GAN auditing framework, which utilizes a novel optimization technique to jointly infer common, novel and missing attributes for a client GAN w.r.t any reference GAN.

Through a large suite of datasets and GAN models, we demonstrate that the proposed method (i) consistently leads to higher quality (disentangled & diverse) attributes, (ii) effectively reveals shared attributes even across challenging distribution shifts, and (iii) accurately identifies the novel/missing attributes in our controlled experiments (i.e., known ground truth).

Second, while using an external feature space enhances the performance of attribute discovery, this becomes an additional component that must be tuned.

While we found advBN ResNet-50 to be a reasonable choice for a variety of face datasets (and AFHQ), a more systematic solution will expand the utility of our approach to other applications. A.

Ablation study First, we investigate the effect of the λ b parameter on KLIEP loss that allows us to discover novel attributes.

In addition to KLIEP loss presented in the main text, we analyze a model trained with simple log loss used to train binary classifiers to predict the likelihood of a given sample.

Cross-Gan Auditing: Unsupervised Identification Of Attribute Level Similarities And Differences Between Pretrained Generative Models

Abstract: Figure 1. We introduce (xGA) an approach for fully unsupervised cross-GAN auditing and validation. Given two pre-trained GANs (Reference & Client), xGA evaluates the client by identifying three types of semantic attributes-(a) Common: those that exist in both models, (b) Novel: those only present in the client and (c) Missing: those that only exist in the reference. On the right, we show results across multiple studies, that among others include notable shifts in distribution between the Reference (CelebA) to Client (Toon, Disney, Met Faces). xGA also lends itself easily to comparing models with different properties on the same dataset as shown on the bottom right for StyleGAN3-T vs. StyleGAN-R. And CelebA-M is a control dataset we create that does not contain glasses, ties and smiles.

Unsupervised Discovery Of Interpretable Directions In The Gan Latent Space

Abstract: The latent spaces of typical GAN models often have semantically meaningful directions. Moving in these directions corresponds to human-interpretable image transformations, such as zooming or recoloring, enabling a more controllable generation process. However, the discovery of such directions is currently performed in a supervised manner, requiring human labels, pretrained models, or some form of self-supervision. These requirements can severely limit a range of directions existing approaches can discover. In this paper, we introduce an unsupervised method to identify interpretable directions in the latent space of a pretrained GAN model. By a simple model-agnostic procedure, we find directions corresponding to sensible semantic manipulations without any form of (self-)supervision. Furthermore, we reveal several non-trivial findings, which would be difficult to obtain by existing methods, e.g., a direction corresponding to background removal. As an immediate practical benefit of our work, we show how to exploit this finding to achieve a new state-of-the-art for the problem of saliency detection.


In the stir tank example, a relatively small NN approximates the dataset well (more than 99 % of predictions are valid solutions to (5) on test data), which is in line with numerical results in #REFR .

First, we show the training effort and approximation quality achieved.

Second, performance in terms of computation time is compared with online optimization.

Finally, we show that our method is safe in all three examples, while naively (always) applying the NN leads to constraint violations in closed loop.

1) Dataset generation and NN training: We now describe the size and computation times of generated datasets, give an indication of NN sizes, and provide NN training times for all three systems.

To gain some intuition into how well each approximator generalizes, it is tested on 10 5 points not used during training by forward simulating and checking whether the NN provides a feasible solution to #OTHEREFR . The results are summarized in Table I .

For the ten-state quadcopter example, a dataset corresponding to approximately 5 gridpoints per state dimension suffices to fit a relatively large approximator, which yields a feasible solution to the MPC problem in over 70 % of initial conditions.

Contributing factors to this could be that the quadcopter system is almost linear (only non-linearity is the tan(φ) in the velocity derivative), only loosely coupled in three Cartesian directions by the MPC cost function, and has few constraints that become active.

The chain mass system, on the other hand, is "very" non-linear and chaotic, which makes the approximation problem inherently harder.

This is reflected in a large dataset and approximator, which yield a feasible sequence for less than 10 % of initial conditions during testing.

The total time required to evaluate samples of the MPC on random initial conditions is the dominant time factor offline. However, this was executed in parallel.

Approximate Non-Linear Model Predictive Control With Safety-Augmented Neural Networks

Abstract: Model predictive control (MPC) achieves stability and constraint satisfaction for general nonlinear systems, but requires computationally expensive online optimization. This paper studies approximations of such MPC controllers via neural networks (NNs) to achieve fast online evaluation. We propose safety augmentation that yields deterministic guarantees for convergence and constraint satisfaction despite approximation inaccuracies. We approximate the entire input sequence of the MPC with NNs, which allows us to verify online if it is a feasible solution to the MPC problem. We replace the NN solution by a safe candidate based on standard MPC techniques whenever it is infeasible or has worse cost. Our method requires a single evaluation of the NN and forward integration of the input sequence online, which is fast to compute on resource-constrained systems. The proposed control framework is illustrated on three non-linear MPC benchmarks of different complexity, demonstrating computational speedups orders of magnitudes higher than online optimization. In the examples, we achieve deterministic safety through the safetyaugmented NNs, where naive NN implementation fails.

Learning An Approximate Model Predictive Controller With Guarantees

Abstract: A supervised learning framework is proposed to approximate a model predictive controller (MPC) with reduced computational complexity and guarantees on stability and constraint satisfaction. The framework can be used for a wide class of nonlinear systems. Any standard supervised learning technique (e.g. neural networks) can be employed to approximate the MPC from samples. In order to obtain closed-loop guarantees for the learned MPC, a robust MPC design is combined with statistical learning bounds. The MPC design ensures robustness to inaccurate inputs within given bounds, and Hoeffding's Inequality is used to validate that the learned MPC satisfies these bounds with high confidence. The result is a closed-loop statistical guarantee on stability and constraint satisfaction for the learned MPC. The proposed learning-based MPC framework is illustrated on a nonlinear benchmark problem, for which we learn a neural network controller with guarantees.


S2. Results From Various Deblurring Methods

In Table S2 , we additionally compare the results from another widely used deblurring method, DeepDeblur #REFR , as the final mesh estimation results might be dependent on the performance of deblurring methods.

In Tables 2 and 3 in our main manuscript, we compared our BlurHandNet with the combination of state-of-the-art 3D hand mesh estimation methods #OTHEREFR and offthe-shelf deblurring method #OTHEREFR .

Please note that NAFNet [3] is a deblurring method that we used in the main manuscript.

Our BlurHandNet still outperforms the case when we use Deep-Deblur #OTHEREFR as the deblurring method.

The results again demonstrate that utilizing temporal information is useful rather than simply adopting deblurring methods. Table S2 . Comparison results on various deblurring methods #OTHEREFR .

Instead of adopting deblurring methods, the proposed BlurHandNet (last row) directly utilizes temporal information from a single blurry image.

Recovering 3D Hand Mesh Sequence From A Single Blurry Image: A New Dataset And Temporal Unfolding

Abstract: Hands, one of the most dynamic parts of our body, suffer from blur due to their active movements. However, previous 3D hand mesh recovery methods have mainly focused on sharp hand images rather than considering blur due to the absence of datasets providing blurry hand images. We first present a novel dataset BlurHand, which contains blurry hand images with 3D groundtruths. The BlurHand is constructed by synthesizing motion blur from sequential sharp hand images, imitating realistic and natural motion blurs. In addition to the new dataset, we propose BlurHandNet, a baseline network for accurate 3D hand mesh recovery from a blurry hand image. Our BlurHandNet unfolds a blurry input image to a 3D hand mesh sequence to utilize temporal information in the blurry input image, while previous works output a static single hand mesh. We demonstrate the usefulness of BlurHand for the 3D hand mesh recovery from blurry images in our experiments. The proposed BlurHandNet produces much more robust results on blurry images while generalizing well to in-the-wild images. The training codes and BlurHand dataset are available at https://github.com/JaehaKim97/BlurHand_RELEASE.

Deep Multi-Scale Convolutional Neural Network For Dynamic Scene Deblurring

Abstract: Non-uniform blind deblurring for general dynamic scenes is a challenging computer vision problem as blurs arise not only from multiple object motions but also from camera shake, scene depth variation. To remove these complicated motion blurs, conventional energy optimization based methods rely on simple assumptions such that blur kernel is partially uniform or locally linear. Moreover, recent machine learning based methods also depend on synthetic blur datasets generated under these assumptions. This makes conventional deblurring methods fail to remove blurs where blur kernel is difficult to approximate or parameterize (e.g. object motion boundaries). In this work, we propose a multi-scale convolutional neural network that restores sharp images in an end-to-end manner where blur is caused by various sources. Together, we present multi-scale loss function that mimics conventional coarse-to-fine approaches. Furthermore, we propose a new large-scale dataset that provides pairs of realistic blurry image and the corresponding ground truth sharp image that are obtained by a high-speed camera. With the proposed model trained on this dataset, we demonstrate empirically that our method achieves the state-of-the-art performance in dynamic scene deblurring not only qualitatively, but also quantitatively.


These coefficients, can thus, be easily computed and stored at the CPU #REFR . We now explain the two-step decoding in detail.

The RF chain output is then fed to the dynamic-resolution ADCs, whose quantized and noisy output, based on the Bussgang model #OTHEREFR , is given as follows: #OTHEREFR .

Principles of two-layer decoding: The CF system considered herein, as shown in Fig. 2 , employs two-layer decoding to mitigate IUI.

In the first-stage, each AP combines its received signal by using local channel estimates, which only partially mitigate the IUI.

To mitigate the residual IUI, all APs sends their locally-combined received signal to the CPU, which performs the secondlayer LSFD.

The CPU computes the LSFD weights based on the large-scale fading coefficients, which remains constant for 100s of coherence intervals #OTHEREFR , #OTHEREFR .

The mth AP first uses channel estimateĥ mk [λ] to combine the distorted received signal as

y ADC,m [n] = A m y RF,m [n]+n ADC,m [n] = A m K k=1 h mk [n]s RF,k [n]+η RF,m [n]+z m [n] +n ADC,m [n].

The matrix A m = diag{ι m,1 , • • • , ι m,N },

B m = A m (I N − A m ) and S m [n] = diag(E{y RF,m [n]y H RF,m [n]|h mk [n]})

Hardware-Impaired Rician-Faded Cell-Free Massive Mimo Systems With Channel Aging

Abstract: We study the impact of channel aging on the uplink of a cell-free (CF) massive multiple-input multiple-output (mMIMO) system by considering i) spatially-correlated Rician-faded channels; ii) hardware impairments at the access points and user equipments (UEs); and iii) two-layer large-scale fading decoding (LSFD). We first derive a closed-form spectral efficiency (SE) expression for this system, and later propose two novel optimization techniques to optimize the non-convex SE metric by exploiting the minorization-maximization (MM) method. The first one requires a numerical optimization solver, and has a high computation complexity. The second one with closed-form transmit power updates, has a trivial computation complexity. We numerically show that i) the two-layer LSFD scheme effectively mitigates the interference due to channel aging for both low-and high-velocity UEs; and ii) increasing the number of AP antennas does not mitigate the SE deterioration due to channel aging. We numerically characterize the optimal pilot length required to maximize the SE for various UE speeds. We also numerically show that the proposed closed-form MM optimization yields the same SE as that of the first technique, which requires numerical solver, and that too with a much reduced time-complexity. Index Terms Cell-free, channel aging, hardware impairments, minorization-maximization (MM), Rician fading.

Joint Power Control And Lsfd For Wireless-Powered Cell-Free Massive Mimo

Abstract: This paper considers wireless uplink information and downlink power transfer in cell-free massive multiple-input multiple-output systems. The single-antenna user equipments (UEs) utilize the energy harvested in the downlink to transmit uplink pilot and information signals to the multiple-antenna access points (APs). We consider Rician fading and maximum ratio processing based on either linear minimum mean-squared error (LMMSE) or least-squares (LS) channel estimation. We derive the average harvested energy by using a practical non-linear energy harvesting circuit model for both coherent and non-coherent transmission schemes. Furthermore, the uplink spectral efficiency (SE) is derived for all the considered methods and the max-min fairness problem is cast where the optimization variables are the AP and UE power control coefficients together with the large-scale fading decoding vectors. The objective is to maximize the minimum SE of the UEs' under APs' and UEs' transmission power constraints. A novel alternating optimization algorithm with guaranteed convergence and improvement at each step is proposed to solve the highly-coupled non-convex problem.


This conceptually differs from previous works #REFR that typically use short videos of around 5-10 seconds.

This problem has been addressed by various approaches Tube-DETR #OTHEREFR , STCAT #OTHEREFR , STVGBert #OTHEREFR , STVGBert #OTHEREFR , STGVT #OTHEREFR , STGRN #OTHEREFR .

These methods rely on proposal networks such as Faster R-CNN #OTHEREFR or MDETR #OTHEREFR to predict bounding box coordinates for learning text-toregion interaction.

All those approaches rely on supervised training with the human-annotated sentence and bounding box supervision, provided, e.g., by datasets such as Vid-STG #OTHEREFR and HC-STVG #OTHEREFR .

While those datasets provide a temporal aspect, temporal detection is usually limited to identifying the start and end frame of a single action in a video.

Compared to that, an untrimmed setting usually comprises multiple actions in a longer video that can be separated by longer background sequences.

What, When, And Where? -Self-Supervised Spatio-Temporal Grounding In Untrimmed Multi-Action Videos From Narrated Instructions

Abstract: Spatio-temporal grounding describes the task of localizing events in space and time, e.g., in video data, based on verbal descriptions only. Models for this task are usually trained with human-annotated sentences and bounding box supervision. This work addresses this task from a multimodal supervision perspective, proposing a framework for spatio-temporal action grounding trained on loose video and subtitle supervision only, without human annotation. To this end, we combine local representation learning, which focuses on leveraging fine-grained spatial information, with a global representation encoding that captures higher-level representations and incorporates both in a joint approach. To evaluate this challenging task in a real-life setting, a new benchmark dataset is proposed providing dense spatiotemporal grounding annotations in long, untrimmed, multiaction instructional videos for over 5K events. We evaluate the proposed approach and other methods on the proposed and standard downstream tasks showing that our method improves over current baselines in various settings, including spatial, temporal, and untrimmed multi-action spatiotemporal grounding.

Weakly-Supervised Spatio-Temporally Grounding Natural Sentence In Video

Abstract: In this paper, we address a novel task, namely weakly-supervised spatio-temporally grounding natural sentence in video. Specifically, given a natural sentence and a video, we localize a spatio-temporal tube in the video that semantically corresponds to the given sentence, with no reliance on any spatio-temporal annotations during training. First, a set of spatio-temporal tubes, referred to as instances, are extracted from the video. We then encode these instances and the sentence using our proposed attentive interactor which can exploit their fine-grained relationships to characterize their matching behaviors. Besides a ranking loss, a novel diversity loss is introduced to train the proposed attentive interactor to strengthen the matching behaviors of reliable instance-sentence pairs and penalize the unreliable ones. Moreover, we also contribute a dataset, called VID-sentence, based on the ImageNet video object detection dataset, to serve as a benchmark for our task. Extensive experimental results demonstrate the superiority of our model over the baseline approaches.


ProdLDA #REFR based on the VAE has better performance in topic coherence and computational efficiency than original LDA.

BPTMs are usually built on generative probabilistic models and can generate a document with a structure of latent variables sampled from prespecified distributions (Dirichlet for LDA). These latent variables are used to obtain topics. The BPTMs are learned through a Bayesian inference process.

LDA performs the inference using Monte Carlo Markov chain sampling #OTHEREFR or Gibbs sampling.

Neural topic models (NTMs) #OTHEREFR incorporate neural components for Bayesian inference, and most NTMs are based on the variational autoencoder (VAE) inference framework #OTHEREFR ).

The VAE adopts an encoder-decoder structure: the encoder trains a neural inference network to directly map a BoW representation of documents into a continuous latent embedding, while the decoder reconstructs the BoW from the latent embedding.

More recently, powerful contextualized embeddings were also incorporated into NTMs, which further improved the topic coherence compared to conventional NTMs that use bag-of-words (BoW) as document representations #OTHEREFR .

However, most of the NTMs are based on the variational autoencoder inference framework #OTHEREFR , which suffers from hyperparameter tuning, computational complexity #OTHEREFR , and an inability to choose the proper number of topics.

G2T: A Simple But Versatile Framework For Topic Modeling Based On Pretrained Language Model And Community Detection

Abstract: It has been reported that clustering-based topic models, which cluster high-quality sentence embeddings with an appropriate word selection method, can generate better topics than generative probabilistic topic models. However, these approaches suffer from the inability to select appropriate parameters and incomplete models that overlook the quantitative relation between words with topics and topics with text. To solve these issues, we propose graph to topic (G2T), a simple but effective framework for topic modelling. The framework is composed of four modules. First, document representation is acquired using pretrained language models. Second, a semantic graph is constructed according to the similarity between document representations. Third, communities in document semantic graphs are identified, and the relationship between topics and documents is quantified accordingly. Fourth, the word-topic distribution is computed based on a variant of TFIDF. Automatic evaluation suggests that G2T achieved state-of-the-art performance on both English and Chinese documents with different lengths. Human judgements demonstrate that G2T can produce topics with better interpretability and coverage than baselines. In addition, G2T can not only determine the topic number automatically but also give the probabilistic distribution of words in topics and topics in documents. Finally, G2T is publicly available, and the distillation experiments provide instruction on how it works.

Autoencoding Variational Inference For Topic Models

Abstract: Topic models are one of the most popular methods for learning representations of ::: text, but a major challenge is that any change to the topic model requires mathematically ::: deriving a new inference algorithm. A promising approach to address ::: this problem is autoencoding variational Bayes (AEVB), but it has proven diffi- ::: cult to apply to topic models in practice. We present what is to our knowledge the ::: first effective AEVB based inference method for latent Dirichlet allocation (LDA), ::: which we call Autoencoded Variational Inference For Topic Model (AVITM). This ::: model tackles the problems caused for AEVB by the Dirichlet prior and by component ::: collapsing. We find that AVITM matches traditional methods in accuracy ::: with much better inference time. Indeed, because of the inference network, we ::: find that it is unnecessary to pay the computational cost of running variational ::: optimization on test data. Because AVITM is black box, it is readily applied ::: to new topic models. As a dramatic illustration of this, we present a new topic ::: model called ProdLDA, that replaces the mixture model in LDA with a product ::: of experts. By changing only one line of code from LDA, we find that ProdLDA ::: yields much more interpretable topics, even if LDA is trained via collapsed Gibbs ::: sampling.


HeCo MU MixUp #REFR is proposed to efficiently improve results in supervised learning by adding arbitrary two samples to create a new one.

First, we train D to identify embeddings from two views as positives and that generated from G as negatives.

Then, we train G to generate samples to fool D.

The two steps are alternated for some interactions to make D and G trained.

(2) We utilize a well-trained G to generate samples, which can be viewed as the new negative samples with high quality.

Then, we continue to train HeCo with the newly generated and original negative samples for some epochs.

MoCHi #OTHEREFR introduces this strategy into contrastive learning , who mixes the hard negatives to make harder negatives.

Inspired by them, we bring this strategy into HIN field for the first time.

We can get cosine similarities between node i and nodes from N i during calculating eq.(11), and sort them in the descending order.

Then, we select first top k negative samples as the hardest negatives, and randomly add them to create new k negatives, which are involved in training.

It is worth mentioning that there are no learnable parameters in this version, which is very efficient.

Hierarchical Contrastive Learning Enhanced Heterogeneous Graph Neural Network

Abstract: Heterogeneous graph neural networks (HGNNs) as an emerging technique have shown superior capacity of dealing with heterogeneous information network (HIN). However, most HGNNs follow a semi-supervised learning manner, which notably limits their wide use in reality since labels are usually scarce in real applications. Recently, contrastive learning, a self-supervised method, becomes one of the most exciting learning paradigms and shows great potential when there are no labels. In this paper, we study the problem of self-supervised HGNNs and propose a novel co-contrastive learning mechanism for HGNNs, named HeCo. Different from traditional contrastive learning which only focuses on contrasting positive and negative samples, HeCo employs cross-view contrastive mechanism. Specifically, two views of a HIN (network schema and meta-path views) are proposed to learn node embeddings, so as to capture both of local and high-order structures simultaneously. Then the cross-view contrastive learning, as well as a view mask mechanism, is proposed, which is able to extract the positive and negative embeddings from two views. This enables the two views to collaboratively supervise each other and finally learn high-level node embeddings. Moreover, to further boost the performance of HeCo, two additional methods are designed to generate harder negative samples with high quality. The essence of HeCo is to make positive samples from different views close to each other by cross-view contrast, and learn the factors invariant to two proposed views. However, besides the invariant factors, view-specific factors complementally provide the diverse structure information between different nodes, which also should be contained into the final embeddings. Therefore, we need to further explore each view independently and propose a modified model, called HeCo++. Specifically, HeCo++ conducts hierarchical contrastive learning, including cross-view and intra-view contrasts, which aims to enhance the mining of respective structures. Extensive experiments conducted on a variety of real-world networks show the superior performance of the proposed methods over the state-of-the-arts.

Mixup: Beyond Empirical Risk Minimization

Abstract: Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.


SinGAN #REFR can generate similar images and perform well on multiple tasks (e.g., image editing) after training on a single image.

The success of diffusion models in the text-to-image synthesis area relies on a large amount of training samples.

An interesting topic is how to train a generative model with a single image, e.g., SinGAN #OTHEREFR .

There are also research on diffusion models with a single image #OTHEREFR , #OTHEREFR , #OTHEREFR .

Single Denoising Diffusion Models (SinDDM) #OTHEREFR proposes a hierarchical diffusion model inspired by the multi-scale SinGAN #OTHEREFR .

The convolutional denoiser is trained on various scales of images which are corrupted by multiple levels of noise.

Compared to SDEdit #OTHEREFR , SinDDM #OTHEREFR can generating images with different dimensions.

By contrast, Single-image Diffusion Model (Sin-Diffusion) #OTHEREFR is trained on a single image at a single scale, avoiding the accumulation of errors.

Text-To-Image Diffusion Model In Generative Ai: A Survey

Abstract: This survey reviews text-to-image diffusion models in the context that diffusion models have emerged to be popular for a wide range of generative tasks. As a self-contained work, this survey starts with a brief introduction of how a basic diffusion model works for image synthesis, followed by how condition or guidance improves learning. Based on that, we present a review of state-of-the-art methods on text-conditioned image synthesis, i.e. text-to-image. We further summarize applications beyond text-to-image generation: text-guided creative generation and text-guided image editing. Beyond the progress made so far, we discuss existing challenges and promising future directions.

Singan: Learning A Generative Model From A Single Natural Image

Abstract: We introduce SinGAN, an unconditional generative model that can be learned from a single natural image. Our model is trained to capture the internal distribution of patches within the image, and is then able to generate high quality, diverse samples that carry the same visual content as the image. SinGAN contains a pyramid of fully convolutional GANs, each responsible for learning the patch distribution at a different scale of the image. This allows generating new samples of arbitrary size and aspect ratio, that have significant variability, yet maintain both the global structure and the fine textures of the training image. In contrast to previous single image GAN schemes, our approach is not limited to texture images, and is not conditional (i.e. it generates samples from noise). User studies confirm that the generated samples are commonly confused to be real images. We illustrate the utility of SinGAN in a wide range of image manipulation tasks.


Evaluation Metrics: We use three widely adopted metrics in previous work to evaluate our performance: Chamfer Distance (CD), Point-to-Surface Distance (P2F) w.r.t ground truth meshes, and Uniformity Score #REFR . For these metrics, a smaller value means better performance.

The outputs of the feature extractor block are the point-wise features F 0 and the global shape code G.

Note that the design of feature extractor is not the major contribution of this paper and we can exploit any other suitable networks.

We then feed these outputs along with the original position of the points P 0 into stacks of upsampling blocks to generate denser point clouds.

To upsample the point cloud 4 times, we arbitrarily set two upsampler blocks with upscale ratios of 1 and 4, respectively.

Note that other combinations of upscale ratios and the number of upsampler blocks are also feasible.

Training Detail: To train this network, we use 2 Tesla V100 GPUs.

We set the batch size to 16 and the total epoch number to 150.

Besides, we use Adam as the optimization function with a learning rate of 0.0005 at the beginning, and we decrease the learning rate by a factor of 0.5 every 50 epochs.

Experiment Results: Table 1 shows the quantitative upsampling results on the PU1K dataset.

We find that our algorithm achieves the best performance over all its counterparts with large improvements.

Parametric Surface Constrained Upsampler Network For Point Cloud

Abstract: Designing a point cloud upsampler, which aims to generate a clean and dense point cloud given a sparse point representation, is a fundamental and challenging problem in computer vision. A line of attempts achieves this goal by establishing a point-to-point mapping function via deep neural networks. However, these approaches are prone to produce outlier points due to the lack of explicit surface-level constraints. To solve this problem, we introduce a novel surface regularizer into the upsampler network by forcing the neural network to learn the underlying parametric surface represented by bicubic functions and rotation functions, where the new generated points are then constrained on the underlying surface. These designs are integrated into two different networks for two tasks that take advantages of upsampling layers-point cloud upsampling and point cloud completion for evaluation. The state-of-the-art experimental results on both tasks demonstrate the effectiveness of the proposed method. The implementation code will be available at https://github.com/corecai163/PSCU.

Pu-Gan: A Point Cloud Upsampling Adversarial Network

Abstract: Point clouds acquired from range scans are often sparse, noisy, and non-uniform. This paper presents a new point cloud upsampling network called PU-GAN, which is formulated based on a generative adversarial network (GAN), to learn a rich variety of point distributions from the latent space and upsample points over patches on object surfaces. To realize a working GAN network, we construct an up-down-up expansion unit in the generator for upsampling point features with error feedback and self-correction, and formulate a self-attention unit to enhance the feature integration. Further, we design a compound loss with adversarial, uniform and reconstruction terms, to encourage the discriminator to learn more latent patterns and enhance the output point distribution uniformity. Qualitative and quantitative evaluations demonstrate the quality of our results over the state-of-the-arts in terms of distribution uniformity, proximity-to-surface, and 3D reconstruction quality.


In this paper, we choose the commonly used WideResNet-38 #REFR as the backbone network for feature extraction, which is used in ablation studies and performance comparisons with state-of-the-art methods.

In particular, to ensure a high-resolution output scale for feature maps and not have too much parameter growth, we replace the traditional convolutions of the last two blocks with dilated/atrous convolutions as in #OTHEREFR , such that the spatial scale of output feature maps is 1/8 of the input image.

All our implementations are deployed on the PyTorch #OTHEREFR deep learning platform. Baseline.

Our baseline is the single-stage semantic segmentation #OTHEREFR model, which is based on the self-supervised training manner with only image-level class labels.

In the training stage, the backbone network first generates the initial CAMs.

Then, a mask refinement module named PAMR is used to refine CAMs as pseudo ground-truth to complete the selfsupervised training process.

Coupling Global Context And Local Contents For Weakly-Supervised Semantic Segmentation

Abstract: Thanks to the advantages of the friendly annotations and the satisfactory performance, Weakly-Supervised Semantic Segmentation (WSSS) approaches have been extensively studied. Recently, the single-stage WSSS was awakened to alleviate problems of the expensive computational costs and the complicated training procedures in multi-stage WSSS. However, results of such an immature model suffer from problems of background incompleteness and object incompleteness. We empirically find that they are caused by the insufficiency of the global object context and the lack of the local regional contents, respectively. Under these observations, we propose a single-stage WSSS model with only the image-level class label supervisions, termed as Weakly-Supervised Feature Coupling Network (WS-FCN), which can capture the multi-scale context formed from the adjacent feature grids, and encode the fine-grained spatial information from the low-level features into the high-level ones. Specifically, a flexible context aggregation module is proposed to capture the global object context in different granular spaces. Besides, a semantically consistent feature fusion module is proposed in a bottom-up parameter-learnable fashion to aggregate the finegrained local contents. Based on these two modules, WS-FCN lies in a self-supervised end-to-end training fashion. Extensive experimental results on the challenging PASCAL VOC 2012 and MS COCO 2014 demonstrate the effectiveness and efficiency of WS-FCN, which can achieve state-of-the-art results by 65.02% and 64.22% mIoU on PASCAL VOC 2012 val set and test set, 34.12% mIoU on MS COCO 2014 val set, respectively. The code and weight have been released at: WS-FCN.

Wider Or Deeper: Revisiting The Resnet Model For Visual Recognition

Abstract: The community has been going deeper and deeper in designing one cutting edge network after another, yet some works are there suggesting that we may have gone too far in this dimension. Some researchers unravelled a residual network into an exponentially wider one, and assorted the success of residual networks to fusing a large amount of relatively shallow models. Since some of their early claims are still not settled, we in this paper dig more on this topic, i.e., the unravelled view of residual networks. Based on that, we try to find a good compromise between the depth and width. Afterwards, we walk through a typical pipeline of developing a deep-learning-based algorithm. We start from a group of relatively shallow networks, which perform as well or even better than the current (much deeper) state-of-the-art models on the ImageNet classification dataset. Then, we initialize fully convolutional networks (FCNs) using our pre-trained models, and tune them for semantic image segmentation. Results show that the proposed networks, as pre-trained features, can boost existing methods a lot. Even without exhausting the sophistical techniques to improve the classic FCN model, we achieve comparable results with the best performers on four widely-used datasets, i.e., Cityscapes, PASCAL VOC, ADE20k and PASCAL-Context. The code and pre-trained models are released for public access 1 .


We used two instances of the network, an 18-layer and a 34-layer Bi-Real net #REFR .

The VGG-Small model adopted uses the straight-through-estimator (STE) to approximate the gradient on non-differentiable layers #OTHEREFR .

VGG-11 #OTHEREFR Network structure is the following: 64−C3+M P 2+128−C3+ M P 2 + 2 × (256 − C3) + M P 2 + 2 × (512 − C3) + M P 2 + 2 × (512 − C3) + M P 2 + Sof tmax.

Even VGG-11 uses the STE estimator for binarization operation during back-propagation.

BiRealNet #OTHEREFR It is a modified version of classical ResNet that proposes to preserve the real activations before the sign function to increase the representational capability of the 1-bit CNN, through a simple shortcut.

Bi-RealNet adopts a tight approximation to the derivative of the non-differentiable sign function with respect to activation and a magnitude-aware gradient to update weight parameters.

ReactNet #OTHEREFR To further compress compact networks, this model constructs a baseline based on MobileNetV1 #OTHEREFR and add shortcut to bypass every 1-bit convolutional layer that has the same number of input and output channels.

The 3 × 3 depth-wise and the 1 × 1 point-wise convolutional blocks of MobileNet are replaced by the 3 × 3 and 1 × 1 vanilla convolutions in parallel with shortcuts in React Net #OTHEREFR .

As for Bi-Real Net, we tested two different versions of React Net: a 18-layer and a 34-layer. of our binarization method w.r.t.

the state-of-the-arts input layer binarization approaches, keeping unaltered the structure of the network except for the input data binarization layer (first half of Tables 2, 3 and 4) .

In this first scenario all the 8-bits planes are exploited, layer F1 ( Fig.

Input Layer Binarization With Bit-Plane Encoding

Abstract: Binary Neural Networks (BNNs) use 1-bit weights and activations to efficiently execute deep convolutional neural networks on edge devices. Nevertheless, the binarization of the first layer is conventionally excluded, as it leads to a large accuracy loss. The few works addressing the first layer binarization, typically increase the number of input channels to enhance data representation; such data expansion raises the amount of operations needed and it is feasible only on systems with enough computational resources. In this work, we present a new method to binarize the first layer using directly the 8-bit representation of input data; we exploit the standard bit-planes encoding to extract features bit-wise (using depth-wise convolutions); after a re-weighting stage, features are fused again. The resulting model is fully binarized and our first layer binarization approach is model independent. The concept is evaluated on three classification datasets (CIFAR10, SVHN and CIFAR100) for different model architectures (VGG and ResNet) and, the proposed technique outperforms state of the art methods both in accuracy and BMACs reduction.

Dorefa-Net: Training Low Bitwidth Convolutional Neural Networks With Low Bitwidth Gradients

Abstract: We propose DoReFa-Net, a method to train convolutional neural networks that have low bitwidth weights and activations using low bitwidth parameter gradients. In particular, during backward pass, parameter gradients are stochastically quantized to low bitwidth numbers before being propagated to convolutional layers. As convolutions during forward/backward passes can now operate on low bitwidth weights and activations/gradients respectively, DoReFa-Net can use bit convolution kernels to accelerate both training and inference. Moreover, as bit convolutions can be efficiently implemented on CPU, FPGA, ASIC and GPU, DoReFa-Net opens the way to accelerate training of low bitwidth neural network on these hardware. Our experiments on SVHN and ImageNet datasets prove that DoReFa-Net can achieve comparable prediction accuracy as 32-bit counterparts. For example, a DoReFa-Net derived from AlexNet that has 1-bit weights, 2-bit activations, can be trained from scratch using 6-bit gradients to get 46.1\% top-1 accuracy on ImageNet validation set. The DoReFa-Net AlexNet model is released publicly.


We argue that this can be attributed to the much higher similarity between pre-training and downstream task compared to RoBERTa which was also observed by #REFR .

The results of feature extraction and fine-tuning are also compared to the current state-of-the-art results of the chosen datasets.

On both single-label and multi-label datasets, fine-tuning on average performs better than feature extraction.

In most cases, the smallest language model, BERT-Tiny, trained with feature extraction is on par with the word embeddings.

However, when increasing model size or using fine-tuning, the word embeddings fall behind regarding recall.

DBpedia is the only dataset on which BERT achieves better results with fine-tuning and feature extraction.

While feature extraction can achieve good results on single-label data compared to fine-tuning it unfortunately falls short when training on multi-label data.

Table 1 : Test Accuracy (%) averaged over 3 runs on the single-label datasets.

We compare feature extraction (FE) with our baselines and state-of-the-art (SOTA) models.

Using Language Models On Low-End Hardware

Abstract: This paper evaluates the viability of using fixed language models for training text classification networks on low-end hardware. We combine language models with a CNN architecture and put together a comprehensive benchmark with 8 datasets covering single-label and multi-label classification of topic, sentiment, and genre. Our observations are distilled into a list of trade-offs, concluding that there are scenarios, where not fine-tuning a language model yields competitive effectiveness at faster training, requiring only a quarter of the memory compared to fine-tuning.

To Tune Or Not To Tune? Adapting Pretrained Representations To Diverse Tasks

Abstract: While most previous work has focused on different pretraining objectives and architectures for transfer learning, we ask how to best adapt the pretrained model to a given target task. We focus on the two most common forms of adaptation, feature extraction (where the pretrained weights are frozen), and directly fine-tuning the pretrained model. Our empirical results across diverse NLP tasks with two state-of-the-art models show that the relative performance of fine-tuning vs. feature extraction depends on the similarity of the pretraining and target tasks. We explore possible explanations for this finding and provide a set of adaptation guidelines for the NLP practitioner.


However, what is more difficult to envision are the problems that are harder to fix because straightforward corrections may not exist, and rules for finding problems are more challenging to specify than those in smell detectors or linters #REFR like language idioms and code smells.

It involves complex challenges like following the best practices, avoiding code smells, using design patterns, and many more decisions before writing code.

The biggest challenge with using tools like Copilot is their training data.

These tools are trained on existing software source code, and training costs are expensive.

Several classes of errors like securuty vlunerabilities #OTHEREFR , license compliance and copyright violation #OTHEREFR , and also simple coding mistakes, such as not allowing for an empty array in a sort routine #OTHEREFR have been discovered, which follow from the presence of these same errors in public (training) data.

Although these are clearly challenges, Copilot seems already to be on its way to fixing them, like a filter introduced by GitHub to suppress code suggestions containing code that matches public code on GitHub.

Developers often discuss software architecture and actual source code implementations in online forums, chat rooms, mailing lists, or in person.

Programming tasks can be solved in more than one way.

The best way to proceed can be determined based on case-specific conditions, limits, and conventions.

Strong standards and a shared vocabulary make communication easier while fostering a shared understanding of the issues and solutions related to software development.

However, this takes time and experience to learn and use idiomatic approaches #OTHEREFR .

From Copilot To Pilot: Towards Ai Supported Software Development

Abstract: AI-supported programming has arrived, as shown by the introduction and successes of large language models for code, such as Copilot/Codex (Github/OpenAI) and AlphaCode (DeepMind). Above human average performance on programming challenges is now possible. However, software engineering is much more than solving programming contests. Moving beyond code completion to AI-supported software engineering will require an AI system that can, among other things, understand how to avoid code smells, to follow language idioms, and eventually (maybe!) propose rational software designs. In this study, we explore the current limitations of AI-supported code completion tools like Copilot and offer a simple taxonomy for understanding the classification of AI-supported code completion tools in this space. We first perform an exploratory study on Copilot's code suggestions for language idioms and code smells. Copilot does not follow language idioms and avoid code smells in most of our test scenarios. We then conduct additional investigation to determine the current boundaries of AI-supported code completion tools like Copilot by introducing a taxonomy of software abstraction hierarchies where 'basic programming functionality' such as code compilation and syntax checking is at the least abstract level, software architecture analysis and design are at the most abstract level. We conclude by providing a discussion on challenges for future development of AI-supported code completion tools to reach the design level of abstraction in our taxonomy.

What To Fix? Distinguishing Between Design And Non-Design Rules In Automated Tools

Abstract: Design problems, frequently the result of optimizing for delivery speed, are a critical part of long-term software costs. Automatically detecting such design issues is a high priority for software practitioners. Software quality tools promise automatic detection of common software quality rule violations. However, since these tools bundle a number of rules, including rules for code quality, it is hard for users to understand which rules identify design issues in particular. Research has focused on comparing these tools on open source projects, but these comparisons have not looked at whether the rules were relevant to design. We conducted an empirical study using a structured categorization approach, and manually classified 466 software quality rules from three industry tools—CAST, SonarQube, and NDepend. We found that most of these rules were easily labeled as either non-design (55%) or design (19%). The remainder (26%) resulted in disagreements among the labelers. Our results are a first step in formalizing a definition of a design rule, to support automatic detection.