For instance, kNN-LM #REFR combines a PLM and a k-nearest neighbors model by linearly interpolating both models' next word distributions, achieving lower perplexity in language modeling.

We categorize and compare representative works of tool-augmented learning in Table 1 .

The most representative tool used for augmentation is the text retriever tool, which develops from the early sparse retriever (Sparck #OTHEREFR to the recent dense retriever .

Early endeavors resort to retrieving knowledge from local repositories to augment language generation.

Some works propose retrieving knowledge using a frozen knowledge retriever.

RETRO #OTHEREFR demonstrates the benefits of extending the retrieval memory to a massive scale (i.e., trillions of tokens).

Another line of work trains the retriever and the PLM in Figure 3 : Tool-augmented learning seeks to augment foundation models with the execution results from tools (i.e., tool for AI); while tool-oriented learning focuses on utilizing models to govern tools and make sequential decisions in place of humans (i.e., AI for tool). Our studied tool learning encompasses both of them.

an end-to-end fashion, achieving superior performance in knowledge-intensive NLP tasks #OTHEREFR .

Later works have gone beyond local repositories, for instance, some leverage the entire web as the knowledge source, which allows for improved temporal generalization and higher factual accuracy #OTHEREFR .

Instead of treating the retriever as a passive agent, researchers further demonstrate that PLMs can actively interact with a search engine like humans.

Tool Learning With Foundation Models

Abstract: Humans possess an extraordinary ability to create and utilize tools, allowing them to overcome physical limitations and explore new frontiers. With the advent of recent powerful foundation models, artificial intelligence systems have the potential to be equally adept in tool use as humans. This paradigm, which is dubbed as tool learning with foundation models, combines the strengths of specialized tools and foundation models to achieve enhanced accuracy, efficiency, and automation in problem-solving. Despite its immense potential, there is still a lack of a comprehensive understanding of key challenges, opportunities, and future endeavors in this field. To this end, we present a systematic investigation of tool learning in this paper. We first introduce the background of tool learning, including its cognitive origins, the paradigm shift of foundation models, and the complementary roles of tools and models. Then we recapitulate existing tool learning research into tool-augmented and tool-oriented learning. We formulate a general tool learning framework: starting from understanding the user instruction, models should learn to decompose a complex task into several subtasks, dynamically adjust their plan through reasoning, and effectively conquer each sub-task by selecting appropriate tools. We also discuss how to train models for improved tool-use capabilities and facilitate the generalization in tool learning. Considering the lack of a systematic tool learning evaluation in prior works, we experiment with 17 representative tools and show the potential of current foundation models in skillfully utilizing tools. Finally, we discuss several open problems that require further investigation for tool learning, such as ensuring safe and trustworthy tool use, enabling tool creation with foundation models, and addressing personalization challenges. Overall, we hope this paper could inspire future research in integrating tools with foundation models. The codes and datasets are publicly available for further research exploration 1 .

Generalization Through Memorization: Nearest Neighbor Language Models

Abstract: We introduce $k$NN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a $k$-nearest neighbors ($k$NN) model. The nearest neighbors are computed according to distance in the pre-trained LM embedding space, and can be drawn from any text collection, including the original LM training data. Applying this transformation to a strong Wikitext-103 LM, with neighbors drawn from the original training set, our $k$NN-LM achieves a new state-of-the-art perplexity of 15.79 -- a 2.9 point improvement with no additional training. We also show that this approach has implications for efficiently scaling up to larger training sets and allows for effective domain adaptation, by simply varying the nearest neighbor datastore, again without further training. Qualitatively, the model is particularly helpful in predicting rare patterns, such as factual knowledge. Together, these results strongly suggest that learning similarity between sequences of text is easier than predicting the next word, and that nearest neighbor search is an effective approach for language modeling in the long tail.


As shown in Figure 7 , the input to the system is a colorless point cloud of a scene, which is processed by PointNet++ #REFR , to produce features for every point.

[SZAB17] also address the limitation of Deep Sliding Shapes by adding orientation classification as an auxiliary task, and demonstrate that speed and accuracy of 3D detection using a sliding window increases when the 3D CNN is jointly trained on object labels, location and pose.

In recent years, voting concepts, specifically, Hough voting, have made a comeback in the space of 3D object detection.

VoteNet #OTHEREFR and ImVoteNet #OTHEREFR are two such works that are built on voting strategies.

VoteNet demonstrates two advantages of using the voting strategy for 3D object detection -first, it does not make use of any 2D object detectors which used to be the de-facto step in 3D object detection, and second, the Hough voting technique used in the work is now differentiably learned in an end-to-end supervised manner.

A voting net, which is nothing but a MLP on these point features, produces virtual points, called votes, for centers of 3D bounding boxes.

The votes are clustered in the 3D space using farthest point sampling and L 2 distance, from where the extent of the 3D bounding boxes and their centroids are regressed using another MLP.

All of this training is done in a su- Figure 7 : An end-to-end learning pipeline for 3D object detection using voting technique (VoteNet) #OTHEREFR that directly operates on 3D data without the need for any 2D image priors, such as 2D object detectors.

#OTHEREFR that was originally proposed for the task of shape classification and segmentation at the object part level, as well as for indoor scenes.

This work has been the basis for many 3D object detection works, especially for developing task-specific feature descriptors.

Advances In Data-Driven Analysis And Synthesis Of 3D Indoor Scenes

Abstract: Figure 1: A sampler of representative results for different indoor scene modeling tasks surveyed in this report-(a) 3D object detection [YWY22] (Section 5.1), (b) 3D scene segmentation [ZJJ * 21] (Section 5.2), (c) scene reconstruction as image-CAD model alignment [GDN22] (Section 5.3), (d) 3D scene similarity [XMZ * 14] (Section 5.4), and (e) 3D scene synthesis [WSCR18, LPX * 19, YGZT21, PKS * 21] (Section 6). We survey advances in these indoor scene modeling tasks mainly in the realm of 3D geometry.

Pointnet++: Deep Hierarchical Feature Learning On Point Sets In A Metric Space

Abstract: Few prior works study deep learning on point sets. PointNet by Qi et al. is a pioneer in this direction. However, by design PointNet does not capture local structures induced by the metric space points live in, limiting its ability to recognize fine-grained patterns and generalizability to complex scenes. In this work, we introduce a hierarchical neural network that applies PointNet recursively on a nested partitioning of the input point set. By exploiting metric space distances, our network is able to learn local features with increasing contextual scales. With further observation that point sets are usually sampled with varying densities, which results in greatly decreased performance for networks trained on uniform densities, we propose novel set learning layers to adaptively combine features from multiple scales. Experiments show that our network called PointNet++ is able to learn deep point set features efficiently and robustly. In particular, results significantly better than state-of-the-art have been obtained on challenging benchmarks of 3D point clouds.


More recently, Karnin, Lang, and Liberty (KLL) #REFR developed the randomized KLL sketch that succeeds at all points with probability 1 − δ and uses O((1/ε) log log(1/δ)) space and gave a matching lower bound.

, x n and a query point q, the rank of q, denoted R(q), is the number of items in {x i } n i=1 such that x i ≤ q.

An ε-approximate quantile sketch is a data structure that, given access to a single pass over the stream elements, can approximate the rank of all query points simultaneously with additive error at most εn.

Given its central importance, the streaming quantiles problem has been studied extensively by both theoreticians and practitioners.

Early work by Manku, Rajagopalan, and Lindsay #OTHEREFR gave a randomized solution that used O((1/ε) log 2 (nε)) space; their technique can also be straightforwardly adapted to a deterministic solution that achieves the same bound #OTHEREFR .

Later, Greenwald and Khanna #OTHEREFR developed a deterministic algo-rithm that requires only O((1/ε) log(nε)) space.

Meanwhile, streaming quantile estimation is of significant interest to practitioners in databases, computer systems, and data science who have studied the problem as well.

Most notably, Dunning #OTHEREFR introduced the celebrated t-digest, a heuristic quantile estimation technique based on 1-dimensional k-means clustering that has seen adoption in numerous systems, including Influx, Apache Arrow, and Apache Spark.

Although tdigest achieves remarkable accuracy on many real-world data sets, it is known to have arbitrarily bad error in the worst case #OTHEREFR .

To illustrate this core tradeoff, Figure 1 shows the rank function of the books dataset from the SOSD benchmark #OTHEREFR , along with KLL and t-digest approximations that use the same amount of space when the data set is randomly shuffled, and when the same data set is streamed in an adversarial order that we found to induce especially bad performance in t-digest.

Recent advances in machine learning have led to the development of learning-augmented algorithms which seek to improve solutions to classical algorithms problems by exploiting empirical properties of the input distribution #OTHEREFR .

Learned Interpolation For Better Streaming Quantile Approximation With Worst-Case Guarantees

Abstract: An ε-approximate quantile sketch over a stream of n inputs approximates the rank of any query point q-that is, the number of input points less than q-up to an additive error of εn, generally with some probability of at least 1−1/ poly(n), while consuming o(n) space. While the celebrated KLL sketch of Karnin, Lang, and Liberty achieves a provably optimal quantile approximation algorithm over worst-case streams, the approximations it achieves in practice are often far from optimal. Indeed, the most commonly used technique in practice is Dunning's t-digest, which often achieves much better approximations than KLL on realworld data but is known to have arbitrarily large errors in the worst case. We apply interpolation techniques to the streaming quantiles problem to attempt to achieve better approximations on real-world data sets than KLL while maintaining similar guarantees in the worst case. * Anthropic. Work done while at MIT.

Optimal Quantile Approximation In Streams

Abstract: This paper resolves one of the longest standing basic problems in the streaming computational model. Namely, optimal construction of quantile sketches. An $\varepsilon$ approximate quantile sketch receives a stream of items $x_1,\ldots,x_n$ and allows one to approximate the rank of any query up to additive error $\varepsilon n$ with probability at least $1-\delta$. The rank of a query $x$ is the number of stream items such that $x_i \le x$. The minimal sketch size required for this task is trivially at least $1/\varepsilon$. Felber and Ostrovsky obtain a $O((1/\varepsilon)\log(1/\varepsilon))$ space sketch for a fixed $\delta$. To date, no better upper or lower bounds were known even for randomly permuted streams or for approximating a specific quantile, e.g.,\ the median. This paper obtains an $O((1/\varepsilon)\log \log (1/\delta))$ space sketch and a matching lower bound. This resolves the open problem and proves a qualitative gap between randomized and deterministic quantile sketching. One of our contributions is a novel representation and modification of the widely used merge-and-reduce construction. This subtle modification allows for an analysis which is both tight and extremely simple. Similar techniques should be useful for improving other sketching objectives and geometric coreset constructions.


We train score-based models with 128 hidden layers in the first block using 10000 multi-coil T2-weighted brain MRI scans from the FastMRI dataset #REFR .

The acquired kspace data from the i th coil is given by:

where n i is additive white Gaussian noise with variance σ 2 w .

Following convention, we define the acceleration factor 1/α relative to a single fully sampled coil, such that the total number of acquired measurements is α • N c • N .

We can represent the MRI multi-coil forward model according to #OTHEREFR .

The SURE-Score model was trained on a subset of the data (2000 samples) due to training time constraints.

We first normalize all slices in the training set by performing a root sum-of-squares (RSS) reconstruction using k-space data from a central window of size 24 × 24.

We divide the k-space of each slice by the 95 th magnitude percentile of this low-resolution reconstruction. We define the SNR as 1/σ 2 w .

1) Denoising Performance: We evaluate denoising performance using the Tweedie re-parameterization in (18) for score models trained (i) naively, (ii) with SURE-Score, and (iii) using noise-free data (supervised).

Table I lists the NRMSE (mean and standard deviation) over 100 validation slices for SNR w levels of 0 dB and 10 dB.

Solving Inverse Problems With Score-Based Generative Priors Learned From Noisy Data

Abstract: We present SURE-Score: an approach for learning score-based generative models using training samples corrupted by additive Gaussian noise. When a large training set of clean samples is available, solving inverse problems via score-based (diffusion) generative models trained on the underlying fullysampled data distribution has recently been shown to outperform end-to-end supervised deep learning. In practice, such a large collection of training data may be prohibitively expensive to acquire in the first place. In this work, we present an approach for approximately learning a score-based generative model of the clean distribution, from noisy training data. We formulate and justify a novel loss function that leverages Stein's unbiased risk estimate to jointly denoise the data and learn the score function via denoising score matching, while using only the noisy samples. We demonstrate the generality of SURE-Score by learning priors and applying posterior sampling to ill-posed inverse problems in two practical applications from different domains: compressive wireless multiple-input multiple-output channel estimation and accelerated 2D multi-coil magnetic resonance imaging reconstruction, where we demonstrate competitive reconstruction performance when learning at signal-to-noise ratio values of 0 and 10 dB, respectively.

Fastmri: An Open Dataset And Benchmarks For Accelerated Mri

Abstract: Accelerating Magnetic Resonance Imaging (MRI) by taking fewer measurements has the potential to reduce medical costs, minimize stress to patients and make MRI possible in applications where it is currently prohibitively slow or expensive. We introduce the fastMRI dataset, a large-scale collection of both raw MR measurements and clinical MR images, that can be used for training and evaluation of machine-learning approaches to MR image reconstruction. By introducing standardized evaluation criteria and a freely-accessible dataset, our goal is to help the community make rapid advances in the state of the art for MR image reconstruction. We also provide a self-contained introduction to MRI for machine learning researchers with no medical imaging background.


The results show that the proposed model has robust performance to discover effective influencers in the groups of all ranges of followers compared to the baseline methods. More specifically, DeepInf #REFR fails to discover effective micro-influencers.

Influencers who are followed by less than 20,000 followers are classified as the Micro influencers.

The Mid-level influencers have followers between 20,000 and 100,000, and Macro influencers have more than 100,000 followers.

In our dataset, around 30% of influencers are the micro-influencers, 45% of them are the mid-level influencers, and the remaining 25% influencers are the macroinfluencers.

To evaluate the performance under the same conditions, we randomly select multiple sets of 1,000 influencers from each category and run the experiment 10 times.

Figure 8 shows the average NDCG scores of Influencer-Rank and four baseline methods, including GCRN #OTHEREFR , DeepInf #OTHEREFR , CasCN #OTHEREFR , and EGCN #OTHEREFR over the micro, mid-level, and macro-influencers.

This is probably because DeepInf disregards the temporal information which is critical to find micro-influencers who have relatively large variance on their features and engagement rates over time compared to macro-influencers who are matured.

On the other hand, our proposed model can accurately find highly effective micro-influencers since their unique features are captured by sequential learning of temporal information.

Influencerrank: Discovering Effective Influencers Via Graph Convolutional Attentive Recurrent Neural Networks

Abstract: As influencers play considerable roles in social media marketing, companies increase the budget for influencer marketing. Hiring effective influencers is crucial in social influencer marketing, but it is challenging to find the right influencers among hundreds of millions of social media users. In this paper, we propose InfluencerRank that ranks influencers by their effectiveness based on their posting behaviors and social relations over time. To represent the posting behaviors and social relations, the graph convolutional neural networks are applied to model influencers with heterogeneous networks during different historical periods. By learning the network structure with the embedded node features, InfluencerRank can derive informative representations for influencers at each period. An attentive recurrent neural network finally distinguishes highly effective influencers from other influencers by capturing the knowledge of the dynamics of influencer representations over time. Extensive experiments have been conducted on an Instagram dataset that consists of 18,397 influencers with their 2,952,075 posts published within 12 months. The experimental results demonstrate that InfluencerRank outperforms existing baseline methods. An in-depth analysis further reveals that all of our proposed features and model components are beneficial to discover effective influencers.

Deepinf: Social Influence Prediction With Deep Learning

Abstract: Social and information networking activities such as on Facebook, Twitter, WeChat, and Weibo have become an indispensable part of our everyday life, where we can easily access friends' behaviors and are in turn influenced by them. Consequently, an effective social influence prediction for each user is critical for a variety of applications such as online recommendation and advertising. Conventional social influence prediction approaches typically design various hand-crafted rules to extract user- and network-specific features. However, their effectiveness heavily relies on the knowledge of domain experts. As a result, it is usually difficult to generalize them into different domains. Inspired by the recent success of deep neural networks in a wide range of computing applications, we design an end-to-end framework, DeepInf, to learn users' latent feature representation for predicting social influence. In general, DeepInf takes a user's local network as the input to a graph neural network for learning her latent social representation. We design strategies to incorporate both network structures and user-specific features into convolutional neural and attention networks. Extensive experiments on Open Academic Graph, Twitter, Weibo, and Digg, representing different types of social and information networks, demonstrate that the proposed end-to-end model, DeepInf, significantly outperforms traditional feature engineering-based approaches, suggesting the effectiveness of representation learning for social applications.


Effect Of Number Of Labeled Training Samples: Full Data Sets

For the ISIC data set, the ISIC 2019 methods paper of the challenge winners Gessert et al. #REFR was used as literature benchmark.

The first set of experiments aimed at a direct comparison with literature values and covered all classes available in the labeled data sets.

For the BM and the Endo data set, balanced accuracy values as reported in the original data set publications #OTHEREFR were considered as literature benchmark and state-of-theart performance.

All benchmark values were obtained through supervised end-to-end trained DL, using the maximum available number of labeled data.

With the SSL image representation as input features, three conventional classifiers with default parameters (see Sec.

3) were trained (support vector common practice in SSL-based image classification #OTHEREFR , we also trained a linear layer (LL) on top of the frozen trained DINO feature extractor.

To provide a DL baseline performance in addition to the literature benchmark data, we also trained the ImageNet-pretrained XCiT network that was used as the DINO backbone network with a linear layer on top in an endto-end supervised DL approach, i.e., using only labeled data instead of SSL training within the DINO framework.

For all approaches, the number of labeled training samples per class was systematically varied between 1 and 1000 (steps: 1, 5, 10, 25, 50, 100, 250, 500, 1000 samples per class).

Self-Supervision For Medical Image Classification: State-Of-The-Art Performance With˜100 Labeled Training Samples Per Class

Abstract: Is self-supervised deep learning (DL) for medical image analysis already a serious alternative to the de facto standard of end-to-end trained supervised DL? We tackle this question for medical image classification, with a particular focus on one of the currently most limiting factor of the field: the (non-)availability of labeled data. Based on three common medical imaging modalities (bone marrow microscopy, gastrointestinal endoscopy, dermoscopy) and publicly available data sets, we analyze the performance of self-supervised DL within the self-distillation with no labels (DINO) framework. After learning an image representation without use of image labels, conventional machine learning classifiers are applied. The classifiers are fit using a systematically varied number of labeled data (1-1000 samples per class). Exploiting the learned image representation, we achieve state-of-the-art classification performance for all three imaging modalities and data sets with only a fraction of between 1% and 10% of the available labeled data and about 100 labeled samples per class.

Skin Lesion Classification Using Ensembles Of Multi-Resolution Efficientnets With Meta Data

Abstract: In this paper, we describe our method for the ISIC 2019 Skin Lesion Classification Challenge. The challenge comes with two tasks. For task 1, skin lesions have to be classified based on dermoscopic images. For task 2, dermoscopic images and additional patient meta data have to be used. A diverse dataset of 25000 images was provided for training, containing images from eight classes. The final test set contains an additional, unknown class. We address this challenging problem with a simple, data driven approach by including external data with skin lesions types that are not present in the training set. Furthermore, multi-class skin lesion classification comes with the problem of severe class imbalance. We try to overcome this problem by using loss balancing. Also, the dataset contains images with very different resolutions. We take care of this property by considering different model input resolutions and different cropping strategies. To incorporate meta data such as age, anatomical site, and sex, we use an additional dense neural network and fuse its features with the CNN. We aggregate all our models with an ensembling strategy where we search for the optimal subset of models. Our best ensemble achieves a balanced accuracy of 74.2% using five-fold cross-validation. On the official test set our method is ranked first for both tasks with a balanced accuracy of 63.6% for task 1 and 63.4% for task 2.


SGD-based: Instead of zeroing out filters, Centripetal SGD (C-SGD) #REFR makes redundant filters identical and merges identical filters into one filter.

Regularized Modernized Dual Averaging (RMDA) #OTHEREFR adds momentum to the RDA algorithm #OTHEREFR and ensures that the trained model has the same structure as the original model.

To prune the model, RMDA adopts Group Lasso to promote structured sparsity.

ADMM-based: Alternating Direction Method of Multipliers (ADMM) #OTHEREFR is an optimization algorithm used to decompose initial problem into two smaller, more tractable subproblems.

StructADMM #OTHEREFR studies the solution of different types of structured sparsity such as filter-wise and shape-wise. Zhang et al.

use a progressive and multi-step ADMM framework: at each step, it uses ADMM to prune and masks out zero weights, leaving the remaining weights as the optimization space for the next step.

Structured Pruning For Deep Convolutional Neural Networks: A Survey

Abstract: The remarkable performance of deep Convolutional neural networks (CNNs) is generally attributed to their deeper and wider architectures, which can come with significant computational costs. Pruning neural networks has thus gained interest since it effectively lowers storage and computational costs. In contrast to weight pruning, which results in unstructured models, structured pruning provides the benefit of realistic acceleration by producing models that are friendly to hardware implementation. The special requirements of structured pruning have led to the discovery of numerous new challenges and the development of innovative solutions. This article surveys the recent progress towards structured pruning of deep CNNs. We summarize and compare the state-of-the-art structured pruning techniques with respect to filter ranking methods, regularization methods, dynamic execution, neural architecture search, the lottery ticket hypothesis, and the applications of pruning. While discussing structured pruning algorithms, we briefly introduce the unstructured pruning counterpart to emphasize their differences. Furthermore, we provide insights into potential research opportunities in the field of structured pruning. A curated list of neural network pruning papers can be found at https://github.com/he-y/Awesome-Pruning.

Centripetal Sgd For Pruning Very Deep Convolutional Networks With Complicated Structure

Abstract: The redundancy is widely recognized in Convolutional Neural Networks (CNNs), which enables to remove unimportant filters from convolutional layers so as to slim the network with acceptable performance drop. Inspired by the linear and combinational properties of convolution, we seek to make some filters increasingly close and eventually identical for network slimming. To this end, we propose Centripetal SGD (C-SGD), a novel optimization method, which can train several filters to collapse into a single point in the parameter hyperspace. When the training is completed, the removal of the identical filters can trim the network with NO performance loss, thus no finetuning is needed. By doing so, we have partly solved an open problem of constrained filter pruning on CNNs with complicated structure, where some layers must be pruned following others. Our experimental results on CIFAR-10 and ImageNet have justified the effectiveness of C-SGD-based filter pruning. Moreover, we have provided empirical evidences for the assumption that the redundancy in deep neural networks helps the convergence of training by showing that a redundant CNN trained using C-SGD outperforms a normally trained counterpart with the equivalent width.


A. Metadata, Filtering And Preprocessing

The SPGC provides the PG corpus on three different levels of granularity: raw text, time series of word tokens, and counts of individual words #REFR .

Texts are tokenized using the "TreebankWordTokenizer" from the Natural Language Toolkit (NLTK) #OTHEREFR .

Only tokens consisting of alphabetic characters are kept, meaning that words containing numbers or other symbols are removed.

While this processing removes any mentions of numerical objects such as years or ages, it is done to ensure that page and chapter numbers are not erroneously included.

Additionally, all tokens are lowercased, as this ensures that words capitalised after full stops or within dialogues are not considered different words to their standard lowercase forms.

PG also provides useful metadata on the texts in the corpus.

Quantifying The Dissimilarity Of Texts

Abstract: Quantifying the dissimilarity of two texts is an important aspect of a number of natural language processing tasks, including semantic information retrieval, topic classification, and document clustering. In this paper, we compared the properties and performance of different dissimilarity measures D using three different representations of texts-vocabularies, word frequency distributions, and vector embeddings-and three simple tasks-clustering texts by author, subject, and time period. Using the Project Gutenberg database, we found that the generalised Jensen-Shannon divergence applied to word frequencies performed strongly across all tasks, that D's based on vector embedding representations led to stronger performance for smaller texts, and that the optimal choice of approach was ultimately task-dependent. We also investigated, both analytically and numerically, the behaviour of the different D's when the two texts varied in length by a factor h. We demonstrated that the (natural) estimator of the Jaccard distance between vocabularies was inconsistent and computed explicitly the h-dependency of the bias of the estimator of the generalised Jensen-Shannon divergence applied to word frequencies. We also found numerically that the Jensen-Shannon divergence and embedding-based approaches were robust to changes in h, while the Jaccard distance was not.

A Standardized Project Gutenberg Corpus For Statistical Analysis Of Natural Language And Quantitative Linguistics

Abstract: The use of Project Gutenberg (PG) as a text corpus has been extremely popular in statistical analysis of language for more than 25 years. However, in contrast to other major linguistic datasets of similar importance, no consensual full version of PG exists to date. In fact, most PG studies so far either consider only a small number of manually selected books, leading to potential biased subsets, or employ vastly different pre-processing strategies (often specified in insufficient details), raising concerns regarding the reproducibility of published results. In order to address these shortcomings, here we present the Standardized Project Gutenberg Corpus (SPGC), an open science approach to a curated version of the complete PG data containing more than 50,000 books and more than 3 × 10 9 word-tokens. Using different sources of annotated metadata, we not only provide a broad characterization of the content of PG, but also show different examples highlighting the potential of SPGC for investigating language variability across time, subjects, and authors. We publish our methodology in detail, the code to download and process the data, as well as the obtained corpus itself on three different levels of granularity (raw text, timeseries of word tokens, and counts of words). In this way, we provide a reproducible, pre-processed, full-size version of Project Gutenberg as a new scientific resource for corpus linguistics, natural language processing, and information retrieval.


Related work has shown that the RFM cannot predict the output of a ReLU perceptron, except the hidden layer size K increases exponentially fast with the input dimension N #REFR .

Based on the bias-variance tradeoff, traditional learning theories suggest that over-parametrized neural networks should generalize poorly as they have more regression parameters than avaiable input examples #OTHEREFR .

However, recent studies have shown that this tradeoff must be extend for ultra-wide neural networks leading to a double descent curve #OTHEREFR .

For the RFM, different analytical studies derived the test error's double descent behavior in the student-teacher framework indicating the accessibility of interesting deep learning phenomena by investigating this model #OTHEREFR .

The connection between kernel ridge regression and random feature approximation can be found in #OTHEREFR .

We verify this result for an error activation function by actually computing the asymptotic generalization error.

Further related studies considered the random feature model for quadratic activation and output functions in the high-dimensional regime K, N → ∞, K/N → τ ∈ (0, ∞) trained by one-pass stochastic gradient descent #OTHEREFR .

These works demonstrated the existence of a gap between the performance of the RFM compared to fully-trained over-parametrized neural networks.

Furthermore, the network performance is highly controlled by the initialization of the weights and becomes optimal only in the case for which the random features are perfectly aligned with those of the teacher.

We extend these investigations by introducing a set of ordinary differential equations for the learning dynamics of a student with infinitely large input and hidden layer size trained by a teacher with a ReLU or error activation function.

Online Learning For The Random Feature Model In The Student-Teacher Framework

Abstract: Deep neural networks are widely used prediction algorithms whose performance often improves as the number of weights increases, leading to over-parametrization. We consider a two-layered neural network whose first layer is frozen while the last layer is trainable, known as the random feature model. We study over-parametrization in the context of a student-teacher framework by deriving a set of differential equations for the learning dynamics. For any finite ratio of hidden layer size and input dimension, the student cannot generalize perfectly, and we compute the non-zero asymptotic generalization error. Only when the student's hidden layer size is exponentially larger than the input dimension, an approach to perfect generalization is possible.

On The Power And Limitations Of Random Features For Understanding Neural Networks

Abstract: Recently, a spate of papers have provided positive theoretical results for training over-parameterized neural networks (where the network size is larger than what is needed to achieve low error). The key insight is that with sufficient over-parameterization, gradient-based methods will implicitly leave some components of the network relatively unchanged, so the optimization dynamics will behave as if those components are essentially fixed at their initial random values. In fact, fixing these explicitly leads to the well-known approach of learning with random features. In other words, these techniques imply that we can successfully learn with neural networks, whenever we can successfully learn with random features. In this paper, we first review these techniques, providing a simple and self-contained analysis for one-hidden-layer networks. We then argue that despite the impressive positive results, random feature approaches are also inherently limited in what they can explain. In particular, we rigorously show that random features cannot be used to learn even a single ReLU neuron with standard Gaussian inputs, unless the network size (or magnitude of the weights) is exponentially large. Since a single neuron is learnable with gradient-based methods, we conclude that we are still far from a satisfying general explanation for the empirical success of neural networks.


Thus by replacing the preprocessing steps with a more modern deep learning-based DIR approaches, we are able to train a pair of models based on the VoxelMorph #REFR framework that are capable of disentangling the different stages of multi-modal registration.

During testing, CBCT0 is unused, and is replaced by the aligned CT.

on related technologies and seemingly similar visually, CT and CBCT present distinct characteristics in pixel intensities, edge resolution and artifacting #OTHEREFR .

Older methods of DIR incorporated an initial intensity correction preprocessing step so that the registration could be seen as a mono-modal task #OTHEREFR .

More recent approaches leveraging deep learning attempted to perform both tasks in a single fully automated way #OTHEREFR .

Nonetheless, we are proposing to follow the former two stage approach precisely to distinguish between multi-modal mapping space from mono-modal transformations resulting uniquely from morphological changes in response to radiotherapy.

And this will become relevant as our proposed method focuses specifically on extracting the transformation related to the second stage: registration resulting from anatomical changes before and during therapy.

To train under this proposed methodology, an initial model learned the mapping from pCT and CBCT 0 (see Figure 1) .

Independently, a second model was trained to register CBCT that were deformed due to changes in anatomy resulting from radiotherapy treatment CBCT 0 → CBCT t ).

Fixing the input image to CBCT 0 has the advantage of resembling the pCT the most since no treatment occurred between those two dates, which will become relevant at testing time.

At that stage then after both models are trained, the pCT is registered first to CBCT 0 , then that image (Aligned CT in Figure 1) is then fed into the second model and registered to the CBCT at the current fraction of treatment (not shown).

Comparing 3D Deformations Between Longitudinal Daily Cbct Acquisitions Using Cnn For Head And Neck Radiotherapy Toxicity Prediction

Abstract: Adaptive radiotherapy is a growing field of study in cancer treatment due to it's objective in sparing healthy tissue. The standard of care in several institutions includes longitudinal cone-beam computed tomography (CBCT) acquisitions to monitor changes, but have yet to be used to improve tumor control while managing side-effects. The aim of this study is to demonstrate the clinical value of pre-treatment CBCT acquired daily during radiation therapy treatment for head and neck cancers for the downstream task of predicting severe toxicity occurrence: reactive feeding tube (NG), hospitalization and radionecrosis. For this, we propose a deformable 3D classification pipeline that includes a component analyzing the Jacobian matrix of the deformation between planning CT and longitudinal CBCT, as well as clinical data. The model is based on a multi-branch 3D residual convolutional neural network, while the CT to CBCT registration is based on a pair of VoxelMorph architectures. Accuracies of 85.8% and 75.3% was found for radionecrosis and hospitalization, respectively, with similar performance as early as after the first week of treatment. For NG tube risk, performance improves with increasing the timing of the CBCT fraction, reaching 83.1% after the 5 th week of treatment.

Voxelmorph: A Learning Framework For Deformable Medical Image Registration

Abstract: We present VoxelMorph, a fast learning-based framework for deformable, pairwise medical image registration. Traditional registration methods optimize an objective function for each pair of images, which can be time-consuming for large datasets or rich deformation models. In contrast to this approach and building on recent learning-based methods, we formulate registration as a function that maps an input image pair to a deformation field that aligns these images. We parameterize the function via a convolutional neural network and optimize the parameters of the neural network on a set of images. Given a new pair of scans, VoxelMorph rapidly computes a deformation field by directly evaluating the function. In this paper, we explore two different training strategies. In the first (unsupervised) setting, we train the model to maximize standard image matching objective functions that are based on the image intensities. In the second setting, we leverage auxiliary segmentations available in the training data. We demonstrate that the unsupervised model’s accuracy is comparable to the state-of-the-art methods while operating orders of magnitude faster. We also show that VoxelMorph trained with auxiliary data improves registration accuracy at test time and evaluate the effect of training set size on registration. Our method promises to speed up medical image analysis and processing pipelines while facilitating novel directions in learning-based registration and its applications. Our code is freely available at https://github.com/voxelmorph/voxelmorph .


B. Cnn Architecture For Interference Classification

All convolutional layers are subject to batch normalization #REFR , however the output layer and fullyconnected layers are excluded.

The CNN model is a pre-trained ResNet18 model described in #OTHEREFR . The architecture is summarized in Table II .

The model consists of many convolutional layers, two fully-connected layers, and one output layer, in that order.

Leaky ReLU (Leaky ReLU(x) = max{αx, x}, where α ∈ (0, 1) is a preset parameter).

All convolutional layers and fully linked layers are subjected to an activation function with α = 0.2.

The resulting (output) layer then has the softmax function applied to it.

Additionally, we use stride = 2 in the convolutional layers rather than 2 × 2 pooling layers for down-sampling.

Such changes enhance performance and lower the variance of the results across various training epochs.

x is normalized as x = x/x max , where x max is the largest input value allowed in x.

We utilize the Adam optimizer with the suggested default values in #OTHEREFR .

The learning rate is 1×10 −4 , and the batch size is 64.

Keep It Simple: Cnn Model Complexity Studies For Interference Classification Tasks

Abstract: The growing number of devices using the wireless spectrum makes it important to find ways to minimize interference and optimize the use of the spectrum. Deep learning models, such as convolutional neural networks (CNNs), have been widely utilized to identify, classify, or mitigate interference due to their ability to learn from the data directly. However, there have been limited research on the complexity of such deep learning models. The major focus of deep learning-based wireless classification literature has been on improving classification accuracy, often at the expense of model complexity. This may not be practical for many wireless devices, such as, internet of things (IoT) devices, which usually have very limited computational resources and cannot handle very complex models. Thus, it becomes important to account for model complexity when designing deep learning based models for interference classification. To address this, we conduct an analysis of CNN based wireless classification that explores the trade-off amongst dataset size, CNN model complexity, and classification accuracy under various levels of classification difficulty: namely, interference classification, heterogeneous transmitter classification, and homogeneous transmitter classification. Our study, based on three wireless datasets, shows that a simpler CNN model with fewer parameters can perform just as well as a more complex model, providing important insights into the use of CNNs in computationally constrained applications.

Batch Normalization: Accelerating Deep Network Training By Reducing Internal Covariate Shift

Abstract: Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.

convolutional layers
batch normalization


Masked Language Models And The Off-Manifold Conjecture

RoBERTa #REFR improves BERT by employing the dynamic masking strategy, which dynamically changes the masking pattern applied to training data.

As mentioned earlier, masked language models are pre-trained on large-scale unlabeled normal data with the MLM objective.

This pretraining equips the models with the capacity of reconstructing input data.

As the most representative example of the masked language model, BERT #OTHEREFR is a bidirectional language model trained with the MLM objective to learn general language representations from unlabeled data, leading to a series of breakthroughs in NLP.

ALBERT #OTHEREFR is pre-trained on the same data as BERT, but it adopts the ngram masking strategy of the MLM objective.

The MLM target consists of up to an n-gram of complete words, which encourages the model to capture more comprehensive language representations.

Additionally, it is trained over a variety of corpora with more sophisticated techniques, such as training the model longer and with bigger batches.

Thus, it has the potential to better approximate the manifold of normal data.

The off-manifold conjecture #OTHEREFR 13, #OTHEREFR states that adversarial examples tend to deviate from the manifold of normal examples.

This conjecture offers a new perspective to interpret the existence of adversarial examples and has attracted significant research attention #OTHEREFR .

In computer vision tasks, under the off-manifold conjecture, substantial detection approaches #OTHEREFR have approximated the manifold of normal images by generative models and designed effective metrics to measure changes in the manifold to detect adversarial images. In NLP, Nguyen et al.

Masked Language Model Based Textual Adversarial Example Detection

Abstract: Adversarial attacks are a serious threat to the reliable deployment of machine learning models in safety-critical applications. They can misguide current models to predict incorrectly by slightly modifying the inputs. Recently, substantial work has shown that adversarial examples tend to deviate from the underlying data manifold of normal examples, whereas pre-trained masked language models can fit the manifold of normal NLP data. To explore how to use the masked language model in adversarial detection, we propose a novel textual adversarial example detection method, namely Masked Language Model-based Detection (MLMD), which can produce clearly distinguishable signals between normal examples and adversarial examples by exploring the changes in manifolds induced by the masked language model. MLMD features a plug and play usage (i.e., no need to retrain the victim model) for adversarial defense and it is agnostic to classification tasks, victim model's architectures, and to-be-defended attack methods. We evaluate MLMD on various benchmark textual datasets, widely studied machine learning models, and state-of-the-art (SOTA) adversarial attacks (in total 3 * 4 * 4 = 48 settings). Experimental results show that MLMD can

Roberta: A Robustly Optimized Bert Pretraining Approach

Abstract: Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.


Unlike VR-based training applications typically striving to maximize user's behavioral validity #REFR , the prime objective of our simulated field study was rather to elicit valid user reflections.

A case could then be made that the phenomenological nature of our enquiry, with expert users drawing on their own personal experience, has permitted some level of abstraction and simplification with regards to the simulated interactions.

Past studies seem to support this; Osterlund et al., for instance, found that a purely audiovisual simulation of aerospace maintenance operations in protective suits could elicit valid and actionable design feedback from subject matter experts #OTHEREFR .

Whilst it is therefore unlikely that the monosensory nature of our VR simulation impeded the overall validity of the presented findings, we did also come across several instances where a broader sensory engagement delivered via a mixed reality interface would have likely resulted in richer and more accurate user feedback.

Below, we discuss key problem areas in this vein and offer our recommendations concerning the development of future interfaces for simulation of lunar surface scenarios.

• Haptics -The lack of physical properties, including a sense of touch, weight and rigidity hampered our participant's ability to assess key ergonomic aspects surrounding the manipulation of cargo containers and other artifacts in our simulation.

Using Virtual Reality To Shape Humanity'S Return To The Moon: Key Takeaways From A Design Study

Abstract: Revived interest in lunar exploration is heralding a new generation of design solutions in support of human operations on the Moon. While space system design has traditionally been guided by prototype deployments in analogue studies, the resource-intensive nature of this approach has largely precluded application of proficient user-centered design (UCD) methods from human-computer interaction (HCI). This paper explores possible use of Virtual Reality (VR) to simulate analogue studies in lab settings and thereby bring to bear UCD in this otherwise engineering-dominated field. Drawing on the ongoing development of the European Large Logistics Lander, we have recreated a prospective lunar operational scenario in VR and evaluated it with a group of astronauts and space experts (n=20). Our qualitative findings demonstrate the efficacy of VR in facilitating UCD, enabling efficient contextual inquiries and improving project team coordination. We conclude by proposing future directions to further exploit VR in lunar systems design. CCS CONCEPTS • Human-centered computing → Virtual reality; Human computer interaction (HCI).

The Heat Is On: Exploring User Behaviour In A Multisensory Virtual Environment For Fire Evacuation

Abstract: Understanding validity of user behaviour in Virtual Environments (VEs) is critical as they are increasingly being used for serious Health and Safety applications such as predicting human behaviour and training in hazardous situations. This paper presents a comparative study exploring user behaviour in VE-based fire evacuation and investigates whether this is affected by the addition of thermal and olfactory simulation. Participants (N=43) were exposed to a virtual fire in an office building. Quantitative and qualitative analyses of participant attitudes and behaviours found deviations from those we would expect in real life (e.g. pre-evacuation actions), but also valid behaviours like fire avoidance. Potentially important differences were found between multisensory and audiovisual-only conditions (e.g. perceived urgency). We conclude VEs have significant potential in safety-related applications, and that multimodality may afford additional uses in this context, but the identified limitations of behavioural validity must be carefully considered to avoid misapplication of the technology.


It also adopts a scaling factor from the Transformer #REFR , which scales the product of activation matrices QK , Q, K ∈ R s×d by 1/ √ d.

Since the initial distribution of parameters is directly controlled by the model designer, the dominant approach to scaling is to select initial parameter variance to trade off forward and backward pass variance scaling (Glorot and Bengio, 2010; He et al., 2015) .

Such schemes were developed to avoid exploding/vanishing gradients in deep multilayer perceptrons.

As such, they do not seek to constrain the scale of parameters and parameter gradients.

They are also limited to computations where scale factors can be moved into trainable parameters.

Example: BERT #OTHEREFR BERT's initialisation scheme does not use the rules of Glorot and Bengio (2010), instead initialising all non-bias parameters from N (0, (0.02) 2 ).

We instrument the model to record histograms of all tensors at the start and end of training, and plot the results in Figures A.4 and A.6.

In light of this analysis, we can understand loss scaling as simply enacting a shift of the gradx and gradw histograms by log 2 (loss scale) bits to the right, trading off underflow and overflow globally across gradient tensors.

BERT with loss scaling illustrates the drawbacks of having just three scales: weight initialisation scale, loss scale, and QK scale.

These are not sufficient to centre most tensors' distributions in the representable range.

Unit Scaling: Out-Of-The-Box Low-Precision Training

Abstract: We present unit scaling, a paradigm for designing deep learning models that simplifies the use of low-precision number formats. Training in FP16 or the recently proposed FP8 formats offers substantial efficiency gains, but can lack sufficient range for out-of-the-box training. Unit scaling addresses this by introducing a principled approach to model numerics: seeking unit variance of all weights, activations and gradients at initialisation. Unlike alternative methods, this approach neither requires multiple training runs to find a suitable scale nor has significant computational overhead. We demonstrate the efficacy of unit scaling across a range of models and optimisers. We further show that existing models can be adapted to be unit-scaled, training BERT LARGE in FP16 and then FP8 with no degradation in accuracy.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


The core idea in most works in this area is that comparing and contrasting the latent representations of examples that are expected to share certain similarities (e.g., augmented versions of the same image) can benefit the trained weights and help with regularizing the learned decision boundaries #REFR .

It has also been shown that physiological readings made by basic smartwatch sensors enable efficient modeling of perceived stress response #OTHEREFR .

In the health analytics domain, data and human annotations are often limited.

Therefore, dealing with overfitting and memorization is a crucial matter.

Additionally, it is beneficial to go beyond the limited number of human annotations available in training efficient inference pipelines.

Less reliance on annotations by focusing on unsupervised and self-supervised approaches has received a lot of research attention in recent years #OTHEREFR - #OTHEREFR .

In short, this work is primarily focused on addressing the limitations in the previous literature on remote mental health monitoring.

The previous works do not go beyond leveraging scarcely available annotations in training network parameters and mainly rely on data augmentation to improve their performance.

They do not focus on encapsulation in embedding different modalities, which can be an obstacle in employing optimal encoders for each modality and can hinder transfer learning.

Additionally, they do not focus on the interpretability of the inference pipeline, which is crucial in health-related applications.

To address these challenges, this work proposes a framework for leveraging smartwatch-based sensor-driven data to recognize perceived stress, enabling a novel approach to remote mental health monitoring.

A Self-Supervised Framework For Improved Data-Driven Monitoring Of Stress Via Multi-Modal Passive Sensing

Abstract: Recent advances in remote health monitoring systems have significantly benefited patients and played a crucial role in improving their quality of life. However, while physiological health-focused solutions have demonstrated increasing success and maturity, mental health-focused applications have seen comparatively limited success in spite of the fact that stress and anxiety disorders are among the most common issues people deal with in their daily lives. In the hopes of furthering progress in this domain through the development of a more robust analytic framework for the measurement of indicators of mental health, we propose a multi-modal semi-supervised framework for tracking physiological precursors of the stress response. Our methodology enables utilizing multi-modal data of differing domains and resolutions from wearable devices and leveraging them to map short-term episodes to semantically efficient embeddings for a given task. Additionally, we leverage an inter-modality contrastive objective, with the advantages of rendering our framework both modular and scalable. The focus on optimizing both local and global aspects of our embeddings via a hierarchical structure renders transferring knowledge and compatibility with other devices easier to achieve. In our pipeline, a task-specific pooling based on an attention mechanism, which estimates the contribution of each modality on an instance level, computes the final embeddings for observations. This additionally provides a thorough diagnostic insight into the data characteristics and highlights the importance of signals in the broader view of predicting episodes annotated per mental health status. We perform training experiments using a corpus of realworld data on perceived stress, and our results demonstrate the efficacy of the proposed approach in performance improvements 1 .

Mixup: Beyond Empirical Risk Minimization

Abstract: Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.

latent representations
learned decision boundaries


In order to achieve accurate and effective extraction of software specifications from natural language texts, we propose to use the state-of-the-art BERT #REFR model.

By leveraging the acquired knowledge and advanced language processing capabilities of a pre-trained BERT model, we aim to extract relevant specifications from the text in a manner that is both accurate and efficient.

Through empirical analysis, we demonstrate the effectiveness of SpecSyn in the context of software specification extraction from natural language texts.

To the best of our knowledge, this is the first paper that uses a large language model for end-to-end synthesis of configurations specifications from natural language texts.

Large Language Models Based Automatic Synthesis Of Software Specifications

Abstract: Software configurations play a crucial role in determining the behavior of software systems. In order to ensure safe and error-free operation, it is necessary to identify the correct configuration, along with their valid bounds and rules, which are commonly referred to as software specifications. As software systems grow in complexity and scale, the number of configurations and associated specifications required to ensure the correct operation can become large and prohibitively difficult to manipulate manually. Due to the fast pace of software development, it is often the case that correct software specifications are not thoroughly checked or validated within the software itself. Rather, they are frequently discussed and documented in a variety of external sources, including software manuals, code comments, and online discussion forums. Therefore, it is hard for the system administrator to know the correct specifications of configurations due to the lack of clarity, organization, and a centralized unified source to look at. To address this challenge, we propose SpecSyn a framework that leverages a state-of-the-art large language model to automatically synthesize software specifications from natural language sources. Our approach formulates software specification synthesis as a sequence-to-sequence learning problem and investigates the extraction of specifications from large contextual texts. This is the first work that uses a large language model for end-to-end specification synthesis from natural language texts. Empirical results demonstrate that our system outperforms prior state-of-the-art specification synthesis tool by 21% in terms of F1 score and can find specifications from single as well as multiple sentences.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

software specifications
natural language texts


Although TorchBench supports other data precisions such as FP16 (half-precision), BF16 (Brain Floating Point Format), and AMP (automatic mixed precision) #REFR , FP32 and TF32 are the most representative and recommended precisions.

For inference, the original model code usually does not specify an optimal batch_size of inputs.

Thus, we run a set of tests with exhaustively enumerating batch_size values (i.e., starting with one and doubling the size in each test) to determine the optimal value that yields the highest GPU utilization.

We believe our configuration enables optimal performance for each model in TorchBench.

We use the 32-bit floating point operations (FP32 or TF32) for all the benchmarks unless models have specific requirements.

On the one hand, GPU vendors such as NVIDIA publish concrete roofline performance numbers for FP32 operations in TFLOPS (Tera Floating Point Operations per Second) as part of the hardware specification, which facilitates our characterization efforts.

For example, the theoretical peaks of an NVIDIA Tesla A100 GPU and an AMD Instinct MI210 GPU are 19.

Recently, TorchBench has evolved to embrace end-to-end models and support beyond PyTorch (e.g., Jax).

However, this evolution is in the preliminary stage and out of the scope of this paper.

Torchbench: Benchmarking Pytorch With High Api Surface Coverage

Abstract: Deep learning (DL) has been a revolutionary technique in various domains. To facilitate the model development and deployment, many deep learning frameworks are proposed, among which Py-Torch is one of the most popular solutions. The performance of ecosystem around PyTorch is critically important, which saves the costs of training models and reduces the response time of model inferences. In this paper, we propose TorchBench, a novel benchmark suite to study the performance of PyTorch software stack. Unlike existing benchmark suites, TorchBench encloses many representative models, covering a large PyTorch API surface. TorchBench is able to comprehensively characterize the performance of the Py-Torch software stack, guiding the performance optimization across models, PyTorch framework, and GPU libraries. We show two practical use cases of TorchBench. (1) We profile TorchBench to identify GPU performance inefficiencies in PyTorch. We are able to optimize many performance bugs and upstream patches to the official PyTorch repository. (2) We integrate TorchBench into PyTorch continuous integration system. We are able to identify performance regression in multiple daily code checkins to prevent PyTorch repository from introducing performance bugs. TorchBench is open source and keeps evolving.

Mixed Precision Training

Abstract: Deep neural networks have enabled progress in a wide variety of applications. Growing the size of the neural network typically results in improved accuracy. As model sizes grow, the memory and compute requirements for training these models also increases. We introduce a technique to train deep neural networks using half precision floating point numbers. In our technique, weights, activations and gradients are stored in IEEE half-precision format. Half-precision floating numbers have limited numerical range compared to single-precision numbers. We propose two techniques to handle this loss of information. Firstly, we recommend maintaining a single-precision copy of the weights that accumulates the gradients after each optimizer step. This single-precision copy is rounded to half-precision format during training. Secondly, we propose scaling the loss appropriately to handle the loss of information with half-precision gradients. We demonstrate that this approach works for a wide variety of models including convolution neural networks, recurrent neural networks and generative adversarial networks. This technique works for large scale models with more than 100 million parameters trained on large datasets. Using this approach, we can reduce the memory consumption of deep learning models by nearly 2x. In future processors, we can also expect a significant computation speedup using half-precision hardware units.


GPipe's #REFR ImageNet top-1 accuracy is improved to 84.3% by scaling up the baseline model by 4 times.

CNNs are often scaled up to achieve better performance.

For example, ResNet #OTHEREFR can be scaled up from ResNet-18 to ResNet-200 by incorporating more layers.

Scaling CNNs up by their depth #OTHEREFR or width #OTHEREFR are the most commonly performed, but scaling up models by image resolution #OTHEREFR is also a popular method.

Only one of the three dimensions -depth, width, and image size -is usually scaled.

Although two or three dimensions can be arbitrarily scaled, it involves tedious manual tuning and still often fails to provide better accuracy and efficiency.

To resolve this, EfficientNet proposes a simple yet effective compound scaling method that uses a constant scaling ratio to scale all three dimensions of network in a controlled and balanced way.

The intuition is: as the input image size increases, it makes sense that the model will need more layers and more channels to extract even finer details from larger images.

Speeding Up Efficientnet: Selecting Update Blocks Of Convolutional Neural Networks Using Genetic Algorithm In Transfer Learning

Abstract: The performance of convolutional neural networks (CNN) depends heavily on their architectures. Transfer learning performance of a CNN relies quite strongly on selection of its trainable layers. Selecting the most effective update layers for a certain target dataset often requires expert knowledge on CNN architecture which many practitioners do not posses. General users prefer to use an available architecture (e.g. GoogleNet, ResNet, EfficientNet etc.) that is developed by domain experts. With the ever-growing number of layers, it is increasingly becoming quite difficult and cumbersome to handpick the update layers. Therefore, in this paper we explore the application of genetic algorithm to mitigate this problem. The convolutional layers of popular pre-trained networks are often grouped into modules that constitute their building blocks. We devise a genetic algorithm to select blocks of layers for updating the parameters. By experimenting with EfficientNetB0 pre-trained on ImageNet and using Food-101, CIFAR-100 and MangoLeafBD as target datasets, we show that our algorithm yields similar or better results than the baseline in terms of accuracy, and requires lower training and evaluation time due to learning less number of parameters. We also devise a metric called block importance to measure each block's efficacy as update block and analyze the importance of the blocks selected by our algorithm.

Gpipe: Efficient Training Of Giant Neural Networks Using Pipeline Parallelism

Abstract: Scaling up deep neural network capacity has been known as an effective approach to improving model quality for several different machine learning tasks. In many cases, increasing model capacity beyond the memory limit of a single accelerator has required developing special algorithms or infrastructure. These solutions are often architecture-specific and do not transfer to other machine learning tasks. To address the need for efficient and task-independent model parallelism, we introduce TensorPipe, a pipeline parallelism library that allows scaling any network that can be expressed as a sequence of layers. By pipelining different sub-sequences of layers on separate accelerators, TensorPipe provides the flexibility of scaling a variety of different networks to gigantic sizes efficiently. Moreover, TensorPipe utilizes a novel batch-splitting pipelining algorithm, resulting in almost linear speedup when a model is partitioned across multiple accelerators. We demonstrate the advantages of TensorPipe by training large-scale neural networks on two different tasks with distinct network architectures: (i)Image Classification: We train a 557-million-parameter AmoebaNet model and attain a top-1 accuracy of 84.4% on ImageNet-2012, (ii)Multilingual Neural Machine Translation: We train a single 6-billion-parameter, 128-layer Transformer model on a corpus spanning over 100 languages and achieve better quality than all bilingual models.


The results presented in table 7 also align with the philosophy of the residual connection #REFR . Note that input-agnostic kernel W inputagnostic is initialized randomly.

#OTHEREFR stabilized the learning process by reducing the kernel space using softmax and demonstrated better performance with fewer kernels and FLOPs.

But, as shown in the table 6, we utilize the sigmoid function to enhance performance by increasing the kernel space, as our residual connection in kernel space helps to alleviate optimization difficulties.

We assume that the previous approach #OTHEREFR of using the dynamic convolution module to calculate both input-agnostic and input-dependent kernels in the kernel attention module results in optimization difficulties.

Therefore, to address this issue, we introduce the residual connection that connects input-specific dynamic kernel to the existing input-agnostic kernel.

We not only use temperature annealing to achieve stable training, but also utilize residual connection in kernel space to ease the optimization and expand the kernel space.

Table 8 demonstrates that our dynamic residual convolution approach enables us to increase the kernel space by increasing the number of static kernels while maintaining stable training and improved performance.

Therefore, all variants of DMF take 8 static kernels, which is zero initialized and use sigmoid activation for generating attention scores.

Dynamic Mobile-Former: Strengthening Dynamic Convolution With Attention And Residual Connection In Kernel Space

Abstract: We introduce Dynamic Mobile-Former(DMF), maximizes the capabilities of dynamic convolution by harmonizing it with efficient operators. Our Dynamic Mobile-Former effectively utilizes the advantages of Dynamic Mo-bileNet (MobileNet equipped with dynamic convolution) using global information from lightweight attention. A Transformer in Dynamic Mobile-Former only requires a few randomly initialized tokens to calculate global features, making it computationally efficient. And a bridge between Dynamic MobileNet and Transformer allows for bidirectional integration of local and global features. We also simplify the optimization process of vanilla dynamic convolution by splitting the convolution kernel into an input-agnostic kernel and an input-dependent kernel. This allows for optimization in a wider kernel space, resulting in enhanced capacity. By integrating lightweight attention and enhanced dynamic convolution, our Dynamic Mobile-Former achieves not only high efficiency, but also strong performance. We benchmark the Dynamic Mobile-Former on a series of vision tasks, and showcase that it achieves impressive performance on image classification, COCO detection, and instanace segmentation. For example, our DMF hits the top-1 accuracy of 79.4% on ImageNet-1K, much higher than PVT-Tiny by 4.3% with only 1/4 FLOPs. Additionally, our proposed DMF-S model performed well on challenging vision datasets such as COCO, achieving a 39.0% mAP, which is 1% higher than that of the Mobile-Former 508M model, despite using 3 GFLOPs less computations. Code and models are available at https://github.com/ ysj9909/DMF

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


Both encoders share the same structure, including an attention layer and a Mixture-of-Expert (MoE) architecture #REFR .

The structure of M 2 SE is shown in Figure 1 (b).

Observe that M 2 SE includes four main components: sequence random dropout, text and image encoders, complementary sequence mixup, and transformer layers. Next, we introduce the details of each component.

1) Sequence Random Dropout: For a user behavior sequence S, M 2 SE randomly drops a portion of items from S with a drop ratio ρ, to help the model achieve better generalization performance.

The obtained sub-sequence after the random dropout operation is denoted by S.

2) Text and Image Encoders: These two encoders are used to adapt the initial modality features of items obtained from the pre-trained language model to learn users' sequential behaviors.

In the text encoder, each item i ∈ S is represented by its initial textual feature x t i .

The attention layer is composed of two linear transformations to fuse i's sentence-level embeddings as follows,

where is the j-th element of α t , and x t i [j, :] denotes the j-th row of feature matrix x t i .

Then, MoE is used to increase the model's capacity for adapting the fused modality representation e t i .

Multimodal Pre-Training Framework For Sequential Recommendation Via Contrastive Learning

Abstract: Sequential recommendation systems utilize the sequential interactions of users with items as their main supervision signals in learning users' preferences. However, existing methods usually generate unsatisfactory results due to the sparsity of user behavior data. To address this issue, we propose a novel pre-training framework, named Multimodal Sequence Mixup for Sequential Recommendation (MSM4SR), which leverages both users' sequential behaviors and items' multimodal content (i.e., text and images) for effectively recommendation. Specifically, MSM4SR tokenizes each item image into multiple textual keywords and uses the pre-trained BERT model to obtain initial textual and visual features of items, for eliminating the discrepancy between the text and image modalities. A novel backbone network, i.e., Multimodal Mixup Sequence Encoder (M 2 SE), is proposed to bridge the gap between the item multimodal content and the user behavior, using a complementary sequence mixup strategy. In addition, two contrastive learning tasks are developed to assist M 2 SE in learning generalized multimodal representations of the user behavior sequence. Extensive experiments on real-world datasets demonstrate that MSM4SR outperforms state-of-the-art recommendation methods. Moreover, we further verify the effectiveness of MSM4SR on other challenging tasks including cold-start and cross-domain recommendation.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer

Abstract: The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.


For example, consider the following three tables: Since metadata and annotations are provided in a well defined format, it is possible to automatically create documentation in form of data cards or datasheets #REFR .

Datasets does not support organising annotations into different tables, or referencing the same audio files or parts of it multiple times.

Each data point that is returned contains the actual audio signal, a link to the corresponding audio file and associated labels.

Whereas in audb there is only a loose connection between audio files and annotations.

This means there is exactly one copy of an audio file, even when it is referenced from different tables or different, possibly overlapping segmentation exist.

It further allows mapping annotations from one table to another.

Figure 2 shows an example data card for the emodb dataset #OTHEREFR .

It summarizes the most important facts in a tabular form, provides a long description of the dataset together with an audio example and lists available tables, columns, and schemes.

Data cards for all datasets available with the default public repositories of audb are available at https://audeering.github.io/datasets/.

Audb -Sharing And Versioning Of Audio And Annotation Data In Python

Abstract: Driven by the need for larger and more diverse datasets to pre-train and fine-tune increasingly complex machine learning models, the number of datasets is rapidly growing. audb is an open-source Python library that supports versioning and documentation of audio datasets. It aims to provide a standardized and simple user-interface to publish, maintain, and access the annotations and audio files of a dataset. To efficiently store the data on a server, audb automatically resolves dependencies between versions of a dataset and only uploads newly added or altered files when a new version is published. The library supports partial loading of a dataset and local caching for fast access. audb is a lightweight library and can be interfaced from any machine learning library. It supports the management of datasets on a single PC, within a university or company, or within a whole research community. audb is available at https://github.com/audeering/audb.

Datasheets For Datasets

Abstract: The machine learning community currently has no standardized process for documenting datasets, which can lead to severe consequences in high-stakes domains. To address this gap, we propose datasheets for datasets. In the electronics industry, every component, no matter how simple or complex, is accompanied with a datasheet that describes its operating characteristics, test results, recommended uses, and other information. By analogy, we propose that every dataset be accompanied with a datasheet that documents its motivation, composition, collection process, recommended uses, and so on. Datasheets for datasets will facilitate better communication between dataset creators and dataset consumers, and encourage the machine learning community to prioritize transparency and accountability.


We also include results for Sentence-BERT (SBERT) #REFR , since its output embeddings are trained to have meaningful cosine similarity scores and thus bear more similarity to other models evaluated with Stroop proving.

For evaluating unimodally trained text encoders, we use BERT #OTHEREFR , RoBERTa #OTHEREFR , DistilBERT and Distil-RoBERTa #OTHEREFR , which are all trained with text-only MLM objectives.

Results on multimodally trained text encoders are reported for CLIP #OTHEREFR and FLAVA #OTHEREFR ; for these models we use only the text encoder with pretrained weights and discard the other subcomponents.

Our tests include checkpoints from both OpenAI and the OpenCLIP open-source implementation of CLIP #OTHEREFR 25] .

Details of the checkpoints used for each model are listed in the supplementary material.

The text encoders of the multimodally trained models range in size from 63M (CLIP) to 109M (FLAVA) param-eters.

We compare to both comparably small unimodally trained text encoders such as DistilBERT (66M parameters) as well as much larger text encoders such as BERT-large (340M).

Is Bert Blind? Exploring The Effect Of Vision-And-Language Pretraining On Visual Language Understanding

Abstract: Alex gave Riley a present Alex gave Riley an ultimatum the ocean is [ * ]-colored a corn chip is [ * ]-shaped How concrete are the words present and ultimatum? What word should be inserted in the blank? Figure 1. In this paper, we propose a suite of visual language understanding tasks for probing the visual reasoning capabilities of text encoder models. While we consider text-only tasks (i.e., processing only the textual descriptions above without associated imagery), we argue that they require visual imagination to complete and can thus benefit from vision-and-language pretraining. For instance, consider the words present and ultimatum. A simple online query (that considers only freely-available images) roughly yields a coherent set of images for the more concrete word (namely present), while the latter cannot be uniquely depicted. Likewise, selecting the most natural color or shape descriptors in cloze contexts as shown in the two examples on the right requires implicit knowledge of the appearance of the referent under consideration (ocean and corn chip respectively).

Sentence-Bert: Sentence Embeddings Using Siamese Bert-Networks

Abstract: BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. ::: In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. ::: We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.


Similarly, in self-supervised learning, Momentum Contrast (MOCO) #REFR uses an external memory along with a separate encoding network to compute the features and uses momentum based updates to ensure slow-drift of encoder parameters.

We only mentioned a few works here and we refer the interested reader to #OTHEREFR 27] for a comprehensive list of metric learning losses and example mining strategies.

In contrast to these works, we focus on expanding the reference set which allows us to compare more examples across minibatches while benefiting from these recent advances.

Using external memory in metric learning is not new #OTHEREFR .

However, an im-portant distinction is that we use an external memory to expand the reference set and ensure that those embeddings are kept up to date so that they can be used for effective learning.

Another conceptually similar approach is cross iteration batch normalization #OTHEREFR , which estimates minibatch statistics across training iterations using first order information.

In contrast to both of these methods, we simply store the feature embeddings at each iteration and directly tackle the representational drift by adapting the distribution parameters for effective metric learning.

Adaptive Cross Batch Normalization For Metric Learning

Abstract: Metric learning is a fundamental problem in computer vision whereby a model is trained to learn a semantically useful embedding space via ranking losses. Traditionally, the effectiveness of a ranking loss depends on the minibatch size, and is, therefore, inherently limited by the memory constraints of the underlying hardware. While simply accumulating the embeddings across minibatches has proved useful [40], we show that it is equally important to ensure that the accumulated embeddings are up to date. In particular, it is necessary to circumvent the representational drift between the accumulated embeddings and the feature embeddings at the current training iteration as the learnable parameters are being updated. In this paper, we model representational drift as distribution misalignment and tackle it using moment matching. The result is a simple method for updating the stored embeddings to match the first and second moments of the current embeddings at each training iteration. Experiments on three popular image retrieval datasets, namely, SOP, IN-SHOP and DEEPFASHION2, demonstrate that our approach significantly improves the performance in all scenarios.

Momentum Contrast For Unsupervised Visual Representation Learning

Abstract: We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.

self-supervised learning
Momentum Contrast


Secrecy gains of the so-called unimodular lattices have been studied for well over a decade #REFR .

In #OTHEREFR - #OTHEREFR , the authors introduce the secrecy function, which is evaluated in terms of the lattice's volume and theta series, and can be interpreted as the security coding gain of a specifically designed lattice for Eve compared to an uncoded lattice.

The maximum of the secrecy function, namely the (strong) secrecy gain, has been shown to be an essential design criterion for wiretap lattice codes. Please see Section V-A for an explicit description.

Another design criterion for wiretap lattice codes, called the flatness factor, was proposed by Ling et al. #OTHEREFR .

The flatness factor quantifies how much secret information can leak to Eve in terms of mutual information, while the secrecy gain characterizes the success probability of Eve to correctly guess the transmitted messages.

Both the secrecy gain and the flatness factor require the minimization of the theta series of the lattice designed to confuse Eve at a given point to guarantee secrecy-goodness #OTHEREFR , #OTHEREFR .

In this pioneering work, Belfiore and Solé discovered that there exists a symmetry point in the secrecy functions.

Further, they conjectured that for unimodular lattices, the secrecy gain is achieved at the symmetry point of its secrecy function.

The conjecture has been further investigated and verified for unimodular (or isodual) lattices in dimensions less than 80 #OTHEREFR , #OTHEREFR - #OTHEREFR .

The study of secrecy gain was recently also extended to the ℓ-modular lattices #OTHEREFR , #OTHEREFR , #OTHEREFR , where it is believed that the higher the parameter ℓ is, the better secrecy gain we can achieve.

Most recently, a new family of lattices, called formally unimodular lattices, or lattices with the same theta series as their dual, was introduced #OTHEREFR , #OTHEREFR .

On The Secrecy Gain Of Lattices From Codes Over The Integers Modulo 4

Abstract: Recently, a design criterion depending on a lattice's volume and theta series, called the secrecy gain, was proposed to quantify the secrecy-goodness of the applied lattice code for the Gaussian wiretap channel. To address the secrecy gain of Construction A 4 lattices from formally self-dual Z 4-linear codes, i.e., codes for which the symmetrized weight enumerator (swe) coincides with the swe of its dual, we present new constructions of Z 4-linear codes which are formally self-dual with respect to the swe. For even lengths, formally self-dual Z 4-linear codes are constructed from nested binary codes and double circulant matrices. For odd lengths, a novel construction called odd extension from double circulant codes is proposed. Moreover, the concepts of Type I/II formally self-dual codes/unimodular lattices are introduced. Next, we derive the theta series of the formally unimodular lattices obtained by Construction A 4 from formally self-dual Z 4-linear codes and describe a universal approach to determine their secrecy gains. The secrecy gain of Construction A 4 formally unimodular lattices obtained from Type I formally self-dual Z 4-linear codes is investigated, both for even and odd dimensions. Numerical evidence shows that for some parameters, Construction A 4 lattices can achieve a higher secrecy gain than the best-known formally unimodular lattices from the literature. Index Terms Lattices, Codes over Z 4 , Construction C, Construction A 4 , secrecy gain, Gaussian wiretap channel.

Secrecy Gain: A Wiretap Lattice Code Design

Abstract: We propose the notion of secrecy gain as a code design criterion for wiretap lattice codes to be used over an additive white Gaussian noise channel. Our analysis relies on the error probabilites of both the legitimate user and the eavesdropper. We focus on geometrical properties of lattices, described by their theta series, to characterize good wiretap codes.

so-called unimodular lattices
Secrecy gains