The results also agree with #REFR , i.e., they indicate that DSCFω outperforms standard SCF.

The decoders with and without the SRM are depicted in dashed and solid lines, respectively.

The SCF decoder is in black, DSCF-2 is in light-blue and DSCF-3 is in light-green with unique markers.

3 , it can be seen that the SRM does not alter the error-correction performance.

This is expected and in line with the definition of the mechanism described in Section III-A.

The DSCF-3 decoder offers the best performance, thus the motivation to reduce the executiontime characteristics of that algorithm. Fig.

4 show the average execution time of SCF and DSCF-ω decoders for P(1024, 128) polar code.

The decoders with and without the SRM are in dashed and solid lines accordingly. The latency of SC decoding is provided for reference.

From the figures, we observe that using the SRM provides greater gain to the DSCF-2 and DSCF-3 decoders throughout the FER range.

We explain this by higher number of additional trials performed by the multi bit-flipping decoders in average.

Successive-Cancellation Flip Decoding Of Polar Codes With A Simplified Restart Mechanism

Abstract: Polar codes are a class of error-correcting codes that provably achieve the capacity of practical channels. The successive-cancellation flip (SCF) decoder is a low-complexity decoder that was proposed to improve the performance of the successive-cancellation (SC) decoder as an alternative to the highcomplexity successive-cancellation list (SCL) decoder. The SCF decoder improves the error-correction performance of the SC decoder, but the variable execution time and the high worst-case execution time pose a challenge for the realization of receivers with fixed-time algorithms. The dynamic SCF (DSCF) variation of the SCF decoder further improves the error-correction performance but the challenge of decoding delay remains. In this work, we propose a simplified restart mechanism (SRM) that reduces the execution time of SCF and DSCF decoders through conditional restart of the additional trials from the second half of the codeword. We show that the proposed mechanism is able to improve the execution time characteristics of SCF and DSCF decoders while providing identical error-correction performance. For a DSCF decoder that can flip up to 3 simultaneous bits per decoding trial, the average execution time, the average additional execution time and the execution-time variance are reduced by approximately 31%, 37% and 57%, respectively. For this setup, the mechanism requires approximately 3.9% additional memory.

Dynamic-Scflip Decoding Of Polar Codes

Abstract: This paper proposes a generalization of the recently introduced Successive Cancellation Flip (SCFlip) decoding of polar codes, characterized by a number of extra decoding attempts, where one or several positions are flipped from the standard Successive Cancellation (SC) decoding. To make such an approach effective, we first introduce the concept of higher-order bit-flips, and propose a new metric to determine the bit-flips that are more likely to correct the trajectory of the SC decoding. We then propose a generalized SCFlip decoding algorithm, referred to as Dynamic-SCFlip (D-SCFlip), which dynamically builds a list of candidate bit-flips, while guaranteeing that extra decoding attempts are performed by decreasing probability of success. Simulation results show that D-SCFlip is an effective alternative to SC-List decoding of polar codes, by providing very good error correcting performance, with an average computation complexity close to the one of the SC decoder.


This is consistent with specaugment #REFR , which converts over-fitting problems into under-fitting problems, and improves model performance through longer training time.

The attention in the ST model tends to be smooth between input frames.

However, after applying DropDim, both types of attention become more concentrated, indicating that DropDim can not only help the model to remove interference but also make the model better align with the target.

It can be seen from the network training curve in Fig.

6 that when DropDim is used, the network can underfit the training loss, which is in sharp contrast with the usual situation where networks tend to overfit to the training data.

This also reveals to us that the combination of Dropdim and specaugment is not suitable.

Because although this combination can improve performance, the training time will be further increased.

Dropdim is more suitable to be combined with Speed Perturbs, because the latter improves the model performance by increasing the amount of data, which is different from Dropdim and specaugment.

Dropdim: A Regularization Method For Transformer Networks

Abstract: We introduce DropDim, a structured dropout method designed for regularizing the self-attention mechanism, which is a key component of the transformer. In contrast to the general dropout method, which randomly drops neurons, DropDim drops part of the embedding dimensions. In this way, the semantic information can be completely discarded. Thus, the excessive co-adapting between different embedding dimensions can be broken, and the self-attention is forced to encode meaningful features with a certain number of embedding dimensions erased. Experiments on a wide range of tasks executed on the MUST-C English-Germany dataset show that DropDim can effectively improve model performance, reduce over-fitting, and show complementary effects with other regularization methods. When combined with label smoothing, the WER can be reduced from 19.1% to 15.1% on the ASR task, and the BLEU value can be increased from 26.90 to 28.38 on the MT task. On the ST task, the model can reach a BLEU score of 22.99, an increase by 1.86 BLEU points compared to the strong baseline.

Specaugment: A Simple Data Augmentation Method For Automatic Speech Recognition

Abstract: We present SpecAugment, a simple data augmentation method for speech recognition. SpecAugment is applied directly to the feature inputs of a neural network (i.e., filter bank coefficients). The augmentation policy consists of warping the features, masking blocks of frequency channels, and masking blocks of time steps. We apply SpecAugment on Listen, Attend and Spell networks for end-to-end speech recognition tasks. We achieve state-of-the-art performance on the LibriSpeech 960h and Swichboard 300h tasks, outperforming all prior work. On LibriSpeech, we achieve 6.8% WER on test-other without the use of a language model, and 5.8% WER with shallow fusion with a language model. This compares to the previous state-of-the-art hybrid system of 7.5% WER. For Switchboard, we achieve 7.2%/14.6% on the Switchboard/CallHome portion of the Hub5'00 test set without the use of a language model, and 6.8%/14.1% with shallow fusion, which compares to the previous state-of-the-art hybrid system at 8.3%/17.3% WER.


To construct the set of models, we follow the construction similar to that in the proof of Theorem 1 of #REFR .

where P a (X) denotes the distribution over X under a model parametrized by β * a . Then the minimax risk is lower bounded as

inf β sup β * ∈B E[ β − β * 2 2 ] ≥ δ 2 2 1 − ξ + log 2 log M ,

whereβ is the output of any statistical estimator.

Intuitively, the theorem says that if we can construct a set of models where every pair of model is sufficiently different (measured in 2 2 distance) yet they parametrize 'similar' distributions (measured in KL divergence), then any statistical estimator will fail to identify the correct model and thus suffer from a minimum estimation error.

We first restate a coding theoretic due to #OTHEREFR . #OTHEREFR

(2015) ) For any α ∈ (0, 1 4 ), there exists a set of M (α) = Θ(exp (m)) binary vectors {z 1 , . . . , z M (α)

Optimal And Private Learning From Human Response Data

Abstract: Item response theory (IRT) is the study of how people make probabilistic decisions, with diverse applications in education testing, recommendation systems, among others. The Rasch model of binary response data, one of the most fundamental models in IRT, remains an active area of research with important practical significance. Recently, Nguyen and Zhang (2022) proposed a new spectral estimation algorithm that is efficient and accurate. In this work, we extend their results in two important ways. Firstly, we obtain a refined entrywise error bound for the spectral algorithm, complementing the 'average error' 2 bound in their work. Notably, under mild sampling conditions, the spectral algorithm achieves the minimax optimal error bound (modulo a log factor). Building on the refined analysis, we also show that the spectral algorithm enjoys optimal sample complexity for top-K recovery (e.g., identifying the best K items from approval/disapproval response data), explaining the empirical findings in the previous work.

Estimation From Pairwise Comparisons: Sharp Minimax Bounds With Topology Dependence

Abstract: Data in the form of pairwise comparisons arises in many domains, including preference elicitation, sporting competitions, and peer grading among others. We consider parametric ordinal models for such pairwise comparison data involving a latent vector $w^* \in \mathbb{R}^d$ that represents the"qualities"of the $d$ items being compared; this class of models includes the two most widely used parametric models--the Bradley-Terry-Luce (BTL) and the Thurstone models. Working within a standard minimax framework, we provide tight upper and lower bounds on the optimal error in estimating the quality score vector $w^*$ under this class of models. The bounds depend on the topology of the comparison graph induced by the subset of pairs being compared via its Laplacian spectrum. Thus, in settings where the subset of pairs may be chosen, our results provide principled guidelines for making this choice. Finally, we compare these error rates to those under cardinal measurement models and show that the error rates in the ordinal and cardinal settings have identical scalings apart from constant pre-factors.


In particular, we are interested in the second version of the StyleGAN, the StyleGAN2 #REFR , which is a revisiting of the architecture of the StyleGAN synthesis network.

StyleGANs are a re-design of the GANs generator architecture, which aim is to control the image synthesis process #OTHEREFR .

The typical architecture of a StyleGAN is shown in Appendix A.1 in Figure 6 , together with the technical details.

Figure 7 in Appendix A.1 exemplifies the changes made to the original architecture up to the final StyleGAN2 network. Further information is provided in Appendix A.1.

The StyleGAN2 architecture makes it possible to control image synthesis via scale-specific style modifications.

In particular, this approach attains layer-wise decomposition of visual properties, allowing StyleGAN to control global aspects of the picture, such as pose, lighting conditions or colour schemes, coherently over the entire image.

However, while this model successfully disentangles global properties, it is more limited in its ability to perform spatial decomposition.

It provides no direct means to control the style of localised regions within the generated image.

Investigating Gansformer: A Replication Study Of A State-Of-The-Art Image Generation Model

Abstract: The field of image generation through generative modelling is abundantly discussed nowadays. It can be used for various applications, such as up-scaling existing images, creating non-existing objects, such as interior design scenes, products or even human faces, and achieving transferlearning processes. In this context, Generative Adversarial Networks (GANs) are a class of widely studied machine learning frameworks first appearing in the paper "Generative adversarial nets" by Goodfellow et al. [1] that achieve the goal above. In our work, we reproduce and evaluate a novel variation of the original GAN network, the GANformer, proposed in "Generative Adversarial Transformers" by Hudson and Zitnick [2]. This project aimed to recreate the methods presented in this paper to reproduce the original results and comment on the authors' claims. Due to resources and time limitations, we had to constrain the network's training times, dataset types, and sizes. Our research successfully recreated both variations of the proposed GANformer model and found differences between the authors' and our results. Moreover, discrepancies between the publication methodology and the one implemented, made available in the code, allowed us to study two undisclosed variations of the presented procedures.

Analyzing And Improving The Image Quality Of Stylegan

Abstract: The style-based GAN architecture (StyleGAN) yields state-of-the-art results in data-driven unconditional generative image modeling. We expose and analyze several of its characteristic artifacts, and propose changes in both model architecture and training methods to address them. In particular, we redesign the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images. In addition to improving image quality, this path length regularizer yields the additional benefit that the generator becomes significantly easier to invert. This makes it possible to reliably attribute a generated image to a particular network. We furthermore visualize how well the generator utilizes its output resolution, and identify a capacity problem, motivating us to train larger models for additional quality improvements. Overall, our improved model redefines the state of the art in unconditional image modeling, both in terms of existing distribution quality metrics as well as perceived image quality.


To acquire such knowledge, we propose to automatically construct a visual relation knowledge graph from tremendous open domain image captions #REFR .

Rather than representing each relationship statically with few-shot training examples #OTHEREFR or word vectors #OTHEREFR , we propose to generate the knowledge-enhanced representation of relationships dynamically with a pre-trained language model, based on the classes of Subject and Object and a simple prompt template.

As a result, the semantic diversity of each relationship can be largely covered with the help of the embedded knowledge in the pre-trained language model. Visual Relation Knowledge.

Though convey a large amount of useful information, knowledge from text suffer from the reporting bias problem #OTHEREFR and misses common relation facts such as letter-painted on-sign and sign-hanging from-pole.

In particular, the wellknown text-based knowledge graph ConceptNet #OTHEREFR covers only 17% of relation facts in visual relation learning datasets #OTHEREFR .

However, visual relation knowledge which explicitly represents the common relationships between objects can be leveraged to bridge the information of entities and relationships from the textual knowledge.

In detail, we adopt a rule-based parser #OTHEREFR to extract relation triplets from captions and then resort to a distributed representation of visual relation knowledge graph with a novel knowledge encoding module.

Our contributions can be summarized as three-fold:

• To the best of our knowledge, this is the first work to combine textual knowledge and visual relation knowledge in VRD.

We devise a Knowledge-augmented Few-shot VRD (KFV) framework, which incorporates both kinds of knowledge to improve the generalization ability of few-shot VRD.

• We present a novel prompt-based method to acquire textual knowledge from pre-trained language models and an efficient way to learn distributed visual relation knowledge graph representation from image captions to augment few-shot VRD.

Knowledge-Augmented Few-Shot Visual Relation Detection

Abstract: Visual Relation Detection (VRD) aims to detect relationships between objects for image understanding. Most existing VRD methods rely on thousands of training samples of each relationship to achieve satisfactory performance. Some recent papers tackle this problem by few-shot learning with elaborately designed pipelines and pre-trained word vectors. However, the performance of existing few-shot VRD models is severely hampered by the poor generalization capability, as they struggle to handle the vast semantic diversity of visual relationships. Nonetheless, humans have the ability to learn new relationships with just few examples based on their knowledge. Inspired by this, we devise a knowledge-augmented, few-shot VRD framework leveraging both textual knowledge and visual relation knowledge to improve the generalization ability of few-shot VRD. The textual knowledge and visual relation knowledge are acquired from a pre-trained language model and an automatically constructed visual relation knowledge graph, respectively. We extensively validate the effectiveness of our framework. Experiments conducted on three benchmarks from the commonly used Visual Genome dataset show that our performance surpasses existing state-of-the-art models with a large improvement.

Microsoft Coco: Common Objects In Context

Abstract: We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.


According to #REFR , the likelihood function p(y p |x, ∆φ) = CN (y p ; f p x, σ 2 e ), where f p is the p-th row of measurement matrix F .

The expectation is taken over the marginal posterior p(x n |y, ∆φ) given measurements y and off-grid offsets ∆φ.

To exploit the inherent statistical structure of the HAP-UAV channel in terms of channel support and value vectors, the marginal posterior distribution can be expressed by Bayes' rule as

where ∝ indicates proportionality up to a constant scale factor, x −m means a vector x excluding the element

The optimal off-grid offsets ∆φ can be obtained by a maximum likelihood (ML) approach

After obtaining the estimated offsets ∆φ * , we can then calculate the corresponding marginal posterior distribution #OTHEREFR .

It's the production of many distribution functions, each of which is determined by only a subset of variables.

Such decomposition structures can be calculated by a graphical model, i.e., a factor graph, which can be an undirected bipartite graph used to represent the connection between random variables (called variable nodes) and related probability density functions (pdfs) (called factor nodes).

Energy-Efficient Urllc Service Provision Via A Near-Space Information Network

Abstract: The integration of a near-space information network (NSIN) with the reconfigurable intelligent surface (RIS) is envisioned to significantly enhance the communication performance of future wireless communication systems by proactively altering wireless channels. This paper investigates the problem of deploying a RIS-integrated NSIN to provide energy-efficient, ultra-reliable and low-latency communications (URLLC) services. We mathematically formulate this problem as a resource optimization problem, aiming to maximize the effective throughput and minimize the system power consumption, subject to URLLC and physical resource constraints. The formulated problem is challenging in terms of accurate channel estimation, RIS phase alignment, theoretical analysis, and effective solution. We propose a joint resource allocation algorithm to handle these challenges. In this algorithm, we develop an accurate channel estimation approach by exploring message passing and optimize phase shifts of RIS reflecting elements to further increase the channel gain. Besides, we derive an analysis-friend expression of decoding error probability and decompose the problem into two-layered optimization problems by analyzing the monotonicity, which makes the formulated problem analytically tractable. Extensive simulations have been conducted to verify the performance of the proposed algorithm. Simulation results show that the proposed algorithm can achieve outstanding channel estimation performance and is more energy-efficient than diverse benchmark algorithms.

Quasi-Static Multiple-Antenna Fading Channels At Finite Blocklength

Abstract: This paper investigates the maximal achievable rate for a given blocklength and error probability over quasi-static multiple-input multiple-output (MIMO) fading channels, with and without channel state information (CSI) at the transmitter and/or the receiver. The principal finding is that outage capacity, despite being an asymptotic quantity, is a sharp proxy for the finite-blocklength fundamental limits of slow-fading channels. Specifically, the channel dispersion is shown to be zero regardless of whether the fading realizations are available at both transmitter and receiver, at only one of them, or at neither of them. These results follow from analytically tractable converse and achievability bounds. Numerical evaluation of these bounds verifies that zero dispersion may indeed imply fast convergence to the outage capacity as the blocklength increases. In the example of a particular $1 \times 2$ single-input multiple-output (SIMO) Rician fading channel, the blocklength required to achieve $90\%$ of capacity is about an order of magnitude smaller compared to the blocklength required for an AWGN channel with the same capacity. For this specific scenario, the coding/decoding schemes adopted in the LTE-Advanced standard are benchmarked against the finite-blocklength achievability and converse bounds.

likelihood function
measurement matrix F


We compare against two NN models: a standard (non-Hamiltonian) Neural ODE (NODE) (Chen et al., 2018) with a neural time derivative f θ (x), and a HNN, based on the Unstructured SymODEN model of #REFR with a neural Hamiltonian H θ (x).

Bottom row: full cycle estimates, where the baseline is able to fit the vector field, but is oblivious to the energy.

challenging, as such good initialisations are very important, particularly for the variational parameters of the Hamiltonian GP.

We can obtain a good initialisation by observing that the derivative function f and H are jointly a GP.

Using the approximate numerical derivatives of the trajectory data, we can form some data for f , which can then be conditioned on, using equations (10) to (12), to obtain an estimate for the mean function of the Hamiltonian.

We use this estimate as the initial value for the variational mean m. See appendix B.2 for details. NN baselines.

Both methods perform poorly when trained on raw trajectories due to vanishing/exploding gradients.

To allow for a fair comparison, we split the trajectories into smaller sub-trajectories by running a sliding window of fixed length over the data, and form these into batches of training data. GP baselines.

In order to evaluate the effect of our Hamiltonian prior, we compare against the non-Hamiltonian shooting GPODE baseline of Hegde et al. (2022) .

We use the same hyperparameter settings and initialisations as in the HGP model, where applicable.

To initialise the inducing variable distribution means, we follow the same procedure described by Hegde et al. (2022) .

Learning Energy Conserving Dynamics Efficiently With Hamiltonian Gaussian Processes

Abstract: Hamiltonian mechanics is one of the cornerstones of natural sciences. Recently there has been significant interest in learning Hamiltonian systems in a free-form way directly from trajectory data. Previous methods have tackled the problem of learning from many short, low-noise trajectories, but learning from a small number of long, noisy trajectories, whilst accounting for model uncertainty has not been addressed. In this work, we present a Gaussian process model for Hamiltonian systems with efficient decoupled parameterisation, and introduce an energy-conserving shooting method that allows robust inference from both short and long trajectories. We demonstrate the method's success in learning Hamiltonian systems in various data settings.

Symplectic Ode-Net: Learning Hamiltonian Dynamics With Control

Abstract: In this paper, we introduce Symplectic ODE-Net (SymODEN), a deep learning framework which can infer the dynamics of a physical system from observed state trajectories. To achieve better generalization with fewer training samples, SymODEN incorporates appropriate inductive bias by designing the associated computation graph in a physics-informed manner. In particular, we enforce Hamiltonian dynamics with control to learn the underlying dynamics in a transparent way which can then be leveraged to draw insight about relevant physical aspects of the system, such as mass and potential energy. In addition, we propose a parametrization which can enforce this Hamiltonian formalism even when the generalized coordinate data is embedded in a high-dimensional space or we can only access velocity data instead of generalized momentum. This framework, by offering interpretable, physically-consistent models for physical systems, opens up new possibilities for synthesizing model-based control strategies.

neural Hamiltonian H
standard (non-Hamiltonian) Neural


Therefore, GAP is better suited than GMP to extract and retain relevant features. This finding was also observed in #REFR .

In our proposed learning model, except for the last regular GAP layer, three additional GAP layers were used in each sub-network.

The place of each GAP/GMP layer governs the level of features to be considered.

To show the contribution of each GAP/GMP layer, we performed an ablation study.

As can be seen in Table 3 , the performance achieved by exploiting GAP layers at various levels is almost consistently better than that of GMP, whatever the combination of the levels.

This is mostly caused by the fact that GMP only maintains one element with the max value in each feature map; as a result, it discards much texture-related information.

In addition, different GAP and GMP combinations were also evaluated, where GAP and GMP units were mixed. However, no performance improvement was observed.

Figure 4 : Comparison of the attention maps generated by the final convolutional layer of each model.

A Confident Labelling Strategy Based On Deep Learning For Improving Early Detection Of Knee Osteoarthritis

Abstract: Knee OsteoArthritis (KOA) is a prevalent musculoskeletal disorder that causes decreased mobility in seniors. The diagnosis provided by physicians is subjective, however, as it relies on personal experience and the semi-quantitative Kellgren-Lawrence (KL) scoring system. KOA has been successfully diagnosed by Computer-Aided Diagnostic (CAD) systems that use deep learning techniques like Convolutional Neural Networks (CNN). In this paper, we propose a novel Siamese-based network, and we introduce a new hybrid loss strategy for the early detection of KOA. The model extends the classical Siamese network by integrating a collection of Global Average Pooling (GAP) layers for feature extraction at each level. Then, to improve the classification performance, a novel training strategy that partitions each training batch into low-, medium-and high-confidence subsets, and a specific hybrid loss function are used for each new label attributed to each sample. The final loss function is then derived by combining the latter loss functions with optimized weights. Our test results demonstrate that our proposed approach significantly improves the detection performance.

Learning Deep Features For Discriminative Localization

Abstract: In this work, we revisit the global average pooling layer proposed in [13], and shed light on how it explicitly enables the convolutional neural network (CNN) to have remarkable localization ability despite being trained on imagelevel labels. While this technique was previously proposed as a means for regularizing training, we find that it actually builds a generic localizable deep representation that exposes the implicit attention of CNNs on an image. Despite the apparent simplicity of global average pooling, we are able to achieve 37.1% top-5 error for object localization on ILSVRC 2014 without training on any bounding box annotation. We demonstrate in a variety of experiments that our network is able to localize the discriminative image regions despite just being trained for solving classification task1.


This is in contrast to the uncapacitated version of the problem, where one can get (1+ǫ)-approximation in 2 O((k log k)/ǫ 2 ) •n O #REFR time as mentioned before.

We also complement our approximability results by hardness bounds.

The NP-hardness of Capacitated Sum of Radii trivially follows from the NP-hardness of Sum of Radii. We strengthen this bound by showing the following result. Theorem 1.5.

Capacitated Sum of Radii with uniform capacities cannot be solved in f (k)n o(k) time for any computable function f , unless ETH is false. Moreover, it does not admit any FPTAS, unless P=NP.

We also show an inapproximability bound in the Euclidean case even with uniform capacities.

Although the above bound does not eradicate the possibility of obtaining a (1+ǫ)-approximation in the Euclidean case, it shows that to obtain such an approximation, even when k = 2, one needs n f (ǫ) time for some non-constant function f that depends on ǫ.

As by products of our techniques we have also obtained improved bi-criteria approximations for the uniform-capacitated version of the problem where we are allowed to use (1+ǫ)U capacity. Theorem 1.7.

There is a randomized algorithm for Capacitated Sum of Radii with uniform capacities that runs in time 2 O(k log(k/ǫ)) • n O #OTHEREFR and returns a solution with constant probability, such that each ball in the solution uses at most (1 + ǫ)U capacity and the cost of the solution is at most (2 + ǫ) • OPT, where OPT is the cost of any optimal solution in which the balls use at most U capacity.

The above theorem improves the approximation factor in Theorem 1.2. In the Euclidean case, we obtain a similar algorithm. Theorem 1.8.

There is a randomized algorithm for the Euclidean version of Capacitated Sum of Radii with uniform capacities that runs in time 2 O((k/ǫ 2 ) log k) •dn #OTHEREFR and returns a solution with constant probability, such that each ball in the solution uses at most (1 + ǫ)U capacity and the cost of the solution is at most (1 + ǫ) • OPT, where OPT is the cost of any optimal solution in which the balls use at most U capacity.

Note that, by Theorem 1.6, a result as in the above theorem is not possible if we are not allowed to violate the capacity.

Fpt Constant-Approximations For Capacitated Clustering To Minimize The Sum Of Cluster Radii *

Abstract: Clustering with capacity constraints is a fundamental problem that attracted significant attention throughout the years. In this paper, we give the first FPT constant-factor approximation algorithm for the problem of clustering points in a general metric into k clusters to minimize the sum of cluster radii, subject to non-uniform hard capacity constraints (Capacitated Sum of Radii). In particular, we give a (15 + ǫ)-approximation algorithm that runs in 2 O(k 2 log k) • n 3 time. When capacities are uniform, we obtain the following improved approximation bounds. • A (4 + ǫ)-approximation with running time 2 O(k log(k/ǫ)) n 3 , which significantly improves over the FPT 28-approximation of Inamdar and Varadarajan [ESA 2020]. • A (2 + ǫ)-approximation with running time 2 O(k/ǫ 2 •log(k/ǫ)) dn 3 and a (1+ǫ)-approximation with running time 2 O(kd log((k/ǫ))) n 3 in the Euclidean space. Here d is the dimension.

Constant Factor Fpt Approximation For Capacitated K-Median

Abstract: Capacitated k-median is one of the few outstanding optimization problems for which the existence of a polynomial time constant factor approximation algorithm remains an open problem. In a series of recent papers algorithms producing solutions violating either the number of facilities or the capacity by a multiplicative factor were obtained. However, to produce solutions without violations appears to be hard and potentially requires different algorithmic techniques. Notably, if parameterized by the number of facilities $k$, the problem is also $W[2]$ hard, making the existence of an exact FPT algorithm unlikely. In this work we provide an FPT-time constant factor approximation algorithm preserving both cardinality and capacity of the facilities. The algorithm runs in time $2^{\mathcal{O}(k\log k)}n^{\mathcal{O}(1)}$ and achieves an approximation ratio of $7+\varepsilon$.


It ensures that the probes are not incentivized to learn extra information over and above what the model has learned, which can become a problem when training probes with ground truth labels #REFR .

We train the translators to minimize KL between the tuned lens logits and the final layer logits:

where f > (h ) refers to the rest of the transformer after layer .

This can be viewed as a distillation loss, using the final layer distribution as a soft label #OTHEREFR .

When readily available, we train translators on a slice of the validation set used during pretraining, and use a separate slice for evaluation.

Since BLOOM and GPT-2 do not have publicly available validation sets, we use the Pile validation set (Gao et al., 2020; .

The OPT validation set is also not publicly available, but a member of the OPT team helped us train a tuned lens on the OPT validation set.

Documents are concatenated and split into uniform chunks of length 2048.

Eliciting Latent Predictions From Transformers With The Tuned Lens

Abstract: We analyze transformers from the perspective of iterative inference, seeking to understand how model predictions are refined layer by layer. To do so, we train an affine probe for each block in a frozen pretrained model, making it possible to decode every hidden state into a distribution over the vocabulary. Our method, the tuned lens, is a refinement of the earlier "logit lens" technique, which yielded useful insights but is often brittle. We test our method on various autoregressive language models with up to 20B parameters, showing it to be more predictive, reliable and unbiased than the logit lens. With causal experiments, we show the tuned lens uses similar features to the model itself. We also find the trajectory of latent predictions can be used to detect malicious inputs with high accuracy. All code needed to reproduce our results can be found at https://github.com/ AlignmentResearch/tuned-lens.

Designing And Interpreting Probes With Control Tasks

Abstract: Probes, supervised models trained to predict properties (like parts-of-speech) from representations (like ELMo), have achieved high accuracy on a range of linguistic tasks. But does this mean that the representations encode linguistic structure or just that the probe has learned the linguistic task? In this paper, we propose control tasks, which associate word types with random outputs, to complement linguistic tasks. By construction, these tasks can only be learned by the probe itself. So a good probe, (one that reflects the representation), should be selective, achieving high linguistic task accuracy and low control task accuracy. The selectivity of a probe puts linguistic task accuracy in context with the probe's capacity to memorize from word types. We construct control tasks for English part-of-speech tagging and dependency edge prediction, and show that popular probes on ELMo representations are not selective. We also find that dropout, commonly used to control probe complexity, is ineffective for improving selectivity of MLPs, but that other forms of regularization are effective. Finally, we find that while probes on the first layer of ELMo yield slightly better part-of-speech tagging accuracy than the second, probes on the second layer are substantially more selective, which raises the question of which layer better represents parts-of-speech.


Moreover, the improvement over the supervised learning with coarse labels (SupCE) on good split (13.98%) is much higher than the bad split (4.41%), which shows the substantial potential of MaskCon in dealing with more realistic coarse-labelled datasets #REFR .

Specifically, the classes 'airplane', 'truck', 'automobile', and 'ship' are designated as Class A, while 'horse', 'dog', 'bird', and 'cat' are assigned to Class B.

Note that Class A comprises non-organic objects, while Class B comprises animals.

Moreover, a bad split of the aforementioned dataset is defined, wherein 'airplane', 'automobile', 'bird', and 'cat' are Class A, and 'horse', 'dog', 'ship', and 'truck' are Class B.

12, our method achieves the best performance in both splits.

3 we visualize the learned features of all test samples in the good split.

Clearly, the learned representations with supervised contrastive learning (SupCon) and self-supervised contrastive learning (SelfCon) tend to under-cluster and over-cluster the samples, respectively.

By contrast, our method gets more compact and clear clusters, in line with the results when trained with fine labels (SupFINE).

Maskcon: Masked Contrastive Learning For Coarse-Labelled Dataset

Abstract: Deep learning has achieved great success in recent years with the aid of advanced neural network structures and large-scale human-annotated datasets. However, it is often costly and difficult to accurately and efficiently annotate large-scale datasets, especially for some specialized domains where fine-grained labels are required. In this setting, coarse labels are much easier to acquire as they do not require expert knowledge. In this work, we propose a contrastive learning method, called masked contrastive learning (MaskCon) to address the under-explored problem setting, where we learn with a coarse-labelled dataset in order to address a finer labelling problem. More specifically, within the contrastive learning framework, for each sample our method generates soft-labels with the aid of coarse labels against other samples and another augmented view of the sample in question. By contrast to self-supervised contrastive learning where only the sample's augmentations are considered hard positives, and in supervised contrastive learning where only samples with the same coarse labels are considered hard positives, we propose soft labels based on sample distances, that are masked by the coarse labels. This allows us to utilize both inter-sample relations and coarse labels. We demonstrate that our method can obtain as special cases many existing state-of-the-art works and that it provides tighter bounds on the generalization error. Experimentally, our method achieves significant improvement over the current state-of-the-art in various datasets, including CIFAR10, CIFAR100, ImageNet-1K, Standford Online Products and Stanford Cars196 datasets. Code and annotations are available at https://github.com/ MrChenFeng/MaskCon_CVPR2023.

Mixmatch: A Holistic Approach To Semi-Supervised Learning

Abstract: Semi-supervised learning has proven to be a powerful paradigm for leveraging unlabeled data to mitigate the reliance on large labeled datasets. In this work, we unify the current dominant approaches for semi-supervised learning to produce a new algorithm, MixMatch, that works by guessing low-entropy labels for data-augmented unlabeled examples and mixing labeled and unlabeled data using MixUp. We show that MixMatch obtains state-of-the-art results by a large margin across many datasets and labeled data amounts. For example, on CIFAR-10 with 250 labels, we reduce error rate by a factor of 4 (from 38% to 11%) and by a factor of 2 on STL-10. We also demonstrate how MixMatch can help achieve a dramatically better accuracy-privacy trade-off for differential privacy. Finally, we perform an ablation study to tease apart which components of MixMatch are most important for its success.


Following #REFR , we adopt the two metrics of part accuracy (Part Acc.) and shape chamfer distance (Shape CD) to evaluate the assembled shape structure.

Part accuracy threshold τ p is chosen to be 0.1.

We propose two metrics for evaluating the second objective of the joint-centric part assembly task, joint accuracy (Joint Acc.) and joint chamfer distance (Joint CD). Specifically, Joint Acc.

Category-Level Multi-Part Multi-Joint 3D Shape Assembly

Abstract: Shape assembly composes complex shapes geometries by arranging simple part geometries and has wide applications in autonomous robotic assembly and CAD modeling. Existing works focus on geometry reasoning and neglect the actual physical assembly process of matching and fitting joints, which are the contact surfaces connecting different parts. In this paper, we consider contacting joints for the task of multi-part assembly. A successful joint-optimized assembly needs to satisfy the bilateral objectives of shape structure and joint alignment. We propose a hierarchical graph learning approach composed of two levels of graph representation learning. The part graph takes part geometries as input to build the desired shape structure. The joint-level graph uses part joints information and focuses on matching and aligning joints. The two kinds of information are combined to achieve the bilateral objectives. Extensive experiments demonstrate that our method outperforms previous methods, achieving better shape structure and higher joint alignment accuracy.

Learning 3D Part Assembly From A Single Image

Abstract: Autonomous assembly is a crucial capability for robots in many applications. For this task, several problems such as obstacle avoidance, motion planning, and actuator control have been extensively studied in robotics. However, when it comes to task specification, the space of possibilities remains underexplored. Towards this end, we introduce a novel problem, single-image-guided 3D part assembly, along with a learningbased solution. We study this problem in the setting of furniture assembly from a given complete set of parts and a single image depicting the entire assembled object. Multiple challenges exist in this setting, including handling ambiguity among parts (e.g., slats in a chair back and leg stretchers) and 3D pose prediction for parts and part subassemblies, whether visible or occluded. We address these issues by proposing a two-module pipeline that leverages strong 2D-3D correspondences and assembly-oriented graph message-passing to infer part relationships. In experiments with a PartNet-based synthetic benchmark, we demonstrate the effectiveness of our framework as compared with three baseline approaches.


This appendix is a review of VC theory, except for novel results presented in Section A.2.3 for the case of unbounded loss functions, where we obtain new bounds for relative type I estimation error by extending the results in #REFR .

and the result is then direct from the proof of Theorem 4.3. But (7.19) is clearly true since

L(h M ) − L(h ) L(h M ) > L(M ) =⇒ L(h M ) − L(h ) > L(h M ) L(M ) ≥ .

Hence, we present results which support those in this paper and outline the main ideas of VC theory, even though are not the tightest available bounds.

We omit the proofs, and note that refined versions of the results presented here may be found at one or more of the references.

We start defining the shatter coefficient and VC dimension of a hypotheses space under loss function . Remark A.3.

If there is no confusion about which loss function we are referring, or when it is not of importance to our argument, we omit and denote the shatter coefficient and VC dimension simply by S(H, N ) and

We note that if the hypotheses in H are binary valued functions and is the simple loss function ((x, y), h) = 1{h(x) = y}, then H = G H, , and its N -th shatter coefficient is actually the maximum number of dichotomies that can be generated by the functions in H with N points.

A.1. Generalized Glivenko-Cantelli Problems.

Distribution-Free Deviation Bounds Of Learning Via Model Selection With Cross-Validation Risk Estimation

Abstract: Cross-validation techniques for risk estimation and model selection are widely used in statistics and machine learning. However, the understanding of the theoretical properties of learning via model selection with cross-validation risk estimation is quite low in face of its widespread use. In this context, this paper presents learning via model selection with cross-validation risk estimation as a general systematic learning framework within classical statistical learning theory and establishes distribution-free deviation bounds in terms of VC dimension, giving detailed proofs of the results and considering both bounded and unbounded loss functions. We also deduce conditions under which the deviation bounds of learning via model selection are tighter than that of learning via empirical risk minimization in the whole hypotheses space, supporting the better performance of model selection frameworks observed empirically in some instances.

Relative Deviation Learning Bounds And Generalization With Unbounded Loss Functions

Abstract: We present an extensive analysis of relative deviation bounds, including detailed proofs of two-sided inequalities and their implications. We also give detailed proofs of two-sided generalization bounds that hold in the general case of unbounded loss functions, under the assumption that a moment of the loss is bounded. We then illustrate how to apply these results in a sample application: the analysis of importance weighting.


As a result, we add Smooth L1 loss #REFR in addition to the initial IoU loss.

where IoU denotes Intersection over Union between prediction results and ground truth. However, IoU loss is not perfect.

The value of IoU would be zero if is no overlap between the bounding box and the ground truth.

The gradient of the loss function is now zero, making it impossible to optimize the parameters.

IoU loss can not satisfy our requirement for regression loss in this situation.

Smooth L1 loss can evaluate the regression offset of the output by computing the distance between each feature map point and four edges in the ground truth.

As a result, we merge the Smooth L1 loss and IoU loss to produce the regression branch's loss function. The specific formula is as follows:

where d stands for the output regression offset andd represents the target offset. L denotes Smooth L1 loss.

As we mention above, the optimization between the two branches in the past approach was independent.

Siamthn: Siamese Target Highlight Network For Visual Tracking

Abstract: Siamese network based trackers develop rapidly in the field of visual object tracking in recent years. The majority of siamese network based trackers now in use treat each channel in the feature maps generated by the backbone network equally, making the similarity response map sensitive to background influence and hence challenging to focus on the target region. Additionally, there are no structural links between the classification and regression branches in these trackers, and the two branches are optimized separately during training. Therefore, there is a misalignment between the classification and regression branches, which results in less accurate tracking results. In this paper, a Target Highlight Module is proposed to help the generated similarity response maps to be more focused on the target region. To reduce the misalignment and produce more precise tracking results, we propose a corrective loss to train the model. The two branches of the model are jointly tuned with the use of corrective loss to produce more reliable prediction results. Experiments on 5 challenging benchmark datasets reveal that the method outperforms current models in terms of performance, and runs at 38 fps, proving its effectiveness and efficiency.

Fast R-Cnn

Abstract: This paper proposes a Fast Region-based Convolutional Network method (Fast R-CNN) for object detection. Fast R-CNN builds on previous work to efficiently classify object proposals using deep convolutional networks. Compared to previous work, Fast R-CNN employs several innovations to improve training and testing speed while also increasing detection accuracy. Fast R-CNN trains the very deep VGG16 network 9x faster than R-CNN, is 213x faster at test-time, and achieves a higher mAP on PASCAL VOC 2012. Compared to SPPnet, Fast R-CNN trains VGG16 3x faster, tests 10x faster, and is more accurate. Fast R-CNN is implemented in Python and C++ (using Caffe) and is available under the open-source MIT License at https://github.com/rbgirshick/fast-rcnn.


Defense Performance With The State-Of-The-Art Model Poisoning Attack

To evaluate the defense performance, we evaluate the CosDefense on top of the state-of-the-art model poisoning attack methods, Inner Product Manipulation (IPM) #REFR attack.

In the following experiments, we set the attacker control 30% of the participants as malicious nodes, and the attack would launch at round 200.

The server randomly samples 10 clients from 100 for local model training in each communication round.

Evaluation Results: Figure 3 shows the accuracy curve over 1000 communication rounds for Krum, Clipping-Median, and CosDefense for MNIST, Fashion-MNIST, and CIFAR-10.

The detailed numerical results and ablation study related to the non-iid degree are shown in Appendix E. We have three observations from the results.

(1) As Krum and Clipping-Median use all layers of the updates, the results of CosDefense indicate that the similarity of the last layer between the collected model update and global model is more sensitive to attacks, which could effectively be used to filter out the malicious model updates.

Secure Federated Learning Against Model Poi-Soning Attacks Via Client Filtering

Abstract: Given the distributed nature, detecting and defending against the backdoor attack under federated learning (FL) systems is challenging. In this paper, we observe that the cosine similarity of the last layer's weight between the global model and each local update could be used effectively as an indicator of malicious model updates. Therefore, we propose CosDefense, a cosine-similarity-based attacker detection algorithm. Specifically, under CosDefense, the server calculates the cosine similarity score of the last layer's weight between the global model and each client update, labels malicious clients whose score is much higher than the average, and filters them out of the model aggregation in each round. Compared to existing defense schemes, CosDefense does not require any extra information besides the received model updates to operate and is compatible with client sampling. Experiment results on three real-world datasets demonstrate that CosDefense could provide robust performance under the state-of-the-art FL poisoning attack.

Fall Of Empires: Breaking Byzantine-Tolerant Sgd By Inner Product Manipulation

Abstract: Recently, new defense techniques have been developed to tolerate Byzantine failures for distributed machine learning. The Byzantine model captures workers that behave arbitrarily, including malicious and compromised workers. In this paper, we break two prevailing Byzantine-tolerant techniques. Specifically we show robust aggregation methods for synchronous SGD -- coordinate-wise median and Krum -- can be broken using new attack strategies based on inner product manipulation. We prove our results theoretically, as well as show empirical validation.


The Virtual-Taobao #REFR is a multiagent reinforcement learning system designed to improve search in the social Taobao web site; it makes use of a GAN for simulating internal distributions.

Trust information is used in #OTHEREFR to make effective recommendations.

They propose a GAN where the discriminator is an MLP model, and the generator is an LSTM model.

Finally, CF datasets are usually imbalanced due to their social data collection (e.g: more young than old people).

To address this limitation, #OTHEREFR proposes a Wasserstein GAN model in the generator, and the PacGAN concept in the discriminator #OTHEREFR , to minimize the mode collapse problem.

A platform for multi-agent RS simulation is the probabilistic-based RecSim #OTHEREFR , that generates synthetic profiles of users and items, and it uses Markov chains and recurrent neural networks.

A simple matrix factorization is used #OTHEREFR to inject topic diversification in the recommendation process.

The DataGenCars #OTHEREFR is a Java-based generator of RS synthetic data; it contains a statistical basement that provides flexibility, but it returns low accuracy compared to deep learning generative models.

Finally, the SynEvaRec framework #OTHEREFR provides synthetic RS datasets generation using the Synthetic Data Vault (SVD) library.

This library models multivariate distributions using copula functions; its CTGAN sub-library includes GAN models.

The main advantage of SynEvaRec, compared to previous frameworks, is that it can take different RS as source; its main drawbacks are the poor quality of the results in most of the cases, and the excessive time consumed to accomplish the training stage.

Creating Synthetic Datasets For Collaborative Filtering Recommender Systems Using Generative Adversarial Networks

Abstract: Research and education in machine learning needs diverse, representative, and open datasets that contain sufficient samples to handle the necessary training, validation, and testing tasks. Currently, the Recommender Systems area includes a large number of subfields in which accuracy and beyond accuracy quality measures are continuously improved. To feed this research variety, it is necessary and convenient to reinforce the existing datasets with synthetic ones. This paper proposes a Generative Adversarial Network (GAN)-based method to generate collaborative filtering datasets in a parameterized way, by selecting their preferred number of users, items, samples, and stochastic variability. This parameterization cannot be made using regular GANs. Our GAN model is fed with dense, short, and continuous embedding representations of items and users, instead of sparse, large, and discrete vectors, to make an accurate and quick learning, compared to the traditional approach based on large and sparse input vectors. The proposed architecture includes a DeepMF model to extract the dense user and item embeddings, as well as a clustering process to convert from the dense GAN generated samples to the discrete and sparse ones, necessary to create each required synthetic dataset. The results of three different source datasets show adequate distributions and expected quality values and evolutions on the generated datasets compared to the source ones. Synthetic datasets and source codes are available to researchers.

Virtual-Taobao: Virtualizing Real-World Online Retail Environment For Reinforcement Learning

Abstract: Applying reinforcement learning in physical-world tasks is extremely challenging. It is commonly infeasible to sample a large number of trials, as required by current reinforcement learning methods, in a physical environment. This paper reports our project on using reinforcement learning for better commodity search in Taobao, one of the largest online retail platforms and meanwhile a physical environment with a high sampling cost. Instead of training reinforcement learning in Taobao directly, we present our environment-building approach: we build Virtual-Taobao, a simulator learned from historical customer behavior data, and then we train policies in Virtual-Taobao with no physical sampling costs. To improve the simulation precision, we propose GAN-SD (GAN for Simulating Distributions) for customer feature generation with better matched distribution; we propose MAIL (Multiagent Adversarial Imitation Learning) for generating better generalizable customer actions. To further avoid overfitting the imperfection of the simulator, we propose ANC (Action Norm Constraint) strategy to regularize the policy model. In experiments, Virtual-Taobao is trained from hundreds of millions of real Taobao customers’ records. Compared with the real Taobao, Virtual-Taobao faithfully recovers important properties of the real environment. We further show that the policies trained purely in Virtual-Taobao, which has zero physical sampling cost, can have significantly superior real-world performance to the traditional supervised approaches, through online A/B tests. We hope this work may shed some light on applying reinforcement learning in complex physical environments.


We adopt a Parallel WaveGAN vocoder #REFR pretrained on the VCTK dataset to convert log mel-spectrograms to waveforms.

We split the dataset into ratios of 60%, 20%, and 20% for train, validation, and test set, respectively, considering speakers and utterances.

For conversion scenarios, we select 20 speakers and generate 600 pairs per Seen-to-Seen (S2S) and Unseen-to-Unseen (U2U) scenarios.

After downsampling the audio to 16 kHz, we extract CPC, f0, and log mel-spectrogram features based on a frame size of 25ms, hop size of 10ms, and mel bins of 80.

For training details, we use a batch size of 64, an epoch of 400, and Adam optimizer with a learning rate of 10 −5 .

For model parameters, we take H = 256, C = 512, and L = 6.

For comparison, benchmark models are reproduced using their official codes.

They are trained with melspectrogram features except for S2VC which uses CPC features. The conversion results are available on the demo page. #OTHEREFR

Triaan-Vc: Triple Adaptive Attention Normalization For Any-To-Any Voice Conversion

Abstract: Voice Conversion (VC) must be achieved while maintaining the content of the source speech and representing the characteristics of the target speaker. The existing methods do not simultaneously satisfy the above two aspects of VC, and their conversion outputs suffer from a trade-off problem between maintaining source contents and target characteristics. In this study, we propose Triple Adaptive Attention Normalization VC (TriAAN-VC), comprising an encoder-decoder and an attention-based adaptive normalization block, that can be applied to non-parallel any-to-any VC. The proposed adaptive normalization block extracts target speaker representations and achieves conversion while minimizing the loss of the source content with siamese loss. We evaluated TriAAN-VC on the VCTK dataset in terms of the maintenance of the source content and target speaker similarity. Experimental results for one-shot VC suggest that TriAAN-VC achieves state-of-the-art performance while mitigating the trade-off problem encountered in the existing VC methods.

Parallel Wavegan: A Fast Waveform Generation Model Based On Generative Adversarial Networks With Multi-Resolution Spectrogram

Abstract: We propose Parallel WaveGAN, a distillation-free, fast, and small-footprint waveform generation method using a generative adversarial network. In the proposed method, a non-autoregressive WaveNet is trained by jointly optimizing multi-resolution spectrogram and adversarial loss functions, which can effectively capture the time-frequency distribution of the realistic speech waveform. As our method does not require density distillation used in the conventional teacher-student framework, the entire model can be easily trained. Furthermore, our model is able to generate high-fidelity speech even with its compact architecture. In particular, the proposed Parallel WaveGAN has only 1.44 M parameters and can generate 24 kHz speech waveform 28.68 times faster than real-time on a single GPU environment. Perceptual listening test results verify that our proposed method achieves 4.16 mean opinion score within a Transformer-based text-to-speech framework, which is comparative to the best distillation-based Parallel WaveNet system.


We first attempted to use the 'lungmask' neural network software package to perform the segmentation #REFR .

Also, the scans with a large number of slices will not be able to fit into memory for the graphics processing units (GPU) available.

That is why the previous version of Cov3d first preprocessed the scan volumes to preset sizes.

This usually mean a loss of detail as the volumes were converted to a lower resolution.

This can cause a problem if the lungs only constitute a small proportion of the volume since a lot of the detail for detecting the effects of COVID-19 will be lost.

To rectify this, first we segmented the lungs in the volume and cropped the volumes to a bounding box containing the voxels identified as part of the lungs.

This worked well for slices where the lungs were visible, many of it produced false positive results on slices where no lungs were present.

Instead we used an automated procedure of 3D image morphological operations, adapted from #OTHEREFR .

After cropping, the volumes were resized into standard sizes using scikit-image #OTHEREFR . The three standard sizes are described in table 3.

Enhanced Detection Of The Presence And Severity Of Covid-19 From Ct Scans Using Lung Segmentation

Abstract: Improving automated analysis of medical imaging will provide clinicians more options in providing care for patients. The 2023 AI-enabled Medical Image Analysis Workshop and Covid-19 Diagnosis Competition (AI-MIA-COV19D) provides an opportunity to test and refine machine learning methods for detecting the presence and severity of COVID-19 in patients from CT scans. This paper presents version 2 of Cov3d, a deep learning model submitted in the 2022 competition. The model has been improved through a preprocessing step which segments the lungs in the CT scan and crops the input to this region. It results in a validation macro F1 score for predicting the presence of COVID-19 in the CT scans at 92.22% which is significantly above the baseline of 74%. It gives a macro F1 score for predicting the severity of COVID-19 on the validation set for task 2 as 67% which is above the baseline of 38%.

Automatic Lung Segmentation In Routine Imaging Is A Data Diversity Problem, Not A Methodology Problem

Abstract: Automated segmentation of anatomical structures is a crucial step in many medical image analysis tasks. For lung segmentation, a variety of approaches exist, involving sophisticated pipelines trained and validated on a range of different data sets. However, during translation to clinical routine the applicability of these approaches across diseases remains limited. Here, we show that the accuracy and reliability of lung segmentation algorithms on demanding cases primarily does not depend on methodology, but on the diversity of training data. We compare 4 generic deep learning approaches and 2 published lung segmentation algorithms on routine imaging data with more than 6 different disease patterns and 3 published data sets. We show that a basic approach - U-net - performs either better, or competitively with other approaches on both routine data and published data sets, and outperforms published approaches once trained on a diverse data set covering multiple diseases. Training data composition consistently has a bigger impact than algorithm choice on accuracy across test data sets. We carefully analyse the impact of data diversity, and the specifications of annotations on both training and validation sets to provide a reference for algorithms, training data, and annotation. Results on a seemingly well understood task of lung segmentation suggest the critical importance of training data diversity compared to model choice.


To further validate that the class-level ECS is well calibrated, we also show the reliability diagram for some of the representative classes from Cityscapes #REFR in Fig.

To further validate the property of ECS s c and ECS t c , we show the changes of the two ECSs during training in Fig. 4 .

Here we smooth the value of raw ECS by the EMA.

where τ is the smoothness weight; j represents the j th iteration during training. As we can see from Fig.

4 , ECS values of almost-all classes are monotonously increasing during training for both the source domain and target domain.

5 where we show the relation between the ECS values and the widely-adopted segmentation performance metric mean-Intersection-over-Union (mIoU).

We can see that in general the mIoU is positively correlated with the value of ECS.

2) Class Selection Strategy: We propose our Informed Mix (IMix) based on the DACS #OTHEREFR which is a variant of ClassMix #OTHEREFR , and build upon the idea of bridging images across domains.

DACS is the first to apply the idea of ClassMix #OTHEREFR to domain adaptation.

In DACS, x a is selected from the source domain while x b is selected from the target domain.

Ida: Informed Domain Adaptive Semantic Segmentation

Abstract: Mixup-based data augmentation has been validated to be a critical stage in the self-training framework for unsupervised domain adaptive semantic segmentation (UDA-SS), which aims to transfer knowledge from a well-annotated (source) domain to an unlabeled (target) domain. Existing self-training methods usually adopt the popular region-based mixup techniques with a random sampling strategy, which unfortunately ignores the dynamic evolution of different semantics across various domains as training proceeds. To improve the UDA-SS performance, we propose an Informed Domain Adaptation (IDA) model, a self-training framework that mixes the data based on class-level segmentation performance, which aims to emphasize small-region semantics during mixup. In our IDA model, the class-level performance is tracked by an expected confidence score (ECS). We then use a dynamic schedule to determine the mixing ratio for data in different domains. Extensive experimental results reveal that our proposed method is able to outperform the state-of-the-art UDA-SS method by a margin of 1.1 mIoU in the adaptation of GTA-V to Cityscapes and of 0.9 mIoU in the adaptation of SYNTHIA to Cityscapes.

The Cityscapes Dataset For Semantic Urban Scene Understanding

Abstract: Visual understanding of complex urban street scenes is an enabling factor for a wide range of applications. Object detection has benefited enormously from large-scale datasets, especially in the context of deep learning. For semantic urban scene understanding, however, no current dataset adequately captures the complexity of real-world urban scenes. To address this, we introduce Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling. Cityscapes is comprised of a large, diverse set of stereo video sequences recorded in streets from 50 different cities. 5000 of these images have high quality pixel-level annotations, 20 000 additional images have coarse annotations to enable methods that leverage large volumes of weakly-labeled data. Crucially, our effort exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity. Our accompanying empirical study provides an in-depth analysis of the dataset characteristics, as well as a performance evaluation of several state-of-the-art approaches based on our benchmark.


The second one (B 2 ) was another sequence-tosequence model that incorporated a copy mechanism #REFR to deal with unseen queries.

We evaluated the performance of the decomposition network based on the corpus introduced in Section 4.2.

In particular, we computed the rate of exactly match between the decomposition results and ground truth, and calculated the averaged SARI scores #OTHEREFR of the decomposition results.

SARI score is commonly used in text simplification tasks, it evaluates the goodness of words that are added, deleted, and kept in the simplified sentences.

Two baseline models introduced in #OTHEREFR were used for comparison.

In particular, the first baseline (B 1 ) was a sequence-to-sequence neural network with a 5-layer LSTM encoder and a cross attention.

We trained these models based on the entire corpus and tested their performance via a validation set containing 30% of data that were randomly selected from the corpus with the data queries being replaced by similar but different questions. The evaluation results are summarized in Table 2 .

We measured the effectiveness of the calibration network via success rate and the retain rate.

Here, the success rate was defined as the percentage of the incorrect decomposition results S d that could be successfully amended by the calibration network via the feedback Sc among all the incorrect results.

The retain rate was defined as the percentage of correct decomposition results that remain correct after updating the model's parameters via calibrations.

We performed the experiment based on the above decomposition network D and calculated the success rate and the retain rate based on the dataset introduced in Section 4.3.

Datamator: An Intelligent Authoring Tool For Creating Datamations Via Data Query Decomposition

Abstract: Fig. 1. A datamation generated by Datamator that visualizes the analysis pipeline regarding to the data query "how many students were born in 2000?" through an action-oriented unit visualization.

Incorporating Copying Mechanism In Sequence-To-Sequence Learning

Abstract: We address an important problem in sequence-to-sequence (Seq2Seq) learning referred to as copying, in which certain segments in the input sequence are selectively replicated in the output sequence. A similar phenomenon is observable in human language communication. For example, humans tend to repeat entity names or even long phrases in conversation. The challenge with regard to copying in Seq2Seq is that new machinery is needed to decide when to perform the operation. In this paper, we incorporate copying into neural network-based Seq2Seq learning and propose a new model called CopyNet with encoder-decoder structure. CopyNet can nicely integrate the regular way of word generation in the decoder with the new copying mechanism which can choose sub-sequences in the input sequence and put them at proper places in the output sequence. Our empirical study on both synthetic data sets and real world data sets demonstrates the efficacy of CopyNet. For example, CopyNet can outperform regular RNN-based model with remarkable margins on text summarization tasks.

unseen queries
another sequence-tosequence model


Matrix And Tensor Cur Approximation Methods

The DEIM method and leverage score sampling methods are also used in #REFR to sample columns of the unfolding matrices to approximate the factor matrices.

The sampling methods based on a prior probability distribution are also widely used in the literature using uniform, length-squared, or leverage-score probability distributions, see #OTHEREFR , for an overview on these sampling approaches.

It has been demonstrated that sampling columns with leverage-score probability distribution can provide approximations with relative error accuracy, which is of more interest in practice #OTHEREFR .

The MCUR was extended to the tensor case for different types of tensors decompositions.

For example, the authors in #OTHEREFR proposed to sample some columns of the unfolding matrices randomly to approximate the factor matrices of the Tucker decomposition.

Also, it is suggested to use the Cross2D method to deterministically sample the fibers instead of random sampling.

The MCUR is used in #OTHEREFR to compute low rank approximation of unfolding matrices to compute the TT approximation.

The cross approximation is generalized based on the t-product in #OTHEREFR , where some horizontal and lateral slices are selected.

Here, the slices are selected based on the tubal leverage scores.

The uniform sampling without replacements are used in #OTHEREFR for image/video completion and compression.

However, there are only these works on tubal CUR approximation and this problem has not been investigated extensively.

Robust Low-Tubal-Rank Tensor Recovery Using Discrete Empirical Interpolation Method With Optimized Slice/Feature Selection

Abstract: In this paper, we extend the Discrete Empirical Interpolation Method (DEIM) to the third-order tensor case based on the t-product and use it to select important/significant lateral and horizontal slices/features. The proposed Tubal DEIM (TDEIM) is investigated both theoretically and numerically. The experimental results show that the TDEIM can provide more accurate approximations than the existing methods. An application of the proposed method to the supervised classification task is also presented.

Hoid: Higher Order Interpolatory Decomposition For Tensors Based On Tucker Representation

Abstract: We derive a CUR-type factorization for tensors in the Tucker format based on interpolatory decomposition, which we will denote as Higher Order Interpolatory Decomposition (HOID). Given a tensor $\mathcal{X}$, the algorithm provides a set of column vectors $\{ \mathbf{C}_n\}_{n=1}^d$ which are columns extracted from the mode-$n$ tensor unfolding, along with a core tensor $\mathcal{G}$ and together, they satisfy some error bounds. Compared to the Higher Order SVD (HOSVD) algorithm, the HOID provides a decomposition that preserves certain important features of the original tensor such as sparsity, non-negativity, integer values, etc. Error bounds along with detailed estimates of computational costs are provided. The algorithms proposed in this paper have been validated against carefully chosen numerical examples which highlight the favorable properties of the algorithms. Related methods for subset selection proposed for matrix CUR decomposition, such as Discrete Empirical Interpolation method (DEIM) and leverage score sampling, have also been extended to tensors and are compared against our proposed algorithms.


In contrast to the previously mentioned study, Cruz and Abreu #REFR , and Meneses-Viveros et al.

Saving energy can also be considered in a broader context, such as that of green Internet of Things, as demonstrated by Arshad et al. #OTHEREFR . Arshad et al.

outlined multiple challenges and proposed a taxonomy of green IoT techniques.

This categorization was based on the technologies which were used in various energy-saving IoT models.

Alongside, the authors evaluated different solutions that can be used to reduce the energy consumption of IoT devices.

Overall, the paper provides a comprehensive overview of the different opportunities for energy savings in IoT but has no explicit implication on how to save software-based energy on mobile applications.

#OTHEREFR propose strategies that have direct implications for mobile application developers to utilize the system and optimize battery usage efficiently.

Specifically, Cruz and Abreu present a comprehensive catalogue of energy patterns that can be applied to mobile applications to enhance energy efficiency.

This catalogue, comprising 22 design patterns, was developed through a review of commits, issues, and pull requests of over 1700 mobile app repositories and related research.

The authors assert that some of these patterns may also apply to the IoT domain.

Energy-Saving Strategies For Mobile Web Apps And Their Measurement: Results From A Decade Of Research

Abstract: In 2022, over half of the web traffic was accessed through mobile devices. By reducing the energy consumption of mobile web apps, we can not only extend the battery life of our devices, but also make a significant contribution to energy conservation efforts. For example, if we could save only 5% of the energy used by web apps, we estimate that it would be enough to shut down one of the nuclear reactors in Fukushima. This paper presents a comprehensive overview of energy-saving experiments and related approaches for mobile web apps, relevant for researchers and practitioners. To achieve this objective, we conducted a systematic literature review and identified 44 primary studies for inclusion. Through the mapping and analysis of scientific papers, this work contributes: (1) an overview of the energy-draining aspects of mobile web apps, (2) a comprehensive description of the methodology used for the energy-saving experiments, and (3) a categorization and synthesis of various energy-saving approaches.

Catalog Of Energy Patterns For Mobile Applications

Abstract: Software engineers make use of design patterns for reasons that range from performance to code comprehensibility. Several design patterns capturing the body of knowledge of best practices have been proposed in the past, namely creational, structural and behavioral patterns. However, with the advent of mobile devices, it becomes a necessity a catalog of design patterns for energy efficiency. In this work, we inspect commits, issues and pull requests of 1027 Android and 756 iOS apps to identify common practices when improving energy efficiency. This analysis yielded a catalog, available online, with 22 design patterns related to improving the energy efficiency of mobile apps. We argue that this catalog might be of relevance to other domains such as Cyber-Physical Systems and Internet of Things. As a side contribution, an analysis of the differences between Android and iOS devices shows that the Android community is more energy-aware.


Definition 2.10 (Slater Point). A Slater Point Of The Set

Note that the maximum in (9) is indeed attained because problem #REFR as well as the dual of the parametric minimization problem in its objective function are both solvable.

The inner minimization problem in (12) has a Slater point because of Assumptions 2.9 (ii) and 2.11 (ii).

By #OTHEREFR Theorem 2] , this minimization problem therefore admits a strong dual of the form

Substituting this dual maximization problem back into (12) finally yields #OTHEREFR .

Fix now any maximizer ( #OTHEREFR , which exists thanks to the results of Step 1.

In Step 2 we will show that this maximizer can be used to construct a maximizer of the dual DRO problem (3) (if (3) is solvable) or a sequence of distributions that are feasible and asymptotically optimal in (3) (if 3is not solvable). To this end, define three disjoint index sets

{ ⋆ , ⋆ , ⋆ } , , { ⋆ , ⋆ } ) of problem

New Perspectives On Regularization And Computation In Optimal Transport-Based Distributionally Robust Optimization

Abstract: We study optimal transport-based distributionally robust optimization problems where a fictitious adversary, often envisioned as nature, can choose the distribution of the uncertain problem parameters by reshaping a prescribed reference distribution at a finite transportation cost. In this framework, we show that robustification is intimately related to various forms of variation and Lipschitz regularization even if the transportation cost function fails to be (some power of) a metric. We also derive conditions for the existence and the computability of a Nash equilibrium between the decision-maker and nature, and we demonstrate numerically that nature's Nash strategy can be viewed as a distribution that is supported on remarkably deceptive adversarial samples. Finally, we identify practically relevant classes of optimal transport-based distributionally robust optimization problems that can be addressed with efficient gradient descent algorithms even if the loss function or the transportation cost function are nonconvex (but not both at the same time).

Multivariate Distributionally Robust Convex Regression Under Absolute Error Loss

Abstract: This paper proposes a novel non-parametric multidimensional convex regression estimator which is designed to be robust to adversarial perturbations in the empirical measure. We minimize over convex functions the maximum (over Wasserstein perturbations of the empirical measure) of the absolute regression errors. The inner maximization is solved in closed form resulting in a regularization penalty involves the norm of the gradient. We show consistency of our estimator and a rate of convergence of order $ \widetilde{O}\left( n^{-1/d}\right) $, matching the bounds of alternative estimators based on square-loss minimization. Contrary to all of the existing results, our convergence rates hold without imposing compactness on the underlying domain and with no a priori bounds on the underlying convex function or its gradient norm.

parametric minimization problem
objective function


The mesh surface contains N t triangles (typically N t ∼ 10 4 ), each described by a triplet t j = [t j,1 , t j,2 , t j,3 ] which references the three vertices #REFR ] from the collection {v n } 1≤n≤Nv .

PyCUDA represents a scripting-based approach to GPU run-time code generation.

Its design philosophy and technical elements have previously been covered by its architect and collaborators in #OTHEREFR and #OTHEREFR .

In this paper, PyCUDA discussion is centered around practicalities, and it is told from the perspective of solving a specific geometry problem, where the goal is to determine if a set of rays intersect a triangle mesh surface.

For simplicity, we use the term 'ray' to refer to a line segment specified by a start and end point; this may be written as l i = (r start i , r end i ).

A typical problem involves a large number of rays, say N r ∈ [10 6 , 10 8 ], which makes parallel computation highly desirable.

The Moller-Trumbore algorithm #OTHEREFR provides an efficient solution to this problem.

However, an exhaustive search [comparing N t triangles with N r rays] is very slow if acceleration structures are not used even when computation is done on a GPU (see Table 1 in #OTHEREFR ).

In the existing CUDA implementation, Morton code is used to encode the location of all mesh triangles.

These are subsequently reordered to preserve spatial proximity and represented by a bounding volume hierarchy (BVH) using a binary radix tree #OTHEREFR .

An Experience With Pycuda: Refactoring An Existing Implementation Of A Ray-Surface Intersection Algorithm

Abstract: This article is a sequel to "GPU implementation of a ray-surface intersection algorithm in CUDA" (arXiv:2209.02878) [1]. Its main focus is PyCUDA which represents a Python scripting approach to GPU run-time code generation in the Compute Unified Device Architecture (CUDA) framework. It accompanies the open-source code distributed in GitHub which provides a PyCUDA implementation of a GPU-based line-segment, surface-triangle intersection test. The objective is to share a PyCUDA learning experience with people who are new to PyCUDA. Using the existing CUDA code and foundation from [1] as the starting point, we document the key changes made to facilitate a transition to PyCUDA. As the CUDA source for the ray-surface intersection test contains both host and device code and uses multiple kernel functions, these notes offer a substantive example and real-world perspective of what it is like to utilise PyCUDA. It delves into custom data structures such as binary radix tree and highlights some possible pitfalls. The case studies present a debugging strategy which may be used to examine complex C structures (such as a triangle-mesh bounding volume hierarchy) in device memory using standard Python tools without the CUDA-GDB debugger. 1 Existing CUDA implementation: https://github.com/raymondleung8/gpu-ray-surface-intersection-in-cuda.

Gpu Scripting And Code Generation With Pycuda

Abstract: Publisher Summary A common lament in the field of scientific computing concerns the ever-widening gap between hypothetical machine capabilities and the effort an individual programmer is able to spend to move a computational application towards exploiting a machine to the full extent of its capability. More sophisticated tools, compilers, and libraries are generally hoped to level this field, enabling users to achieve good results even with modest investment. PyCUDA is a contribution to the tools for graphics processing unit (GPU) computing. PyCUDA has a two-fold aim. First, it aims to simplify the usage of existing basic concepts of CUDA C. Importantly, it does not attempt to change or reinvent the basic notions of GPU programming, but instead, as a foundation for further tools, just exposes them as is through a carefully engineered interface. Key features of this interface include generous use of sensible defaults, automatic error checking, and automatic resource management. Second, and strictly on top of the first, basic layer, PyCUDA provides abstractions that support a number of very common usage patterns involving a GPU equivalent of NumPy arrays. Although PyCUDA gives the user access to a compiled language (CUDA C), it attempts to avoid the development iteration penalty commonly associated with compiled languages, instead retaining the satisfaction and immediacy of scripting. PyCUDA and PyOpenCL, as open-source projects, thrive on user feedback, particularly feedback regarding limitations, bugs, or missing features that users encounter.


Multi-Label Classification On Graph-Structured Data

GCN-LPA #REFR utilizes LPA serves as regularization to assist the GCN in learning proper edge weights that lead to improved classification performance.

Prior to the advent of GNNs label propagation (LPA) algorithms constituted popular approaches for the task of node classification. Both LPA and GNNs are based on message passing.

While GNNs propagate and transform node features, LPA propagates node label information along the edges of the graph to predict the label distribution of the unlabelled nodes.

A few recent works (Yang et al., 2021; #OTHEREFR have explored the possibilities of combining LPA and GNNs.

(Yang et al., 2021) employs knowledge distillation in which the trained GNN model (teacher model) is distilled into a combination of GNN and parameterized label propagation module.

Different from our work both of the above works implicitly assume a multi-class setting.

Moreover, (Yang et al., 2021) focuses on the interpretable extraction of knowledge from trained GNN and can only discover the patterns learned by the teacher GNN.

Multi-Label Node Classification On Graph-Structured Data

Abstract: Graph Neural Networks (GNNs) have shown state-of-the-art improvements in node classification tasks on graphs. While these improvements have been largely demonstrated in a multi-class classification scenario, a more general and realistic scenario in which each node could have multiple labels has so far received little attention. The first challenge in conducting focused studies on multi-label node classification is the limited number of publicly available multi-label graph datasets. Therefore, as our first contribution, we collect and release three real-world biological datasets and develop a multi-label graph generator to generate datasets with tunable properties. While high label similarity (high homophily) is usually attributed to the success of GNNs, we argue that a multi-label scenario does not follow the usual semantics of homophily and heterophily so far defined for a multi-class scenario. As our second contribution, besides defining homophily for the multi-label scenario, we develop a new approach that dynamically fuses the feature and label correlation information to learn label-informed representations. Finally, we perform a large-scale comparative study with 10 methods and 9 datasets which also showcase the effectiveness of our approach. We release our benchmark at https://anonymous.4open.science/r/LFLF-5D8C/.

Unifying Graph Convolutional Neural Networks And Label Propagation

Abstract: Label Propagation (LPA) and Graph Convolutional Neural Networks (GCN) are both message passing algorithms on graphs. Both solve the task of node classification but LPA propagates node label information across the edges of the graph, while GCN propagates and transforms node feature information. However, while conceptually similar, theoretical relation between LPA and GCN has not yet been investigated. Here we study the relationship between LPA and GCN in terms of two aspects: (1) feature/label smoothing where we analyze how the feature/label of one node is spread over its neighbors; And, (2) feature/label influence of how much the initial feature/label of one node influences the final feature/label of another node. Based on our theoretical analysis, we propose an end-to-end model that unifies GCN and LPA for node classification. In our unified model, edge weights are learnable, and the LPA serves as regularization to assist the GCN in learning proper edge weights that lead to improved classification performance. Our model can also be seen as learning attention weights based on node labels, which is more task-oriented than existing feature-based attention models. In a number of experiments on real-world graphs, our model shows superiority over state-of-the-art GCN-based methods in terms of node classification accuracy.


B. Learning With L 2 -Polynomial Regression

However, we note that under special cases (e.g., d k = ⌈n r ⌉) the regression can be done in O(nd kω(r) ), where ω(r) is a constant given in #REFR .

The factor d k is because a polynomial of degree up to k is a linear combination of monomials of the form k j=1 X ij , where

Linear regression can be implemented via Moore-Penrose (generalized) inverse.

The generalized inverse is computed using classical methods in O(nD 2 + D 3 ).

Hence, given that n > D, we can perform polynomial regression in O(nD 2 ) = O(nd 2k ).

Hence, the computational complexity of this algorithm is O(nd Θ(k) ) as noted in Table I .

Agnostic Pac Learning Of K-Juntas Using L 2 -Polynomial Regression

Abstract: Many conventional learning algorithms rely on loss functions other than the natural 0-1 loss for computational efficiency and theoretical tractability. Among them are approaches based on absolute loss (L 1 regression) and square loss (L 2 regression). The first is proved to be an agnostic PAC learner for various important concept classes such as juntas, and half-spaces. On the other hand, the second is preferable because of its computational efficiency, which is linear in the sample size. However, PAC learnability is still unknown as guarantees have been proved only under distributional restrictions. The question of whether L 2 regression is an agnostic PAC learner for 0-1 loss has been open since 1993 and yet has to be answered. This paper resolves this problem for the junta class on the Boolean cube-proving agnostic PAC learning of k-juntas using L 2 polynomial regression. Moreover, we present a new PAC learning algorithm based on the Boolean Fourier expansion with lower computational complexity. Fourier-based algorithms, such as Linial et al. (1993), have been used under distributional restrictions, such as uniform distribution. We show that with an appropriate change, one can apply those algorithms in agnostic settings without any distributional assumption. We prove our results by connecting the PAC learning with 0-1 loss to the minimum mean square estimation (MMSE) problem. We derive an elegant upper bound on the 0-1 loss in terms of the MMSE error. Based on that, we show that the sign of the MMSE is a PAC learner for any concept class containing it.

Improved Rectangular Matrix Multiplication Using Powers Of The Coppersmith-Winograd Tensor

Abstract: In the past few years, successive improvements of the asymptotic complexity of square matrix multiplication have been obtained by developing novel methods to analyze the powers of the Coppersmith-Winograd tensor, a basic construction introduced thirty years ago. In this paper we show how to generalize this approach to make progress on the complexity of rectangular matrix multiplication as well, by developing a framework to analyze powers of tensors in an asymmetric way. By applying this methodology to the fourth power of the Coppersmith-Winograd tensor, we succeed in improving the complexity of rectangular matrix multiplication. Let $\alpha$ denote the maximum value such that the product of an $n\times n^\alpha$ matrix by an $n^\alpha\times n$ matrix can be computed with $O(n^{2+\epsilon})$ arithmetic operations for any $\epsilon>0$. By analyzing the fourth power of the Coppersmith-Winograd tensor using our methods, we obtain the new lower bound $\alpha>0.31389$, which improves the previous lower bound $\alpha>0.30298$ obtained five years ago by Le Gall (FOCS'12) from the analysis of the second power of the Coppersmith-Winograd tensor. More generally, we give faster algorithms computing the product of an $n\times n^k$ matrix by an $n^k\times n$ matrix for any value $k\neq 1$. (In the case $k=1$, we recover the bounds recently obtained for square matrix multiplication). These improvements immediately lead to improvements in the complexity of a multitude of fundamental problems for which the bottleneck is rectangular matrix multiplication, such as computing the all-pair shortest paths in directed graphs with bounded weights.


Therefore, models such as BERT #REFR commonly limit the maximum sequence length to 512 tokens.

Transformer-based architectures #OTHEREFR have become ubiquitous in natural language processing research.

A key attribute to their success is the multi-head self-attention mechanism #OTHEREFR .

However, its computational and memory requirements grow quadratically with input sequence length.

Longer documents are truncated #OTHEREFR or staggered position embeddings are used .

This limitation motivated long-text transfomers, such as Big Bird #OTHEREFR and Longformer #OTHEREFR , which reduce the complexity of self-attention through the use of sparse attention and improve the performance of transformers on long documents.

The task of rationale extraction focuses on selecting a subset of input as a justification for the model's output.

In particular, our work focuses on extracting token-level rationales from long documents.

The models presented are supervised using document labels only, with no token-level annotations used during training.

Finding The Needle In A Haystack: Unsupervised Rationale Extraction From Long Text Classifiers

Abstract: Long-sequence transformers are designed to improve the representation of longer texts by language models and their performance on downstream document-level tasks. However, not much is understood about the quality of token-level predictions in long-form models. We investigate the performance of such architectures in the context of document classification with unsupervised rationale extraction. We find standard soft attention methods to perform significantly worse when combined with the Longformer language model. We propose a compositional soft attention architecture that applies RoBERTa sentence-wise to extract plausible rationales at the tokenlevel. We find this method to significantly outperform Longformer-driven baselines on sentiment classification datasets, while also exhibiting significantly lower runtimes.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).