In #REFR the authors find that this approach is capable of capturing acoustic correlates of prosody from the reference, even if spoken by an unseen speaker or containing verbal content not related to the target-text.

Most prosody transfer approaches follow #OTHEREFR and normally comprise a core acoustic model, most commonly Tacotron #OTHEREFR , and a prosody encoder.

The acoustic model and the prosody encoder are jointly trained.

The prosody encoder produces a fixed-size latent prosody embedding from the reference mel-spectrogram which is combined with the intermediate acoustic model representation then passed to the decoder which generates the output mel-spectrogram.

In #OTHEREFR , the joint model is trained to minimize target mel-spectrogram loss, which can be viewed as a reconstruction loss.

The information capacity of the latent prosody representation has an important effect in the downstream TTS task.

Representations that are lower-capacity than that in #OTHEREFR , such as #OTHEREFR , for example], are effectively bottle-necked by their capacity and cannot model the same level of detail as #OTHEREFR .

These models are used for tasks such as speaking-style or emotion modelling.

Representations of higher capacity than #OTHEREFR , such as #OTHEREFR , capture a considerably higher-detailed representation of the reference and are highly reference-specific.

Therefore, high-capacity representations are used for tasks such as voice-puppetry where we expect the lexical content of the reference to be similar to the target text.

Do Prosody Transfer Models Transfer Prosody?

Abstract: Some recent models for Text-to-Speech synthesis aim to transfer the prosody of a reference utterance to the generated target synthetic speech. This is done by using a learned embedding of the reference utterance, which is used to condition speech generation. During training, the reference utterance is identical to the target utterance. Yet, during synthesis, these models are often used to transfer prosody from a reference that differs from the text or speaker being synthesized. To address this inconsistency, we propose to use a different, but prosodically-related, utterance during training too. We believe this should encourage the model to learn to transfer only those characteristics that the reference and target have in common. If prosody transfer methods do indeed transfer prosody they should be able to be trained in the way we propose. However, results show that a model trained under these conditions performs significantly worse than one trained using the target utterance as a reference. To explain this, we hypothesize that prosody transfer models do not learn a transferable representation of prosody, but rather a utterance-level representation which is highly dependent on both the reference speaker and reference text.

Towards End-To-End Prosody Transfer For Expressive Speech Synthesis With Tacotron

Abstract: We present an extension to the Tacotron speech synthesis architecture that learns a latent embedding space of prosody, derived from a reference acoustic representation containing the desired prosody. We show that conditioning Tacotron on this learned embedding space results in synthesized audio that matches the prosody of the reference signal with fine time detail even when the reference and synthesis speakers are different. Additionally, we show that a reference prosody embedding can be used to synthesize text that is different from that of the reference utterance. We define several quantitative and subjective metrics for evaluating prosody transfer, and report results with accompanying audio samples from single-speaker and 44-speaker Tacotron models on a prosody transfer task.


There have been a number of studies on analyzing the characteristics of different layers of transformer models. #REFR ; Van Aken et al.

(2019) used probing to report that in BERT models the lower layers carry the most information about linear word order, syntactic information is most prominent in the middle layers, and the final layers of BERT are the most task-specific. Van Aken et al.

(2019) also show that similar behavior is observed in other transformer models such as GPT2. #OTHEREFR

( , 2022 studied the role of feed-forward layers in transformer models.

They demonstrate that representations across different layers capture meaningful semantic and syntactic patterns, and describe how model predictions are gradually refined as they progress across the different layers.

Aiming to reduce the computational load of transformers, multiple works have explored earlyexiting, i.e., performing some calculations without passing through all of the model layers.

The Benefits Of Bad Advice: Autocontrastive Decoding Across Model Layers

Abstract: Applying language models to natural language processing tasks typically relies on the representations in the final model layer, as intermediate hidden layer representations are presumed to be less informative. In this work, we argue that due to the gradual improvement across model layers, additional information can be gleaned from the contrast between higher and lower layers during inference. Specifically, in choosing between the probable next token predictions of a generative model, the predictions of lower layers can be used to highlight which candidates are best avoided. We propose a novel approach that utilizes the contrast between layers to improve text generation outputs, and show that it mitigates degenerative behaviors of the model in open-ended generation, significantly improving the quality of generated texts. Furthermore, our results indicate that contrasting between model layers at inference time can yield substantial benefits to certain aspects of general language model capabilities, more effectively extracting knowledge during inference from a given set of model parameters.

A Primer In Bertology: What We Know About How Bert Works

Abstract: Transformer-based models are now widely used in NLP, but we still do not understand a lot about their inner workings. This paper describes what is known to date about the famous BERT model (Devlin et al. 2019), synthesizing over 40 analysis studies. We also provide an overview of the proposed modifications to the model and its training regime. We then outline the directions for further research.


Similar observation can be made on TGAN #REFR , i.e., simple fine-tuning based method without explicit knowledge preservation. In contrast, our method (we discuss it in Sec. 5) can address this issue.

More importantly, the root cause of such incompatible knowledge transfer is the least important filters in G s determined to be irrelevant to the target domain adaptation, and fine-tuning is not adequate for removing the incompatible knowledge after adaptation.

Specifically, we summarize our observations in Figure 1 and Figure 2 :

Observation 1: In Figure 1 (c), we visualize the generated images by different methods with fixed noise input.

Interestingly, features that are incompatible for the target domain are indeed transferred after adaptation with different knowledge preservation criteria, e.g., "tree on sea" where "tree" is from Church domain, and "Cat with glasses" where "glasses" is from FFHQ domain.

All these incompatible source features severely curtail the realism of generated target images.

Observation 2: In Figure 2 , we dissect and visualize the incompatible features observed in Figure 1 , and find their mostly correlated filters in G s and G t .

Surprisingly, we find that the filters in G s identified with least importance for target domain are mostly relevant to incompatible features transferred from the source, which is the root cause of degradation of realism of generated images.

After adaptation, the same filters will still cause the same type of incompatible features, and fine-tuning for knowledge update cannot effectively address this issue.

When the target domains become distant, this observation is more obvious.

Our analysis uncovers an important issue of existing SOTA FSIG methods: source domain features that are incompatible to the target domain are transferred to the target generator, with various knowledge preservation criteria.

Exploring Incompatible Knowledge Transfer In Few-Shot Image Generation

Abstract: ProgressiveGAN StyleGAN-V2 Church ↦ Sailboat FFHQ ↦ AFHQ-Cat Church ↦ Sailboat FFHQ ↦ AFHQ-Cat Pretraining on large source dataset (e.g., LSUN-Church, FFHQ…) Adaptation on few-shot target samples

Transferring Gans: Generating Images From Limited Data

Abstract: Transferring knowledge of pre-trained networks to new domains by means of fine-tuning is a widely used practice for applications based on discriminative models. To the best of our knowledge this practice has not been studied within the context of generative deep networks. Therefore, we study domain adaptation applied to image generation with generative adversarial networks. We evaluate several aspects of domain adaptation, including the impact of target domain size, the relative distance between source and target domain, and the initialization of conditional GANs. Our results show that using knowledge from pre-trained networks can shorten the convergence time and can significantly improve the quality of the generated images, especially when target data is limited. We show that these conclusions can also be drawn for conditional GANs even when the pre-trained model was trained without conditioning. Our results also suggest that density is more important than diversity and a dataset with one or few densely sampled classes is a better source model than more diverse datasets such as ImageNet or Places.


Echoing the previous findings in transfer learning #REFR , we also observe a performance boost of models of larger size pretrained on larger datasets.

As shown in Table 1 , our method can outperform the baseline methods over different datasets, including large-scale dataset, ImageNet, for both CNN and ViT classifiers.

Our empirical result confirms the previous findings that ViT can generally perform better than CNN classifiers in accuracy on uncertainty estimation-related tasks #OTHEREFR .

Effect of Different Foundation Models Given the diversity of foundation models, which can vary in terms of architecture, training data, and optimization losses, it is important to further investigate the impact of different pretrained models on our proposed method.

We demonstrate the average performance over base models and datasets (excluding ImageNet 1 ) for different pretrained models in Figure 2 , which shows that every pretrained model in our model candidate pool can surpass MSP on average.

We compare the performance of models pretrained on datasets of increasing size: ImageNet-1K, ImageNet-21K, and CLIP WebImageText dataset (400 million image-text pairs).

With the similar model size and pretrained on the same dataset, ViTs generally perform better than CNNs, except for the cases where the dataset is in a smaller scale, e.g., ImageNet-1K. The self-supervised pretrained models, e.g.

MoCov3 ViT/ResNet, achieve comparable results with models pretrained with supervision signals.

Similar to the previous findings in #OTHEREFR , which suggests that better performing models transfer better to the downstream tasks, we find the failure detection performance of pretrained models related to these models' accuracy performance in ImagetNet-1K, as shown in Figure  8 .

Thus, we suggest to select the pretrained model with the highest accuracy on ImageNet-1K for failure detection, without any prior knowledge about the trained models, e.g. the training datasets.

Great Models Think Alike: Improving Model Reliability Via Inter-Model Latent Agreement

Abstract: Reliable application of machine learning is of primary importance to the practical deployment of deep learning methods. A fundamental challenge is that models are often unreliable due to overconfidence (Hendrycks & Gimpel, 2017). In this paper, we estimate a model's reliability by measuring the agreement between its latent space, and the latent space of a foundation model. However, it is challenging to measure the agreement between two different latent spaces due to their incoherence, e.g., arbitrary rotations and different dimensionality. To overcome this incoherence issue, we design a neighborhood agreement measure between latent spaces and find that this agreement is surprisingly well-correlated with the reliability of a model's predictions. Further, we show that fusing neighborhood agreement into a model's predictive confidence in a post-hoc way significantly improves its reliability. Theoretical analysis and extensive experiments on failure detection across various datasets verify the effectiveness of our method on both in-distribution and out-of-distribution settings.

Big Transfer (Bit): General Visual Representation Learning

Abstract: Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.


In the midst of increased privacy protection and data oversight, the phenomenon of data existing as isolated islands has emerged #REFR .

Therefore, the acquisition, storage, analysis, and protection of big data is a crucial and challenging task.

In the metaverse, the constructed digital space is essentially data.

In other words, the metaverse can be regarded as a fusion mechanism for big data and various information technologies.

When building a bridge between virtual and reality, real-time data gathering and processing capabilities of sensors and smart devices are the cornerstones for supporting large numbers of users online at once.

The rich data of multiple business departments in an enterprise is defined by each other independently.

Data from a single department is similar to an isolated island that is incapable of interacting with other rich data #OTHEREFR .

In the era of the metaverse, data is undoubtedly one of the most important assets of an enterprise.

Therefore, it is vital to shatter the isolated island of data in order to actualize information exchange and the hidden value underlying big data #OTHEREFR .

Federated Learning For Metaverse: A Survey

Abstract: The metaverse, which is at the stage of innovation and exploration, faces the dilemma of data collection and the problem of private data leakage in the process of development. This can seriously hinder the widespread deployment of the metaverse. Fortunately, federated learning (FL) is a solution to the above problems. FL is a distributed machine learning paradigm with privacy-preserving features designed for a large number of edge devices. Federated learning for metaverse (FL4M) will be a powerful tool. Because FL allows edge devices to participate in training tasks locally using their own data, computational power, and model-building capabilities. Applying FL to the metaverse not only protects the data privacy of participants but also reduces the need for high computing power and high memory on servers. Until now, there have been many studies about FL and the metaverse, respectively. In this paper, we review some of the early advances of FL4M, which will be a research direction with unlimited development potential. We first introduce the concepts of metaverse and FL, respectively. Besides, we discuss the convergence of key metaverse technologies and FL in detail, such as big data, communication technology, the Internet of Things, edge computing, blockchain, and extended reality. Finally, we discuss some key challenges and promising directions of FL4M in detail. In summary, we hope that our up-to-date brief survey can help people better understand FL4M and build a fair, open, and secure metaverse. CCS CONCEPTS • Computing methodologies → Federated learning.

Federated Machine Learning: Concept And Applications

Abstract: Today's AI still faces two major challenges. One is that in most industries, data exists in the form of isolated islands. The other is the strengthening of data privacy and security. We propose a possible solution to these challenges: secure federated learning. Beyond the federated learning framework first proposed by Google in 2016, we introduce a comprehensive secure federated learning framework, which includes horizontal federated learning, vertical federated learning and federated transfer learning. We provide definitions, architectures and applications for the federated learning framework, and provide a comprehensive survey of existing works on this subject. In addition, we propose building data networks among organizations based on federated mechanisms as an effective solution to allow knowledge to be shared without compromising user privacy.


A T E X I T S H A 1 _ B A S E 6 4 = " K H E L A V A O A S U R C + R 0 Z R A V F N P T B G 4 = " > A

The problem is known to be hard as it relates to the challenging close-loop controlled sensing problem #REFR .

v g v T e 8 7 F J v T T I A m O f Z E = " > A A A C F X i c b V D L S s N A F J 3 U V 4 2 v q k s 3 g 0 V w I S W p g o K b o p s u K 9 o H N L F M p p N m 6 G Q S Z i Z C C f k J N / 6 K G x e K u B X c + T d O 2 i y 0 9 c L A m X P P f R 0 v Z l Q q y / o 2 S k v L K 6 t r 5 X V z Y 3

N r e 6 e y u 9 e R U S I w a e O I R a L n I U k Y 5 a S t q G K k F w u C Q o + R r j e + z possible values depending on a underlying hypothesis, and the radar receiver aims to guess this hypothesis.

Sensing performance is measured in terms of exponential decay-rate of the probability of error, either the minimum exponential decay-rate over all hypotheses #OTHEREFR , #OTHEREFR or the set of decayrates that are simultaneously achievable under the different hypotheses #OTHEREFR .

The work #OTHEREFR also studies a mono-static version of this problem, assuming that Tx coincides with the radar receiver and thus can use the generalized-feedback signals also for communication purposes.

For this mono-static radar scenario however only a coding scheme but no converse is presented.

v P d B y I k j f i d m s T E D d G I U 5 9 i p D Q 1 q J w 4 A V K p E y I V Y M T S Z p Y 5 l 6 b j I T y W D M k A w v y b S 2 6 z e y 2 v W j V r G n A R 2 A W o g i J a g 8 q X M 4 x w E h K u s G 4 n + 7 Y V K z d F Q l H M S G Y 6 i S S x n o V G p K 8 h R y G R b j q 9 K o N H m h l C P x L 6 c Q W n 7 O + K F I V S T k J P K / P 1 5 X w u J / / L 9 R P l X 7 g p 5 X G i C M e z Q X 7 C o I p g b h E c U k G w Y h M N E B Z U 7 w p x g A T C S h

The first information-theoretic work #OTHEREFR on ISAC determined the fundamental limits of the rate-distortion version of the ISAC problem in Figure 1 .

assuming that the Tx can use the feedback signals also for coding (i.e., under the closeloop coding assumption).

Extensions to network scenarios and to scenarios with secrecy constraints were subsequently presented in #OTHEREFR - #OTHEREFR .

In this paper, we consider both the rate-distortion version and the hypothesis testing versions of the model in Figure 1 .

Strong Converses For Memoryless Bi-Static Isac

Abstract: The paper characterizes the fundamental limits of integrated sensing and communication (ISAC) systems with a bi-static radar, where the radar receiver is located close to the transmitter and estimates or detects the state based on the transmitter's channel inputs and the backscattered signals. Two models are considered. In the first model, the memoryless state sequence is distributed according to a fixed distribution and the goal of the radar receiver is to reconstruct this statesequence with smallest possible distortion. In the second model, the memoryless state is distributed either according to PS or to QS and the radar's goal is to detect this underlying distribution so that the missed-detection error probability has maximum exponential decay-rate (maximum Stein exponent). Similarly to previous results, our fundamental limits show that the tradeoff between sensing and communication solely stems from the empirical statistics of the transmitted codewords which influences both performances. The main technical contribution are two strong converse proofs that hold for all probabilities of communication error and excess-distortion probability or false-alarm probability δ summing to less than 1, +δ < 1. These proofs are based on two parallel change-of-measure arguments on the sets of typical sequences, one change-of-measure to obtain the desired bound on the communication rate, and the second to bound the sensing performance. Index Terms-Integrated sensing and communication, strong converse, change of measure arguments.

Controlled Sensing For Multihypothesis Testing

Abstract: The problem of multiple hypothesis testing with observation control is considered in both fixed sample size and sequential settings. In the fixed sample size setting, for binary hypothesis testing, the optimal exponent for the maximal error probability corresponds to the maximum Chernoff information over the choice of controls, and a pure stationary open-loop control policy is asymptotically optimal within the larger class of all causal control policies. For multihypothesis testing in the fixed sample size setting, lower and upper bounds on the optimal error exponent are derived. It is also shown through an example with three hypotheses that the optimal causal control policy can be strictly better than the optimal open-loop control policy. In the sequential setting, a test based on earlier work by Chernoff for binary hypothesis testing, is shown to be first-order asymptotically optimal for multihypothesis testing in a strong sense, using the notion of decision making risk in place of the overall probability of error. Another test is also designed to meet hard risk constrains while retaining asymptotic optimality. The role of past information and randomization in designing optimal control policies is discussed.


Following the DeepSDF model proposed by #REFR , we represent this SDF function in an implicit neural representation (INR) .

GANs excel at this second stage owed in part to image-to-image translation methods such as pix2pix #OTHEREFR , which have found widespread application in microscopy image synthesis.

However, the first stage requires fundamentally different models and, most importantly, shape representations that can deal with the variation in cell shapes. As exemplified in Fig.

1 , cells exhibit a range of visually observable phenomena including global cell body changes such as cell growth, cell division, deformation in cell tissue, or projections of leading edge during cell motility, as well as localized body changes, such as blebbing (i.e., randomly growing and retracting blobs on the cell surface) or growing filopodia (i.e., highly motile thin protrusions budding off the main cell body).

In this work, we synthesize living cell shapes in 3D+time.

We propose to model the cell surface as the zero level-set of a continuous signed distance function (SDF) in 3D space and time.

Such an INR consists of a multilayer perceptron (MLP) that we jointly optimize with a latent space using a large set of 3D cell shape time-lapse sequences.

Once trained, owing to the continuous implicit representation, the optimized model can be used to synthesize completely new cell shape sequences at any given spatial or temporal resolution.

In contrast to voxel-based or mesh-based cell shape representations with discretized sampling rates, our approach alleviates the limitations on spatial and temporal resolution and allows us to model and represent cell shapes at an arbitrary level of detail.

By parametrizing the cell surface implicitly through differentiable and, therefore, trainable neural networks, the complexity of the resulting model is independent of the spatial and temporal resolution and instead scales with the complexity of the cell surface.

Hence, in contrast to voxel-based or mesh-based representations, the resulting network has fixed memory requirements.

Medical Image Analysis

Abstract: Data-driven cell tracking and segmentation methods in biomedical imaging require diverse and information-rich training data. In cases where the number of training samples is limited, synthetic computer-generated data sets can be used to improve these methods. This requires the synthesis of cell shapes as well as corresponding microscopy images using generative models. To synthesize realistic living cell shapes, the shape representation used by the generative model should be able to accurately represent fine details and changes in topology, which are common in cells. These requirements are not met by 3D voxel masks, which are restricted in resolution, and polygon meshes, which do not easily model processes like cell growth and mitosis. In this work, we propose to represent living cell shapes as level sets of signed distance functions (SDFs) which are estimated by neural networks. We optimize a fully-connected neural network to provide an implicit representation of the SDF value at any point in a 3D+time domain, conditioned on a learned latent code that is disentangled from the rotation of the cell shape. We demonstrate the effectiveness of this approach on cells that exhibit rapid deformations (Platynereis dumerilii), cells that grow and divide (C. elegans), and cells that have growing and branching filopodial protrusions (A549 human lung carcinoma cells). A quantitative evaluation using shape features, Hausdorff distance, and Dice similarity coefficients of real and synthetic cell shapes shows that our model can generate topologically plausible complex cell shapes in 3D+time with high similarity to real living cell shapes. Finally, we show how microscopy images of living cells that correspond to our generated cell shapes can be synthesized using an image-to-image model.

Deepsdf: Learning Continuous Signed Distance Functions For Shape Representation

Abstract: Computer graphics, 3D computer vision and robotics communities have produced multiple approaches to representing 3D geometry for rendering and reconstruction. These provide trade-offs across fidelity, efficiency and compression capabilities. In this work, we introduce DeepSDF, a learned continuous Signed Distance Function (SDF) representation of a class of shapes that enables high quality shape representation, interpolation and completion from partial and noisy 3D input data. DeepSDF, like its classical counterpart, represents a shape's surface by a continuous volumetric field: the magnitude of a point in the field represents the distance to the surface boundary and the sign indicates whether the region is inside (-) or outside (+) of the shape, hence our representation implicitly encodes a shape's boundary as the zero-level-set of the learned function while explicitly representing the classification of space as being part of the shapes interior or not. While classical SDF's both in analytical or discretized voxel form typically represent the surface of a single shape, DeepSDF can represent an entire class of shapes. Furthermore, we show state-of-the-art performance for learned 3D shape representation and completion while reducing the model size by an order of magnitude compared with previous work.


Evaluation Of The Quality Of The Embeddings In Similarity Task

In the case of Basque, we used the pure similarity RG dataset and the relatedness dataset WS created by #REFR .

In this work, we have chosen three types of embeddings, which perform differently in pure semantic similarity #OTHEREFR and relatedness #OTHEREFR measures.

Broadly, wordnet embeddings are more precise for measuring pure similarity relations, text embeddings are more sensitive to compute relatedness, and hybrid embeddings have been proven to be more robust for capturing semantic relations in general #OTHEREFR .

Hence in this work, we chose the available similarity and relatedness datasets for evaluating all embeddings in both languages.

In Spanish, we operated with the pure similarity datasets RG65 (RG) #OTHEREFR and

SimLex999 (SL) #OTHEREFR , and the relatedness dataset Wordsim353 (WS) #OTHEREFR .

In both Spanish and Basque, F T txt , F T kb and F T hyb representations have been compared with the baseline, using publicly available text-based represen- • F T txt embeddings: F T txt results perform similarly to the baseline ones in both languages, with two exceptions.

One, the Basque F T txt result in the WS dataset is higher than the baseline, likely due to the greater corpus size Euscrawl.

Second, the Spanish F T txt in the SL dataset is lower than the baseline.

The only plausible explanation for this result in the Spanish SL may lie in the differences in the pre-processing of the corpus; the baseline text corpus was tokenized with Europarl pre-processing tools 14 , whereas the NLTK tokenizer has been used in the present work.

• F T kb embeddings: being this type of embeddings more suited for pure similarity datasets (RG and SL) rather than for relatedness ones (WS), F T kb results are higher than the baseline in the Basque RG and the Spanish SL, but not in the Spanish RG.

Bridging Natural Language Processing And Psycholinguistics: Computationally Grounded Semantic Similarity Datasets For Basque And Spanish

Abstract: We present a computationally-grounded word similarity dataset based on two well-known Natural Language Processing resources; text corpora and knowledge bases. This dataset aims to fulfil a gap in psycholinguistic research by providing a variety of quantifications of semantic similarity in an extensive set of noun pairs controlled by variables that play a significant role in lexical processing. The dataset creation has consisted in three steps, 1) computing four key psycholinguistic features for each noun; concreteness, frequency, semantic and phonological neighbourhood density; 2) pairing nouns across these four variables; 3) for each noun pair, assigning three types of word similarity measurements, computed out of text, Wordnet and hybrid embeddings. The present dataset includes noun pairs' information in Basque and European Spanish, but further work intends to extend it to more languages.

Bilingual Embeddings With Random Walks Over Multilingual Wordnets

Abstract: Bilingual word embeddings represent words of two languages in the same space, and allow to transfer knowledge from one language to the other without machine translation. The main approach is to train monolingual embeddings first and then map them using bilingual dictionaries. In this work, we present a novel method to learn bilingual embeddings based on multilingual knowledge bases (KB) such as WordNet. Our method extracts bilingual information from multilingual wordnets via random walks and learns a joint embedding space in one go. We further reinforce cross-lingual equivalence adding bilingual constraints in the loss function of the popular Skip-gram model. Our experiments on twelve cross-lingual word similarity and relatedness datasets in six language pairs covering four languages show that: 1) our method outperforms the state-of-the-art mapping method using dictionaries; 2) multilingual wordnets on their own improve over text-based systems in similarity datasets; 3) the combination of wordnet-generated information and text is key for good results. Our method can be applied to richer KBs like DBpedia or BabelNet, and can be easily extended to multilingual embeddings. All our software and resources are open source.


As a baseline, we also plot the O(n −2 ) approximation of the finite blocklength Polyanskyi-Poor-Verdú (PPV) meta-converse bound #REFR .

We evaluate the performance of rate-compatible polar codes with 64 ≤ N ≤ 1024 for the additive white Gaussian noise (AWGN) channel with binary phase shift keying (BPSK) mapping.

We compare the proposed s-symmetric β -expansionbased codes under automorphism ensemble successive cancellation (AE-SC) decoding with ensemble size M = 8 to the 5G standardized codes under CRC-aided successive cancellation list (CA-SCL) decoding with list size L = 8.

The 5G codes use a 6-bit CRC code for K < 20 and a 11-bit CRC for K ≥ 20 #OTHEREFR .

Note that AED uses random automorphisms from the full affine automorphism group of each code, which may be larger than the specified minimum symmetry in the code design. Fig.

2 shows the required signal-to-noise-ratio (SNR) to reach a BLER of 10 −3 .

As we can see, across all values of N and rates considered, the proposed codes using AE-SC perform similarly to the standardized codes using CA-SCL, and outperform it for shorter blocklengths.

However, for longer blocklengths (N ≥ 512), AED falls behind SCL, but still remains within 1 dB of the 5G polar code.

Rate-Compatible Polar Codes For Automorphism Ensemble Decoding

Abstract: Recently, automorphism ensemble decoding (AED) has drawn research interest as a more computationally efficient alternative to successive cancellation list (SCL) decoding of polar codes. Although AED has demonstrated superior performance for specific code parameters, a flexible code design that can accommodate varying code rates does not yet exist. This work proposes a theoretical framework for constructing ratecompatible polar codes with a prescribed automorphism group, which is a key requirement for AED. We first prove that a one-bit granular sequence with useful automorphisms cannot exist. However, by allowing larger steps in the code dimension, flexible code sequences can be constructed. An explicit synthetic channel ranking based on the β-expansion is then proposed to ensure that all constructed codes possess the desired symmetries. Simulation results, covering a broad range of code dimensions and blocklengths, show a performance comparable to that of 5G polar codes under cyclic redundancy check (CRC)-aided SCL decoding, however, with lower complexity.

Coding In The Finite-Blocklength Regime: Bounds Based On Laplace Integrals And Their Asymptotic Approximations

Abstract: In this paper, we provide new compact integral expressions and associated simple asymptotic approximations for converse and achievability bounds in the finite blocklength regime. The chosen converse and random coding union bounds were taken from the recent work of Polyanskyi–Poor–Verdu, and are investigated under parallel AWGN channels, the AWGN channel, the BI-AWGN channel, and the BSC. The technique we use, which is a generalization of some recent results available from the literature, is to map the probabilities of interest into a Laplace integral, and then solve (or approximate) the integral by use of a steepest descent technique. The proposed results are particularly useful for short packet lengths, where the normal approximation may provide unreliable results.


Moreover, algorithms with improved approximation guarantees can be obtained that is FPT in k and #REFR .

Coreset construction was popularized by an initial set of works that obtained small-sized coresets in low-dimensional Euclidean spaces #OTHEREFR .

Chen #OTHEREFR obtained the first coreset for Euclidean spaces with polynomial dependence on the dimension and the first coreset in general metrics, where the size is O(k 2 −2 log 2 n) for k-median.

Subsequently, the dependence on the dimension has been further improved #OTHEREFR . Finally, such dependence was removed in #OTHEREFR .

See also #OTHEREFR for recent improvements.

Both k-median and k-means admit polynomial-time O(1)-approximations in general metrics #OTHEREFR .

Naturally, the problems have also been studied in specialized metrics.

Polynomial-time approximation schemes (PTASes) are known for Euclidean k-median #OTHEREFR and kmeans #OTHEREFR . See #OTHEREFR for other improvements.

Similar to geometric clustering, clustering in graphic setting is also widely studied. PTASes are known for excluded-minor graphs #OTHEREFR .

Also, FPT approximation schemes are known for graphs of bounded-treewidth #OTHEREFR and graphs of bounded highway dimension #OTHEREFR .

Coresets For Clustering In Geometric Intersection Graphs

Abstract: Designing coresets-small-space sketches of the data preserving cost of the solutions within (1 ±)-approximate factor-is an important research direction in the study of center-based kclustering problems, such as k-means or k-median. Feldman and Langberg [STOC'11] have shown that for k-clustering of n points in general metrics, it is possible to obtain coresets whose size depends logarithmically in n. Moreover, such a dependency in n is inevitable in general metrics. A significant amount of recent work in the area is devoted to obtaining coresests whose sizes are independent of n for special metrics, like d-dimensional Euclidean space [Huang,

Tight Fpt Approximations For $K$-Median And $K$-Means

Abstract: We investigate the fine-grained complexity of approximating the classical k-Median/k-Means clustering problems in general metric spaces. We show how to improve the approximation factors to (1+2/e+epsilon) and (1+8/e+epsilon) respectively, using algorithms that run in fixed-parameter time. Moreover, we show that we cannot do better in FPT time, modulo recent complexity-theoretic conjectures.


Following this work, BigNAS #REFR introduces several technologies to train a high-quality single-stage model, whose child models can be directly deployed without extra retraining or post-processing steps.

One-shot NAS is an important branch of neural architecture search (NAS) #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR that aims to drastically reduce the training and search time for the whole pipeline.

Along this direction, #OTHEREFR trains a once-for-all (OFA) network with progressive shrinking and knowledge distillation to support all kinds of architectural settings.

Both OFA and BigNAS aim at simultaneously training and searching various networks with different resolutions, depths, widths and operations.

Differently, the proposed method aims at improving a given residual network and can be seamlessly applied to the searched model of NAS.

In addition, as OFA and BigNAS are not designed to solve the loafing problem and improve the main network results, their subnet input designs, sampling rules, and supervision signals are all different from the proposed method.

Stimulative Training++: Go Beyond The Performance Limits Of Residual Networks

Abstract: Residual networks have shown great success and become indispensable in recent deep neural network models. In this work, we aim to re-investigate the training process of residual networks from a novel social psychology perspective of loafing, and further propose a new training scheme as well as three improved strategies for boosting residual networks beyond their performance limits. Previous research has suggested that residual networks can be considered as ensembles of shallow networks, which implies that the final performance of a residual network is influenced by a group of subnetworks. We identify a previously overlooked problem that is analogous to social loafing, where subnetworks within a residual network are prone to exert less effort when working as part of a group compared to working alone. We define this problem as network loafing. Similar to the decreased individual productivity and overall performance as demonstrated in society, network loafing inevitably causes sub-par performance. Inspired by solutions from social psychology, we first propose a novel training scheme called stimulative training, which randomly samples a residual subnetwork and calculates the KL divergence loss between the sampled subnetwork and the given residual network for extra supervision. In order to unleash the potential of stimulative training, we further propose three simple-yet-effective strategies, including a novel KL-loss that only aligns the network logits direction, random smaller inputs for subnetworks, and inter-stage sampling rules. Comprehensive experiments and analysis verify the effectiveness of stimulative training as well as its three improved strategies. For example, the proposed method can boost the performance of ResNet50 on ImageNet to 80.5% Top1 accuracy without using any extra data, model, trick, or changing the structure. With only uniform augment, the performance can be further improved to 81.0% Top1 accuracy, better than the best training recipes provided by Timm library and PyTorch official version. We also verify its superiority on various typical models, datasets, and tasks and give some theoretical analysis. As such, we advocate utilizing the proposed method as a general and next-generation technology to train residual networks. The code is available at https://github.com/Sunshine-Ye/NIPS22-ST.

Bignas: Scaling Up Neural Architecture Search With Big Single-Stage Models

Abstract: Neural architecture search (NAS) has shown promising results discovering models that are both accurate and fast. For NAS, training a one-shot model has become a popular strategy to rank the relative quality of different architectures (child models) using a single set of shared weights. However, while one-shot model weights can effectively rank different network architectures, the absolute accuracies from these shared weights are typically far below those obtained from stand-alone training. To compensate, existing methods assume that the weights must be retrained, finetuned, or otherwise post-processed after the search is completed. These steps significantly increase the compute requirements and complexity of the architecture search and model deployment. In this work, we propose BigNAS, an approach that challenges the conventional wisdom that post-processing of the weights is necessary to get good prediction accuracies. Without extra retraining or post-processing steps, we are able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs. Our discovered model family, BigNASModels, achieve top-1 accuracies ranging from 76.5% to 80.9%, surpassing state-of-the-art models in this range including EfficientNets and Once-for-All networks without extra retraining or post-processing. We present ablative study and analysis to further understand the proposed BigNASModels.


We use BART #REFR , a sequence-to-sequence pretrained Transformer, to initialize both the encoder and decoder.

The self-attention and cross-attention layers can then be written as:

where h enc represents the final encoder hidden states.

Based on the self-attention and cross-attention layers, the encoder (ENC) and decoder (DEC) layers can be expressed as:

Spec: Summary Preference Decomposition For Low-Resource Abstractive Summarization

Abstract: Neural abstractive summarization has been widely studied and achieved great success with large-scale corpora. However, the considerable cost of annotating data motivates the need for learning strategies under low-resource settings. In this paper, we investigate the problems of learning summarizers with only few examples and propose corresponding methods for improvements. First, typical transfer learning methods are prone to be affected by data properties and learning objectives in the pretext tasks. Therefore, based on pretrained language models, we further present a meta learning framework to transfer few-shot learning processes from source corpora to the target corpus. Second, previous methods learn from training examples without decomposing the content and preference. The generated summaries could therefore be constrained by the preference bias in the training set, especially under low-resource settings. As such, we propose decomposing the contents and preferences during learning through the parameter modulation, which enables control over preferences during inference. Third, given a target application, specifying required preferences could be non-trivial because the preferences may be difficult to derive through observations. Therefore, we propose a novel decoding method to automatically estimate suitable preferences and generate corresponding summary candidates from the few training examples. Extensive experiments demonstrate that our methods achieve state-of-the-art performance on six diverse corpora with 30.11%/33.95%/27.51% and 26.74%/31.14%/24.48% average improvements on ROUGE-1/2/L under 10-and 100example settings.

Bart: Denoising Sequence-To-Sequence Pre-Training For Natural Language Generation, Translation, And Comprehension

Abstract: We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and many other more recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of the original sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa with comparable training resources on GLUE and SQuAD, achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 6 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also report ablation experiments that replicate other pretraining schemes within the BART framework, to better measure which factors most influence end-task performance.


According to the DOTA #REFR dataset annotation format, a line in the text file corresponding to the annotation of a rotated table instance is shown 2.

Point A represents the top-left point of the table, and points ABCD are arranged clockwise.

Point D represents the detection difficulty of the sample, which is uniformly defined as 0 in TRR360D.

In this format, x1, y1, x2, y2, x3, y3, x4, and y4 represent the x and y coordinates of the four points A, B, C, and D, respectively.

The last column represents the detection difficulty, which is set to 0 for all samples in TRR360D.

Trr360D: A Dataset For 360 Degree Rotated Rectangular Box Table Detection

Abstract: To address the problem of scarcity and high annotation costs of rotated image table detection datasets, this paper proposes a method for building a rotated image table detection dataset. Based on the ICDAR2019MTD modern table detection dataset, we refer to the annotation format of the DOTA dataset to create the TRR360D rotated table detection dataset. The training set contains 600 rotated images and 977 annotated instances, and the test set contains 240 rotated images and 499 annotated instances. The AP50(T<90) evaluation metric is defined, and this dataset is available for future researchers to study rotated table detection algorithms and promote the development of table detection technology. The TRR360D rotated table detection dataset was created by constraining the starting point and annotation direction, and is publicly available at https://github.com/vansin/ TRR360D.

Dota: A Large-Scale Dataset For Object Detection In Aerial Images

Abstract: Object detection is an important and challenging problem in computer vision. Although the past decade has witnessed major advances in object detection in natural scenes, such successes have been slow to aerial imagery, not only because of the huge variation in the scale, orientation and shape of the object instances on the earth's surface, but also due to the scarcity of well-annotated datasets of objects in aerial scenes. To advance object detection research in Earth Vision, also known as Earth Observation and Remote Sensing, we introduce a large-scale Dataset for Object deTection in Aerial images (DOTA). To this end, we collect $2806$ aerial images from different sensors and platforms. Each image is of the size about 4000-by-4000 pixels and contains objects exhibiting a wide variety of scales, orientations, and shapes. These DOTA images are then annotated by experts in aerial image interpretation using $15$ common object categories. The fully annotated DOTA images contains $188,282$ instances, each of which is labeled by an arbitrary (8 d.o.f.) quadrilateral To build a baseline for object detection in Earth Vision, we evaluate state-of-the-art object detection algorithms on DOTA. Experiments demonstrate that DOTA well represents real Earth Vision applications and are quite challenging.

rotated table instance
DOTA dataset annotation


As all transactions are globally shared in GeoBFT #REFR , each cluster has access to all transactions, making cross-cluster transactions cheap.

There are two broad approaches to this: either with the actual transactions #OTHEREFR , #OTHEREFR - #OTHEREFR , or with "prepackaged" blocks of transactions #OTHEREFR , #OTHEREFR arranged in the upper layer.

For example, in GeoBFT #OTHEREFR or DP-Hybrid #OTHEREFR , every transaction is shared through the upper layer with the other groups.

In C-PBFT #OTHEREFR , on the other hand, the lower layer already constructs blocks of transactions, which are then confirmed by adding the block header to the upper-layer blockchain.

As with sharding, it can happen that some transaction requires data from multiple groups.

For this, affected groups can coordinate with their least common ancestor group #OTHEREFR .

As global sharing is performed before execution, this leads to higher latencies.

Different layers can have different failure modes #OTHEREFR , #OTHEREFR , e. g., when different applications use a shared blockchain. This can be used to optimize the system, e.

g., if the applications require different security or fault tolerance levels.

Then it might be possible to use different consensus protocols in separate lower layers, such as BFT or crash fault-tolerant protocols.

Further, there are protocols that measure the behavior of nodes and reward good behavior, e.

Sok: Scalability Techniques For Bft Consensus

Abstract: With the advancement of blockchain systems, many recent research works have proposed distributed ledger technology (DLT) that employs Byzantine fault-tolerant (BFT) consensus protocols to decide which block to append next to the ledger. Notably, BFT consensus can offer high performance, energy efficiency, and provable correctness properties, and it is thus considered a promising building block for creating highly resilient and performant blockchain infrastructures. Yet, a major ongoing challenge is to make BFT consensus applicable to large-scale environments. A large body of recent work addresses this challenge by developing novel ideas to improve the scalability of BFT consensus, thus opening the path for a new generation of BFT protocols tailored to the needs of blockchain. In this survey, we create a systematization of knowledge about the novel scalability-enhancing techniques that state-of-the-art BFT consensus protocols use. For our comparison, we closely analyze the efforts, assumptions, and trade-offs these protocols make.

Resilientdb: Global Scale Resilient Blockchain Fabric

Abstract: Recent developments in blockchain technology have inspired innovative new designs in distributed resilient database systems. At their core, these database systems typically use Byzantine fault-tolerant consensus protocols to maintain a common state across all replicas, even if some replicas are faulty or malicious. Unfortunately, existing consensus protocols are not designed to deal with geo-scale deployments in which many replicas spread across a geographically large area participate in consensus. To address this, we present the Geo-Scale Byzantine Fault-Tolerant consensus protocol (GeoBFT). GeoBFT is designed for excellent scalability by using a topological-aware grouping of replicas in local clusters, by introducing parallelization of consensus at the local level, and by minimizing communication between clusters. To validate our vision of high-performance geo-scale resilient database systems using GeoBFT, we implement GeoBFT in our efficient ResilientDB blockchain and database fabric. We show that GeoBFT is not only sound and provides great scalability, but also outperforms state-of-the-art consensus protocols by a factor of six in geo-scale deployments.


We modify the encoder of the U-Net #REFR , and design our DAIF-Net, as shown in Figure 4 .

Taking advantage that the defocus images in a focal stack are sharing the same depth and AIF image.

Thus, we can train our model on a focal stack to make this task solvable, and our DAIF-Net has the following expression:

where that the accurate depth map and AIF image is the only solution to this function.

Consequently, from the aspect of model architecture, the DAIF-Net is supposed to take a focal stack with an arbitrary number of images and output the underlying AIF image and depth map.

In order to support a flexible number of input branches, every image will be passed through the same encoder and bottleneck.

In order to learn the depth and AIF image from a focal stack simultaneously, we observe that sharpness is the connection between them.

Intuitively, a sharper point indicates a closer distance between the depth and the focus distance.

Meanwhile, sharper points better preserve the corresponding colors of the AIF images.

Therefore, training a model that is sensitive to the sharpest points in the focal stacks is beneficial to predicting both AIF images and depth maps.

Fully Self-Supervised Depth Estimation From Defocus Clue

Abstract: Depth-from-defocus (DFD), modeling the relationship between depth and defocus pattern in images, has demonstrated promising performance in depth estimation. Recently, several self-supervised works try to overcome the difficulties in acquiring accurate depth ground-truth. However, they depend on the all-in-focus (AIF) images, which cannot be captured in real-world scenarios. Such limitation discourages the applications of DFD methods. To tackle this issue, we propose a completely self-supervised framework that estimates depth purely from a sparse focal stack. We show that our framework circumvents the needs for the depth and AIF image ground-truth, and receives superior predictions, thus closing the gap between the theoretical success of DFD works and their applications in the real world. In particular, we propose (i) a more realistic setting for DFD tasks, where no depth or AIF image ground-truth is available; (ii) a novel selfsupervision framework that provides reliable predictions of depth and AIF image under the challenging setting. The proposed framework uses a neural model to predict the depth and AIF image, and utilizes an optical model to validate and refine the prediction. We verify our framework on three benchmark datasets with rendered focal stacks and real focal stacks. Qualitative and quantitative evaluations show that our method provides a strong baseline for selfsupervised DFD tasks. The source code is publicly available at https://github.com/Ehzoahis/DEReD.

U-Net: Convolutional Networks For Biomedical Image Segmentation

Abstract: There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .


Background And Related Work 2.1 Large Language Model

Recently, researchers have begun adopting LLMs for code-related tasks, e.g., via further fine-tuning LLMs on code snippets from open-source repositories #REFR .

Large Language Models (LLMs) have demonstrated impressive performance across a wide range of NLP tasks by training on large corpora of text data scraped from the Internet #OTHEREFR .

LLMs can be classified based on variations of the popular Transformer architecture #OTHEREFR into: Encoder-only, Decoder-only and Encoder-Decoder models.

Decoder-only LLMs (e.g., GPT #OTHEREFR 48] , Codex [11] and CodeGen #OTHEREFR ) focus on autoregressive completion tasks by learning to predict the probability of the next token given previously generated tokens.

Encoder-only (e.g., CodeBERT #OTHEREFR and GraphCodeBERT #OTHEREFR ) and Encoder-Decoder (e.g., CodeT5 #OTHEREFR and PLBART #OTHEREFR ) models on the other hand are designed for infilling tasks, where the pre-training objective is to recover masked-out tokens or token spans in the training data by using bi-directional context.

In order to apply LLMs for down-stream applications, there are two main paradigms: Fine-tuning #OTHEREFR and In-context learning #OTHEREFR .

Fine-tuning is the process of updating the model weights by learning the desired output from the given input of a specific downstream task dataset.

Large Language Models Are Edge-Case Fuzzers: Testing Deep Learning Libraries Via Fuzzgpt

Abstract: Bugs in Deep Learning (DL) libraries may affect almost all downstream DL applications, and it is crucial to ensure the quality of such systems. It is challenging to generate valid input programs for fuzzing DL libraries, since the input programs need to satisfy both the syntax/semantics of the supported languages (e.g., Python) and the tensor/operator constraints for constructing valid computational graphs. Recently, the TitanFuzz work demonstrates that modern Large Language Models (LLMs) can be directly leveraged to implicitly learn all the language and DL computation constraints to generate valid programs for fuzzing DL libraries. However, LLMs tend to generate ordinary programs following similar patterns/tokens with typical programs seen in their massive training corpora (e.g., GitHub), while fuzzing favors unusual inputs that cover edge cases or are unlikely to be manually produced. To fill this gap, this paper proposes FuzzGPT, the first technique to prime LLMs to synthesize unusual programs for fuzzing. FuzzGPT is built on the well-known hypothesis that historical bug-triggering programs may include rare/valuable code ingredients important for bug finding. Meanwhile, while traditional techniques leveraging such historical information require intensive human efforts to both design dedicated generators and ensure the syntactic/semantic validity of generated programs, FuzzGPT demonstrates that this process can be fully automated via the intrinsic capabilities of LLMs (including fine-tuning and in-context learning), while being generalizable and applicable to challenging domains. While FuzzGPT can be applied with different LLMs, this paper focuses on the powerful GPT-style models: Codex and CodeGen. Moreover, FuzzGPT also shows the potential of directly leveraging the instruct-following capability of the recent ChatGPT for effective fuzzing. The experimental study on two popular DL libraries (PyTorch and TensorFlow) shows that FuzzGPT can substantially outperform TitanFuzz, detecting 76 bugs, with 49 already confirmed as previously unknown bugs, including 11 high-priority bugs or security vulnerabilities.

Codebert: A Pre-Trained Model For Programming And Natural Languages

Abstract: We present CodeBERT, a bimodal pre-trained model for programming language (PL) and nat-ural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language codesearch, code documentation generation, etc. We develop CodeBERT with Transformer-based neural architecture, and train it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us to utilize both bimodal data of NL-PL pairs and unimodal data, where the former provides input tokens for model training while the latter helps to learn better generators. We evaluate CodeBERT on two NL-PL applications by fine-tuning model parameters. Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation tasks. Furthermore, to investigate what type of knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and evaluate in a zero-shot setting where parameters of pre-trained models are fixed. Results show that CodeBERT performs better than previous pre-trained models on NL-PL probing.


The work of #REFR follows a similar paradigm, but randomly selects one non-empty dimension from s dimensions.

It remains an open question whether it holds the same bounds as the general case or has tighter bounds.

Recently, for a broad family of -LDP estimation problems that can be cast as mean estimation problems, #OTHEREFR studies sample complexity lower bounds under certain error tolerance α, but their sample complexity results for s-sparse numerical vectors exhibit at least a 1/α gap compared to our minimax optimal sample complexity results.

In practice, numerous -LDP mechanisms have been proposed for statistical estimation, such as multinomial distribution estimation on categorical data #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR and one-dimensional mean estimation on numerical values #OTHEREFR , #OTHEREFR .

For -LDP numerical vector or key-value data aggregation, existing approaches handle both dense numerical vectors (e.g., in #OTHEREFR , #OTHEREFR , #OTHEREFR ) and sparse numerical vectors (e.g., in #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR ).

Specifically, #OTHEREFR , #OTHEREFR uniformly and randomly select one dimension from [d] and transform the multi-dimensional estimation problem to a one-dimensional numerical/categorical problem.

However, as we will show in Section 3, these mechanisms are sub-optimal.

To mitigate the high noise needed for LDP, #OTHEREFR , #OTHEREFR introduce a (semi-trusted) shuffler to hide private views in the crowd.

The seminal work #OTHEREFR shows that n shuffled -LDP views can preserve (O( log(1/δ)/n), δ)-differential privacy.

The work #OTHEREFR derives a similar conclusion specifically for binary randomized response messages.

A later work #OTHEREFR considers private views from other users as a "privacy blanket" and derives tighter privacy amplification bounds (O(min{ 0 , 1}e 0 log(1/δ)/n), δ).

Differentially Private Numerical Vector Analyses In The Local And Shuffle Model

Abstract: Numerical vector aggregation plays a crucial role in privacy-sensitive applications, such as distributed gradient estimation in federated learning and statistical analysis of key-value data. In the context of local differential privacy, this study provides a tight minimax error bound of O(ds n 2), where d represents the dimension of the numerical vector and s denotes the number of non-zero entries. By converting the conditional/unconditional numerical mean estimation problem into a frequency estimation problem, we develop an optimal and efficient mechanism called Collision. In contrast, existing methods exhibit sub-optimal error rates of O(d 2 n 2) or O(ds 2 n 2). Specifically, for unconditional mean estimation, we leverage the negative correlation between two frequencies in each dimension and propose the CoCo mechanism, which further reduces estimation errors for mean values compared to Collision. Moreover, to surpass the error barrier in local privacy, we examine privacy amplification in the shuffle model for the proposed mechanisms and derive precisely tight amplification bounds. Our experiments validate and compare our mechanisms with existing approaches, demonstrating significant error reductions for frequency estimation and mean estimation on numerical vectors.

Pckv: Locally Differentially Private Correlated Key-Value Data Collection With Optimized Utility

Abstract: Data collection under local differential privacy (LDP) has been mostly studied for homogeneous data. Real-world applications often involve a mixture of different data types such as key-value pairs, where the frequency of keys and mean of values under each key must be estimated simultaneously. For key-value data collection with LDP, it is challenging to achieve a good utility-privacy tradeoff since the data contains two dimensions and a user may possess multiple key-value pairs. There is also an inherent correlation between key and values which if not harnessed, will lead to poor utility. In this paper, we propose a locally differentially private key-value data collection framework that utilizes correlated perturbations to enhance utility. We instantiate our framework by two protocols PCKV-UE (based on Unary Encoding) and PCKV-GRR (based on Generalized Randomized Response), where we design an advanced Padding-and-Sampling mechanism and an improved mean estimator which is non-interactive. Due to our correlated key and value perturbation mechanisms, the composed privacy budget is shown to be less than that of independent perturbation of key and value, which enables us to further optimize the perturbation parameters via budget allocation. Experimental results on both synthetic and real-world datasets show that our proposed protocols achieve better utility for both frequency and mean estimations under the same LDP guarantees than state-of-the-art mechanisms.


For the validation method, we opted for a stratified 10-fold cross-validation with an MCC score metric since k-fold cross-validation segments the training data set in folds, each containing approximately the same distribution of class labels as in the training data set #REFR .

The wallclock-time represents the time in seconds between starting the SMAC optimisation and terminating it.

In order to find the wallclock-time SMAC needs to find near-optimal hyperparameter values, we proceeded as follows: First, we split the entire data set into 67% training and 33% test set, and then, with ["linear", "poly", "rbf", "sigmoid"]

Categorical increment-steps of 25 seconds, we performed 15 independent optimisation runs per wallclock-time on the pipeline configuration yielding the most hyperparameters (i.e., noise removal with an RF model).

This method resulted in a wallclock-time of 500 seconds at which the highest performance reflected in the MCC score was reached. This optimal value is adopted in the bootstrapping protocol.

A single pipeline optimization within the bootstrapping protocol #OTHEREFR consists of the following steps: First, we split the entire data set again into a 67% training and 33% test set after which we exclusively utilise the training set to perform the optimisation with SMAC.

Given that SMAC targets HPO as a minimisation problem #OTHEREFR , we aggregated over the 10 resulting performances and subtracted the outcome from 1 as an indicator of the respective pipeline configuration performance.

In this respect, we let SMAC optimise a single pipeline configuration 15 times on the training data set.

Next, from the 15 generated ones, we randomly sample 5 incumbents from which we select the one yielding the lowest performance score.

We repeated this process 50 times for every pipeline configuration.

Subsequently, we retrained 50 models according to the 50 best-found incumbents on the training data set.

Automated Classification Of Pre-Defined Movement Patterns: A Comparison Between Gnss And Uwb Technology

Abstract: Advanced real-time location systems (RTLS) allow for collecting spatio-temporal data from human movement behaviours. Tracking individuals in small areas such as schoolyards or nursing homes might impose difficulties for RTLS in terms of positioning accuracy. However, to date, few studies have investigated the performance of different localisation systems regarding the classification of human movement patterns in small areas. The current study aims to design and evaluate an automated framework to classify human movement trajectories obtained from two different RTLS: Global Navigation Satellite System (GNSS) and Ultra-wideband (UWB), in areas of approximately 100 square meters. Specifically, we designed a versatile framework which takes GNSS or UWB data as input, extracts features from these data and classifies them according to the annotated spatial patterns. The automated framework contains three choices for applying noise removal: (i) no noise removal, (ii) Savitzky Golay filter on the raw location data or (iii) Savitzky Golay filter on the extracted features, as well as three choices regarding the classification algorithm: Decision Tree (DT), Random Forest (RF) or Support Vector Machine (SVM). We integrated different stages within the framework with the Sequential Model-Based Algorithm Configuration (SMAC) to perform automated hyperparameter optimisation. The best performance is achieved with a pipeline consisting of noise removal applied to the raw location data with an RF model for the GNSS and no noise removal with an SVM model for the UWB. We further demonstrate through statistical analysis that the UWB achieves significantly higher results than the GNSS in classifying movement patterns.

Scikit-Learn: Machine Learning In Python

Abstract: Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing machine learning to non-specialists using a general-purpose high-level language. Emphasis is put on ease of use, performance, documentation, and API consistency. It has minimal dependencies and is distributed under the simplified BSD license, encouraging its use in both academic and commercial settings. Source code, binaries, and documentation can be downloaded from http://scikit-learn.sourceforge.net.


Scordelis-Lo Roof With Four Holes Under Single Load

The deformation plot has its maximum and minimum at u z,min ≈ −3.3 • 10 −1 and u z,max ≈ 3.7 • 10 −1 , respectively. These results are also obtained in #REFR .

25 with the trimming curves shown in red and the point load applied in the center in green.

The results of the model can be compared to deformation plots presented in #OTHEREFR Figure 11 (a) ], which proposes an IGA shell with adaptive refinement. Fig.

18 shows the deformed structure with the displacement in z-direction for the SB-IGA approach of p = 3, r = 1 and h = 0.125.

u z Figure 18 : Deformed Scordelis-Lo roof with four holes with coloring of the z-displacement obtained for p = 3, h = 1/6, and r = 1.

The results point out, that the approach derived herein is capable of computing the deformation for trimmed domains of shells.

Good agreement is visible on the whole domain including the area of the load application and the free edges.

Smooth Multi-Patch Scaled Boundary Isogeometric Analysis For Kirchhoff-Love Shells

Abstract: In this work, a linear Kirchhoff-Love shell formulation in the framework of scaled boundary isogeometric analysis is presented that aims to provide a simple approach to trimming for NURBS-based shell analysis. To obtain a global C 1-regular test function space for the shell discretization, an inter-patch coupling is applied with adjusted basis functions in the vicinity of the scaling center to ensure the approximation ability. Doing so, the scaled boundary geometries are related to the concept of analysis-suitable G 1 parametrizations.This yields a coupling of patch boundaries in a strong sense that is restricted to G 1-smooth surfaces. The proposed approach is advantageous to trimmed geometries due to the incorporation of the trimming curve in the boundary representation that provides an exact representation in the planar domain. The potential of the approach is demonstrated by several problems of untrimmed and trimmed geometries of Kirchhoff-Love shell analysis evaluated against error norms and displacements. Lastly, the applicability is highlighted in the analysis of a violin structure including arbitrarily shaped patches.

Adaptive Isogeometric Analysis On Two-Dimensional Trimmed Domains Based On A Hierarchical Approach

Abstract: The focus of this work is on the development of an error-driven isogeometric framework, capable of automatically performing an adaptive simulation in the context of second- and fourth-order, elliptic partial differential equations defined on two-dimensional trimmed domains. The method is steered by an a posteriori error estimator, which is computed with the aid of an auxiliary residual-like problem formulated onto a space spanned by splines with single element support. The local refinement of the basis is achieved thanks to the use of truncated hierarchical B-splines. We prove numerically the applicability of the proposed estimator to various engineering-relevant problems, namely the Poisson problem, linear elasticity and Kirchhoff-Love shells, formulated on trimmed geometries. In particular, we study several benchmark problems which exhibit both smooth and singular solutions, where we recover optimal asymptotic rates of convergence for the error measured in the energy norm and we observe a substantial increase in accuracy per-degree-of-freedom compared to uniform refinement. Lastly, we show the applicability of our framework to the adaptive shell analysis of an industrial-like trimmed geometry modeled in the commercial software Rhinoceros, which represents the B-pillar of a car.


However, due to the 0 -norm constraint, exactly solving the non-convex and non-differentiable problem (9) is NP-hard #REFR .

where A and y are obtained from A and y by normalizing each column, i.e., the column norms are 1, and

represents the number of groups with non-zero coefficients.

The constraint enforces group sparsity by explicitly specifying that only k groups of features have nonzero coefficients.

The solution of (9) corresponds to a PDE model with exactly k features that best fits the given data.

We propose Group Projected Subspace Pursuit (GPSP) to find a group k-sparse vector c * (k) for k = 1, 2, . . . , K max .

Given a fixed k, the proposed GPSP iteratively searches for k groups highly correlated to the residuals in a greedy manner (See Section 3).

For each sparsity level k, we denote the index set corresponding to the active group features by T (k) ⊆ {1, . . . , G}.

We obtain K max candidate PDEs whose active features are indexed by T (k) for each sparsity level k = 1, . . . , K max respectively.

Group Projected Subspace Pursuit For Identification Of Variable Coefficient Differential Equations (Gp-Ident)

Abstract: We propose an effective and robust algorithm for identifying partial differential equations (PDEs) with space-time varying coefficients from a single trajectory of noisy observations. Identifying unknown differential equations from noisy observations is a difficult task, and it is even more challenging with space and time varying coefficients in the PDE. The proposed algorithm, GP-IDENT, has three ingredients: (i) we use B-spline bases to express the unknown space and time varying coefficients, (ii) we propose Group Projected Subspace Pursuit (GPSP) to find a sequence of candidate PDEs with various levels of complexity, and (iii) we propose a new criterion for model selection using the Reduction in Residual (RR) to choose an optimal one among the pool of candidates. The new GPSP considers group projected subspaces which is more robust than existing methods in distinguishing correlated group features. We test GP-IDENT on a variety of PDEs and PDE systems, and compare it with the state-of-the-art parametric PDE identification algorithms under different settings to illustrate its outstanding performance. Our experiments show that GP-IDENT is effective in identifying the correct terms from a large dictionary and the model selection scheme is robust to noise.

Subspace Pursuit For Compressive Sensing Signal Reconstruction

Abstract: We propose a new method for reconstruction of sparse signals with and without noisy perturbations, termed the subspace pursuit algorithm. The algorithm has two important characteristics: low computational complexity, comparable to that of orthogonal matching pursuit techniques when applied to very sparse signals, and reconstruction accuracy of the same order as that of linear programming (LP) optimization methods. The presented analysis shows that in the noiseless setting, the proposed algorithm can exactly reconstruct arbitrary sparse signals provided that the sensing matrix satisfies the restricted isometry property with a constant parameter. In the noisy setting and in the case that the signal is not exactly sparse, it can be shown that the mean-squared error of the reconstruction is upper-bounded by constant multiples of the measurement and signal perturbation energies.


Global perturbation means reinitializing the currently designed algorithm with a certain probability to escape from local optima #REFR .

The iterative local search #OTHEREFR was utilized as the design strategy in #OTHEREFR .

It is often used to search the design space with algorithmic operators. Its key steps are local improvement and global perturbation.

Here, local improvement refers to changing only one dimension of the representation of the currently designed algorithm at each iteration.

A local optima is reached once the change in this dimension cannot bring algorithm performance improvement.

Apart from the iterative local search, SA #OTHEREFR was also employed as the design strategy #OTHEREFR .

The workflow of local search for metaheuristic algorithm design is shown in Fig. 5 . 4.1.2 Evolutionary Search.

The evolutionary search #OTHEREFR serves as the design strategy in the GP-based algorithm design literature.

It uses crossover and mutation to globally search the design space.

The sub-tree crossover and mutation are common for the GP-based design with tree representation #OTHEREFR .

A Survey On Automated Design Of Metaheuristic Algorithms

Abstract: Metaheuristic algorithms have attracted wide attention from academia and industry due to their capability of conducting search independent of problem structures and problem domains. Often, human experts are requested to manually tailor algorithms to fit for solving a targeted problem. The manual tailoring process may be laborious, error-prone, and require intensive specialized knowledge. This gives rise to increasing interests and demands for automated design of metaheuristic algorithms with less human intervention. The automated design could make high-performance algorithms accessible to a much broader range of researchers and practitioners; and by leveraging computing power to fully explore the potential design choices, automated design could reach or even surpass human-level design. This paper presents a broad picture of the formalization, methodologies, challenges, and research trends of automated design of metaheuristic algorithms, by conducting a survey on the common grounds and representative techniques in this field. In the survey, we first present the concept of automated design of metaheuristic algorithms and provide a taxonomy by abstracting the automated design process into four parts, i.e., design space, design strategies, performance evaluation strategies, and targeted problems. Then, we overview the techniques concerning the four parts of the taxonomy and discuss their strengths, weaknesses, challenges, and usability, respectively. Finally, we present research trends in this field. CCS Concepts: • Computing methodologies → Search methodologies.

Paramils: An Automatic Algorithm Configuration Framework

Abstract: The identification of performance-optimizing parameter settings is an important part of the development and application of algorithms. We describe an automatic framework for this algorithm configuration problem. More formally, we provide methods for optimizing a target algorithm's performance on a given class of problem instances by varying a set of ordinal and/or categorical parameters. We review a family of local-search-based algorithm configuration procedures and present novel techniques for accelerating them by adaptively limiting the time spent for evaluating individual configurations. We describe the results of a comprehensive experimental evaluation of our methods, based on the configuration of prominent complete and incomplete algorithms for SAT. We also present what is, to our knowledge, the first published work on automatically configuring the CPLEX mixed integer programming solver. All the algorithms we considered had default parameter settings that were manually identified with considerable effort. Nevertheless, using our automated algorithm configuration procedures, we achieved substantial and consistent performance improvements.


Analysis of Group Quantities: The cross-layer weight sharing technique used in ALBert #REFR simply shares the weights of all layers in the model.

Moreover, the proposed multiplexing technique has also been applied in baseline Transformer model to verify its impact on the encoder and decoder, respectively. The experimental results are shown in TABLE VI.

When only applying the multiplexing technique in the encoder, the CER can be improved by 0.1% over the baseline Transformer on the Test dataset with a model size of 17.21M and a 43% compression.

On the other hand, 0.1% CER improvement over the baseline Transformer on the Dev dataset with a model size of 22.47M and a 26% compression can be obtained if only the decoder has been applied the multiplexing technique.

Implementation of the weight reuse technique in both the encoder and decoder can compress the model size to 9.32M and improve 0.1% CER on the Dev dataset.

Therefore, it is logic to say that the model can be compressed with no sacrifice of the model performance regardless of the implementation of the multiplexing technique in either encoder or decoder.

A similar technique is also adopted in Universal Transformer #OTHEREFR and MiniVit #OTHEREFR .

Different from these two approaches, the Sim-T proposed in this study divides all encoder layers into multiple groups, and the proposed multiplexing technique is used in each group.

For It should be highlighted that the model performance degrades with the decreasing number of encoder groups.

Also, the model size gradually decreases from 9.32M to 7.99M with the increasing number of encoder groups.

From the table, one can note that the model performance is undesirable when the encoder group is one, although a greater degree of model compression can be obtained.

Sim-T: Simplify The Transformer Network By Multiplexing Technique For Speech Recognition

Abstract: In recent years, a great deal of attention has been paid to the Transformer network for speech recognition tasks due to its excellent model performance. However, the Transformer network always involves heavy computation and large number of parameters, causing serious deployment problems in devices with limited computation sources or storage memory. In this paper, a new lightweight model called Sim-T has been proposed to expand the generality of the Transformer model. Under the help of the newly developed multiplexing technique, the Sim-T can efficiently compress the model with negligible sacrifice on its performance. To be more precise, the proposed technique includes two parts, that are, module weight multiplexing and attention score multiplexing. Moreover, a novel decoder structure has been proposed to facilitate the attention score multiplexing. Extensive experiments have been conducted to validate the effectiveness of Sim-T. In Aishell-1 dataset, when the proposed Sim-T is 48% parameter less than the baseline Transformer, 0.4% CER improvement can be obtained. Alternatively, 69% parameter reduction can be achieved if the Sim-T gives the same performance as the baseline Transformer. With regard to the HKUST and WSJ eval92 datasets, CER and WER will be improved by 0.3% and 0.2%, respectively, when parameters in Sim-T are 40% less than the baseline Transformer.

Albert: A Lite Bert For Self-Supervised Learning Of Language Representations

Abstract: Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations, longer training times, and unexpected model degradation. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and SQuAD benchmarks while having fewer parameters compared to BERT-large.


NovoGrad #REFR optimizer is employed for training across a maximum of 15000 iterations on a batch of 64 randomly sampled channel realizations with early stopping.

Our proposed feedforward UWMMSE architecture is composed of 3 unfolded-WMMSE layers with a 2-layered CV-GCN -shared by the unfolded layers -modeling the function Ψ in #OTHEREFR .

We set the hidden layer of CV-GNN as F = 32 and that of CV-MLP as G = 16. The model consists of 3302 trainable parameters.

The initial learning rate is set to 1 × 10 −2 .

An interesting observation was that while 3 unfolded-layers offered the best performance trade-off in terms of sum-rate and time at inference, the best training performance was achieved with just 1 unfolded layer.

Recalling that our model is flexible insofar as to have different numbers of unfolded-layers at training and inference on account of parameter-sharing, we established a model setup wherein UWMMSE is trained with a single layer and then 2 more layers are appended at inference, which share the learned components.

Experimental results presented in this section demonstrate the effectiveness of this setup.

At inference, we average the achieved sum-rate over 10000 channel realizations for all experiments.

Deep Graph Unfolding For Beamforming In Mu-Mimo Interference Networks

Abstract: We develop an efficient and near-optimal solution for beamforming in multiuser multiple-input-multiple-output single-hop wireless ad-hoc interference networks. Inspired by the weighted minimum mean squared error (WMMSE) method, a classical approach to solving this problem, and the principle of algorithm unfolding, we present unfolded WMMSE (UWMMSE) for MU-MIMO. This method learns a parameterized functional transformation of key WMMSE parameters using graph neural networks (GNNs), where the channel and interference components of a wireless network constitute the underlying graph. These GNNs are trained through gradient descent on a network utility metric using multiple instances of the beamforming problem. Comprehensive experimental analyses illustrate the superiority of UWMMSE over the classical WMMSE and stateof-the-art learning-based methods in terms of performance, generalizability, and robustness.

Stochastic Gradient Methods With Layer-Wise Adaptive Moments For Training Of Deep Networks

Abstract: We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with momentum and Adam or AdamW. Additionally, NovoGrad (1) is robust to the choice of learning rate and weight initialization, (2) works well in a large batch setting, and (3) has two times smaller memory footprint than Adam.


For finding orbits, we used nauty 2.7r3, a software library for computing automorphism groups of graphs by McKay and Piperno #REFR .

This explains why the only solved instances are for m = Z(d) values.

When m = Z(d), the resulting graph is d-regular or almost d-regular, which has an equivalent edge count to the bound we discussed.

From an inspection of the results, we can conclude that our initial approaches are not good enough for decreasing the upper bound found by CPLEX, but they are promising for finding feasible solutions. This might be due to symmetry.

In our next experiment, we test the efficacy of using Orbital Branching with Basic Formulation (Subsection 3.1).

We implemented Orbital Branching using the callback mechanism of CPLEX.

At each branch-and-cut node, we find the variables set to 0 and 1 from CPLEX and update the constraints accordingly.

Then, we construct the updated bipartite graph using binding constraints. The resulting graph (and coloring) is sent to nauty.

For calculating orbits, nauty calculates the automorphism groups utilizing graph isomorphism problem, which is a GI −complete problem #OTHEREFR .

This implies that the problem of calculating orbits is neither known to be NP-complete nor known to be polynomial-time solvable.

For large instances, run time can be as high as 150 seconds, while it is around 0.1 seconds for small instances.

Constructing Extremal Triangle-Free Graphs Using Integer Programming

Abstract: The maximum number of edges in a graph with matching number m and maximum degree d has been determined in [1] and [2], where some extremal graphs have also been provided. Then, a new question has emerged: how the maximum edge count is affected by forbidding some subgraphs occurring in these extremal graphs? In [3], the problem is solved in triangle-free graphs for d ≥ m, and for d < m with either Z(d) ≤ m < 2d or d ≤ 6, where Z(d) is approximately 5d/4. The authors derived structural properties of triangle-free extremal graphs, which allows us to focus on constructing small extremal components to form an extremal graph. Based on these findings, in this paper, we develop an integer programming formulation for constructing extremal graphs. Since our formulation is highly symmetric, we use our own implementation of Orbital Branching to reduce symmetry. We also implement our integer programming formulation so that the feasible region is restricted iteratively. Using a combination of the two approaches, we expand the solution into d ≤ 10 instead of d ≤ 6 for m > d. Our results endorse the formula for the number of edges in all extremal triangle-free graphs conjectured in [3].

Practical Graph Isomorphism, Ii

Abstract: We report the current state of the graph isomorphism problem from the practical point of view. After describing the general principles of the refinement-individualization paradigm and proving its validity, we explain how it is implemented in several of the key programs. In particular, we bring the description of the best known program nauty up to date and describe an innovative approach called Traces that outperforms the competitors for many difficult graph classes. Detailed comparisons against saucy, Bliss and conauto are presented.


We used an initial learning rate of λ = 10 −4 and a weight decay of 10 −4 and trained for 30 (600) epochs for indoor (outdoor) localization. Our loss is initialized as in #REFR .

We follow the training procedure of #OTHEREFR and apply the augmentation method suggested by #OTHEREFR .

Specifically, during training, images are first rescaled such that their smaller edge is resized to 256 pixels.

We extract random 224 × 224 crops and randomly jitter the brightness, saturation, and contrast.

At test time, images are rescaled, and the center crop is used for inference, with no further augmentations.

Our model is optimized using Adam, with β 1 = 0.9, β 2 = 0.999 and = 10 −10 and a batch size of 8.

The training pairs for the 7Scenes dataset are generated as in #OTHEREFR .

We further extend this dataset with pairs from #OTHEREFR .

Pairs from the CambridgeLandmarks dataset are computed by encoding images with a pre-trained NetVLAD model #OTHEREFR , and fetching the nearest 40 neighbors based on cosine similarity.

At train time, we sample from the pool of neighbors to generate different pairs.

We provide the training pairs, code, configuration files, and pre-trained models in order to support the reproducibility of our results.

Learning To Localize In Unseen Scenes With Relative Pose Regressors

Abstract: Relative pose regressors (RPRs) localize a camera by estimating its relative translation and rotation to a poselabelled reference. Unlike scene coordinate regression and absolute pose regression methods, which learn absolute scene parameters, RPRs can (theoretically) localize in unseen environments, since they only learn the residual pose between camera pairs. In practice, however, the performance of RPRs is significantly degraded in unseen scenes. In this work, we propose to aggregate paired feature maps into latent codes, instead of operating on global image descriptors, in order to improve the generalization of RPRs. We implement aggregation with concatenation, projection, and attention operations (Transformer Encoders) and learn to regress the relative pose parameters from the resulting latent codes. We further make use of a recently proposed continuous representation of rotation matrices, which alleviates the limitations of the commonly used quaternions. Compared to stateof-the-art RPRs, our model is shown to localize significantly better in unseen environments, across both indoor and outdoor benchmarks, while maintaining competitive performance in seen scenes. We validate our findings and architecture design through multiple ablations. Our code and pretrained models is publicly available 1 .

Geometric Loss Functions For Camera Pose Regression With Deep Learning

Abstract: Deep learning has shown to be effective for robust and real-time monocular image relocalisation. In particular, PoseNet [22] is a deep convolutional neural network which learns to regress the 6-DOF camera pose from a single image. It learns to localize using high level features and is robust to difficult lighting, motion blur and unknown camera intrinsics, where point based SIFT registration fails. However, it was trained using a naive loss function, with hyper-parameters which require expensive tuning. In this paper, we give the problem a more fundamental theoretical treatment. We explore a number of novel loss functions for learning camera pose which are based on geometry and scene reprojection error. Additionally we show how to automatically learn an optimal weighting to simultaneously regress position and orientation. By leveraging geometry, we demonstrate that our technique significantly improves PoseNets performance across datasets ranging from indoor rooms to a small city.