This method takes as input surface elevation data including buildings and other structures and uses the fluid-inspired method presented in #REFR to generate layers of air corridors.

First, they provide [closelyspaced] smooth paths that offer appropriate clearance from buildings, terrain, other structures, and neighboring UAS.

The consideration environmental maps and safe separation distances significantly simplifies UAS flight planning and dynamic rerouting.

Second, they define safely separated candidate paths that can assure collision avoidance so long as each UAS tracks its planned trajectory to within expected error bounds.

The authors previously presented a method to automatically generate a dense air network for any urban environment that wraps structures #OTHEREFR .

Each layer is a fixedaltitude plane with air corridors oriented in a fixed nominal direction that wrap around buildings and other structures.

This paper presents several advancements to the authors' previous work #OTHEREFR which increases its scalabity and usability.

More specifically, this paper offers the following distinct and novel contributions:

1) The acquisition method for surface elevation data has been fully automated for US cities.

In our previous work #OTHEREFR , three dimensional models for buildings were manually created using Google Maps data which is not scalable.

Can A Laplace Pde Define Air Corridors Through Low-Altitude Airspace?

Abstract: Urban Uncrewed Aircraft System (UAS) flight will require new regulations that assure safety and accommodate unprecedented traffic density levels. Multi-UAS coordination is essential to both objectives. This paper models UAS coordination as an ideal fluid flow with a stream field governed by the Laplace partial differential equation. Streamlines spatially define closely-spaced deconflicted routes through the airspace and define air corridors that safely wrap buildings and other structures so UAS can avoid collision even when flying among low-altitude vertical obstacles and near mountainous terrain. We divide a city into zones, with each zone having its own sub-network, to allow for modularity and assure computation time for route generation is linear as a function of total area. We demonstrate the strength of our proposed approach by computing air corridors through low altitude airspace of select cities with tall buildings. For US cities, we use open LiDAR elevation data to determine surface elevation maps. We select non-US cities with existing high-fidelity three-dimensional landscape models.

Physics-Based Freely Scalable Continuum Deformation For Uas Traffic Coordination

Abstract: This paper develops a novel physics-inspired traffic coordination approach and applies it to Unmanned Aircraft System (UAS) traffic management. We extend available physics-inspired approaches previously applied to 1-D traffic flow on highways and urban streets to support models of traffic coordination in higher dimension airspace for cases where no predefined paths exist. The paper considers airspace as a finite control volume while UAS coordination, treated as continuum deformation, is controlled at the airspace boundaries. By partitioning airspace into planned and unplanned spaces, the paper models nominal coordination in the planned airspace as the solution of a partial differential equation with spatiotemporal parameters. This paper also improves resilience to vehicle failures with a resilient boundary control algorithm to update the geometry of the planned space when UAS problems threaten safe coordination in existing navigable airspace channels. To support UAS coordination at the microscopic level, we propose clustering vehicles based on vehicle performance limits. UAS clusters, with each UAS treated as a particle of a virtual rigid body, use leader-follower containment to acquire the macroscopic desired trajectory.


In the Walk task, we note it actually improves over the asymptotic performance of the base TD3 #REFR algorithm within 10% of training budget.

We additionally test state-action representations of MTM by using the latent representation of the state and action encoded jointly with MTM.

We allow end to end finetuning of the representations during training.

We compare training TD3 on raw states to training TD3 with (a) state representations from the MTM model, and (b) state-action representations from the MTM model with the offline RL loss (i.e. TD3 objective).

Figure 7 depicts the learning curves for the aforementioned experiment.

In all cases we see significant improvement in training efficiency by using MTM representations -both with state and state-action representations.

Additionally, we find that the state-action representation from MTM can provide significant benefits, as in the case of the Walk task.

Here, finetuning state-action representation from MTM leads to better asymptotic performance compared to state-only representation or learning from scratch.

We provide additional plots of MTM frozen representations in Appendix E.3

Masked Trajectory Models For Prediction, Representation, And Control

Abstract: We introduce Masked Trajectory Models (MTM) as a generic abstraction for sequential decision making. MTM takes a trajectory, such as a stateaction sequence, and aims to reconstruct the trajectory conditioned on random subsets of the same trajectory. By training with a highly randomized masking pattern, MTM learns versatile networks that can take on different roles or capabilities, by simply choosing appropriate masks at inference time. For example, the same MTM network can be used as a forward dynamics model, inverse dynamics model, or even an offline RL agent. Through extensive experiments in several continuous control tasks, we show that the same MTM network-i.e. same weights-can match or outperform specialized networks trained for the aforementioned capabilities. Additionally, we find that state representations learned by MTM can significantly accelerate the learning speed of traditional RL algorithms. Finally, in offline RL benchmarks, we find that MTM is competitive with specialized offline RL algorithms, despite MTM being a generic self-supervised learning method without any explicit RL components.

Addressing Function Approximation Error In Actor-Critic Methods

Abstract: In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and critic. Our algorithm takes the minimum value between a pair of critics to restrict overestimation and delays policy updates to reduce per-update error. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.


These issues are addressed by the authors of #REFR , who introduced an arithmetic and geometric mean (AGM) robustness measure for STL.

Ahmad, Roberto Tron, and Calin Belta ({ahmadgh,tron,cbelta}@bu.edu) are with the Division of System Engineering, Boston University, Boston, MA 02215, USA.

#OTHEREFR Cristian Vasile (cvr519@lehigh.edu) is with Mechanical Engineering and Mechanics Department at Lehigh University, Bethlehem, PA, 18015, USA robustly satisfy the specifications #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR .

The work in #OTHEREFR considers planning for syntactically cosafe LTL using RRT * , in addition to the task specifications, other spatial requirements are expressed using fragment-STL where its robustness is used as the optimality criterion for RRT * .

In #OTHEREFR , the authors synthesize controllers for time-critical systems for which they quantify a temporal robustness measure that needs to be optimized.

The traditional robustness metric is not differentiable and it is mostly determined by one value of the signal, i.e., it "masks" most of the signal.

TWTL has several advantages over STL, MTL, and other concrete-time TLs.

First, its syntax and semantics can express serial tasks in an efficient and explicit way.

This is important in many applications, especially in robotics #OTHEREFR . Second, TWTL formulae can be efficiently translated into automata.

The complexity of the translation algorithm is independent of the formula time bounds #OTHEREFR .

This makes this logic suitable for automata-based synthesis and planning problems (see #OTHEREFR for a planning application).

Robustness Measures And Monitors For Time Window Temporal Logic

Abstract: Temporal logics (TLs) have been widely used to formalize interpretable tasks for cyber-physical systems. Time Window Temporal Logic (TWTL) has been recently proposed as a specification language for dynamical systems. In particular, it can easily express robotic tasks, and it allows for efficient, automata-based verification and synthesis of control policies for such systems. In this paper, we define two quantitative semantics for this logic, and two corresponding monitoring algorithms, which allow for real-time quantification of satisfaction of formulas by trajectories of discrete-time systems. We demonstrate the new semantics and their runtime monitors on numerical examples.

Arithmetic-Geometric Mean Robustness For Control From Signal Temporal Logic Specifications

Abstract: We present a new average-based robustness for Signal Temporal Logic (STL) and a framework for optimal control of a dynamical system under STL constraints. By averaging the scores of different specifications or subformulae at different time points, our definition highlights the frequency of satisfaction as well as how robustly each specification is satisfied. Its usefulness in control synthesis problems is illustrated through case studies.


We follow a strategy similar to that used for the sample complexity of EOT #REFR . In Thm.

Let τ, λ > 0, define d = 2 d/2 and assume that c ∈ C 1+d /2 (X × X).

Let µ λ,τ be the empirical barycenter and µ * λ,τ the population barycenter.

Then there is C > 0 independent of (ν k ) k such that

4.1, one can replace the homogeneous Sobolev normḢ −p by the (larger) inhomogeneous norm H −p .

For p = 1 + d /2, it is known that H p is a Reproducible Kernel Hilbert space norm, and by standard empirical process theory results #OTHEREFR one has

Doubly Regularized Entropic Wasserstein Barycenters

Abstract: We study a general formulation of regularized Wasserstein barycenters that enjoys favorable regularity, approximation, stability and (grid-free) optimization properties. This barycenter is defined as the unique probability measure that minimizes the sum of entropic optimal transport (EOT) costs with respect to a family of given probability measures, plus an entropy term. We denote it (λ, τ)-barycenter, where λ is the inner regularization strength and τ the outer one. This formulation recovers several previously proposed EOT barycenters for various choices of λ, τ ≥ 0 and generalizes them. First, in spite of-and in fact owing to-being doubly regularized, we show that our formulation is debiased for τ = λ/2: the suboptimality in the (unregularized) Wasserstein barycenter objective is, for smooth densities, of the order of the strength λ 2 of entropic regularization, instead of max{λ, τ } in general. We discuss this phenomenon for isotropic Gaussians where all (λ, τ)-barycenters have closed form. Second, we show that for λ, τ > 0, the barycenter has a smooth density and is strongly stable under perturbation of the marginals. In particular, it can be estimated efficiently: given n samples from each of the probability measures, it converges in relative entropy to the population barycenter at a rate n −1/2. And finally, this formulation lends itself naturally to a grid-free optimization algorithm: we propose a simple noisy particle gradient descent which, in the mean-field limit, converges globally at an exponential rate to the barycenter.

Sample Complexity Of Sinkhorn Divergences

Abstract: Optimal transport (OT) and maximum mean discrepancies (MMD) are now routinely used in machine learning to compare probability measures. We focus in this paper on \emph{Sinkhorn divergences} (SDs), a regularized variant of OT distances which can interpolate, depending on the regularization strength $\varepsilon$, between OT ($\varepsilon=0$) and MMD ($\varepsilon=\infty$). Although the tradeoff induced by that regularization is now well understood computationally (OT, SDs and MMD require respectively $O(n^3\log n)$, $O(n^2)$ and $n^2$ operations given a sample size $n$), much less is known in terms of their \emph{sample complexity}, namely the gap between these quantities, when evaluated using finite samples \emph{vs.} their respective densities. Indeed, while the sample complexity of OT and MMD stand at two extremes, $1/n^{1/d}$ for OT in dimension $d$ and $1/\sqrt{n}$ for MMD, that for SDs has only been studied empirically. In this paper, we \emph{(i)} derive a bound on the approximation error made with SDs when approximating OT as a function of the regularizer $\varepsilon$, \emph{(ii)} prove that the optimizers of regularized OT are bounded in a Sobolev (RKHS) ball independent of the two measures and \emph{(iii)} provide the first sample complexity bound for SDs, obtained,by reformulating SDs as a maximization problem in a RKHS. We thus obtain a scaling in $1/\sqrt{n}$ (as in MMD), with a constant that depends however on $\varepsilon$, making the bridge between OT and MMD complete.


As recently #REFR showed, Rademacher random projections are asymptotically dimensionoptimal with exact constant ; this result improves upon a previous suboptimal bound of Kane and Nelson [26] .

Yet the quantitative analysis of the property #OTHEREFR has remained a difficult challenge, resulting in complex proofs simplified many times #OTHEREFR , crude statistical bounds (for example, sparse variants have an exponential gap with respect to the sharp no-go results #OTHEREFR ), and a lack of finitedimensional insights (bounds are input-oblivious which widens the gap between theory predictions and empirical performance #OTHEREFR ).

This work addresses the aforementioned gap by revisiting the most promising construction of Rademacher random projections, which uses the following matrix

More specifically, this paper solves the following problem:

Give a precise, non-asymptotic, non-oblivious analysis of random projections #OTHEREFR .

The statistical performance of Rademacher projections is superior to the sparse ones, as demonstrated empirically in 1.

Furthermore, the theoretical bounds for Rademacher random projections are much better than those available for sparse analogues #OTHEREFR .

The best, prior to this paper, analysis of (2) is given by Achlioptas in #OTHEREFR .

It is worth noting that Rademacher projections are also superior to their Gaussian counterparts; indeed, we know that they are dominated by the gaussian-based projections #OTHEREFR .

The relation of statistical performance and input structure has not been understood in-depth yet; as for conceptually similar research, we note that recent results show that for sparse data one can improve the sparsity of random projections, gaining in computing time #OTHEREFR .

Exact Non-Oblivious Performance Of Rademacher Random Embeddings

Abstract: This paper revisits the performance of Rademacher random projections, establishing novel statistical guarantees that are numerically sharp and non-oblivious with respect to the input data. More specifically, the central result is the Schur-concavity property of Rademacher random projections with respect to the inputs. This offers a novel geometric perspective on the performance of random projections, while improving quantitatively on bounds from previous works. As a corollary of this broader result, we obtained the improved performance on data which is sparse or is distributed with small spread. This non-oblivious analysis is a novelty compared to techniques from previous work, and bridges the frequently observed gap between theory and practise. The main result uses an algebraic framework for proving Schur-concavity properties, which is a contribution of independent interest and an elegant alternative to derivative-based criteria.

Optimal Bounds For Johnson-Lindenstrauss Transformations

Abstract: In 1984, Johnson and Lindenstrauss proved that any finite set of data in a high-dimensional space can be projected to a lower-dimensional space while preserving the pairwise Euclidean distance between points up to a bounded relative error. If the desired dimension of the image is too small, however, Kane, Meka, and Nelson (2011) and Jayram and Woodruff (2013) independently proved that such a projection does not exist. In this paper, we provide a precise asymptotic threshold for the dimension of the image, above which, there exists a projection preserving the Euclidean distance, but, below which, there does not exist such a projection.


B. Camera-Lidar Fusion For Relocalization With Multi-Head Self-Attention

Like the Transformer encoder in #REFR , a normalization layer is applied before MHSA, and a residual connection is attached after MHSA.

W p f T Att1 , . . . , f T Attj , . . . , f T Att N h T ,

where f Attj is the output of the j-th scaled dot-product attention, N h is the number of the attention heads, and

is the concatenation of {a T i } n i=1 .

In this operation, each attention is scaled by a scaling factor N h so that its output has the same dimension as the input.

We employ batch normalization (BN, #OTHEREFR ) instead of layer normalization (LN, #OTHEREFR ) different from #OTHEREFR .

It was demonstrated in #OTHEREFR that LN is more effective than BN for recurrent networks.

However, we find out from experiments that BN is more effective than LN in this work.

Also, we do not use the positional encoding, another input of the Transformer encoder, because the order of elements in the fusion feature is not important in this task, unlike a sequence.

This MHSA block with identical architecture repeats N l times as in #OTHEREFR .

Fusionloc: Camera-2D Lidar Fusion Using Multi-Head Self-Attention For End-To-End Serving Robot Relocalization

Abstract: As technology advances in autonomous mobile robots, mobile service robots have been actively used more and more for various purposes. Especially, serving robots have been not surprising products anymore since the COVID-19 pandemic. One of the practical problems in operating serving a robot is that it often fails to estimate its pose on a map that it moves around. Whenever the failure happens, servers should bring the serving robot to its initial location and reboot it manually. In this paper, we focus on end-to-end relocalization of serving robots to address the problem. It is to predict robot pose directly from only the onboard sensor data using neural networks. In particular, we propose a deep neural network architecture for the relocalization based on camera-2D LiDAR sensor fusion. We call the proposed method FusionLoc. In the proposed method, the multi-head selfattention complements different types of information captured by the two sensors to regress the robot pose. Our experiments on a dataset collected by a commercial serving robot demonstrate that FusionLoc can provide better performances than previous endto-end relocalization methods taking only a single image or a 2D LiDAR point cloud as well as a straightforward fusion method concatenating their features.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


UNet++ #REFR uses nested and dense skip connections to reduce the semantic gap between the feature maps of encoder and decoder, as shown in Fig. 1 (c) .

There are three general challenges in accurate segmentation: Firstly, U-shape structures #OTHEREFR , #OTHEREFR have received considerable attention due to their abilities of utilizing multi-level information to reconstruct high-resolution feature maps.

In UNet #OTHEREFR , the up-sampled feature maps are concatenated with feature maps skipped from the encoder and convolutions and non-linearities are added between up-sampling steps, as shown in Fig. 1 (a) .

Subsequent UNet-based methods design diverse feature enhancement modules via attention mechanism #OTHEREFR , #OTHEREFR , gate mechanism #OTHEREFR , #OTHEREFR , transformer technique #OTHEREFR , #OTHEREFR , as shown in Fig. 1 (b) .

Generally speaking, different level features in encoder have different characteristics.

High-level ones have more semantic information which helps localize the objects, while low-level ones have more detailed information which can capture the subtle boundaries of objects.

The decoder leverages the levelspecific and cross-level characteristics to generate the final high-resolution prediction.

Nevertheless, the aforementioned methods directly use an element-wise addition or concatenation to fuse any two level features from the encoder and transmit them to the decoder.

These simple operations do not pay more attention to differential information between different levels.

M 2 Snet: Multi-Scale In Multi-Scale Subtraction Network For Medical Image Segmentation

Abstract: Accurate medical image segmentation is critical for early medical diagnosis. Most existing methods are based on U-shape structure and use element-wise addition or concatenation to fuse different level features progressively in decoder. However, both the two operations easily generate plenty of redundant information, which will weaken the complementarity between different level features, resulting in inaccurate localization and blurred edges of lesions. To address this challenge, we propose a general multi-scale in multi-scale subtraction network (M 2 SNet) to finish diverse segmentation from medical image. Specifically, we first design a basic subtraction unit (SU) to produce the difference features between adjacent levels in encoder. Next, we expand the single-scale SU to the intralayer multi-scale SU, which can provide the decoder with both pixel-level and structure-level difference information. Then, we pyramidally equip the multi-scale SUs at different levels with varying receptive fields, thereby achieving the inter-layer multi-scale feature aggregation and obtaining rich multi-scale difference information. In addition, we build a training-free network "LossNet" to comprehensively supervise the task-aware features from bottom layer to top layer, which drives our multi-scale subtraction network to capture the detailed and structural cues simultaneously. Without bells and whistles, our method performs favorably against most state-of-the-art methods under different evaluation metrics on eleven datasets of four different medical image segmentation tasks of diverse image modalities, including color colonoscopy imaging, ultrasound imaging, computed tomography (CT), and optical coherence tomography (OCT). The source code can be available at https: //github.com/Xiaoqi-Zhao-DLUT/MSNet.

Unet++: Redesigning Skip Connections To Exploit Multiscale Features In Image Segmentation

Abstract: The state-of-the-art models for medical image segmentation are variants of U-Net and fully convolutional networks (FCN). Despite their success, these models have two limitations: (1) their optimal depth is apriori unknown, requiring extensive architecture search or inefficient ensemble of models of varying depths; and (2) their skip connections impose an unnecessarily restrictive fusion scheme, forcing aggregation only at the same-scale feature maps of the encoder and decoder sub-networks. To overcome these two limitations, we propose UNet++, a new neural architecture for semantic and instance segmentation, by (1) alleviating the unknown network depth with an efficient ensemble of U-Nets of varying depths, which partially share an encoder and co-learn simultaneously using deep supervision; (2) redesigning skip connections to aggregate features of varying semantic scales at the decoder sub-networks, leading to a highly flexible feature fusion scheme; and (3) devising a pruning scheme to accelerate the inference speed of UNet++. We have evaluated UNet++ using six different medical image segmentation datasets, covering multiple imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and electron microscopy (EM), and demonstrating that (1) UNet++ consistently outperforms the baseline models for the task of semantic segmentation across different datasets and backbone architectures; (2) UNet++ enhances segmentation quality of varying-size objects -- an improvement over the fixed-depth U-Net; (3) Mask RCNN++ (Mask R-CNN with UNet++ design) outperforms the original Mask R-CNN for the task of instance segmentation; and (4) pruned UNet++ models achieve significant speedup while showing only modest performance degradation. Our implementation and pre-trained models are available at this https URL.


The approach taken in #REFR to place PAC guarantees applies this concept by attempting to estimate the VC-dimension of the classification algorithm.

There are a number of papers that address the specific topic using PAC-based guarantees to generalize error bounds within CPSs.

Notable investigations in this area include #OTHEREFR , #OTHEREFR , #OTHEREFR and #OTHEREFR .

Similar to the objective of this study, the PAC-based guarantees in the aforementioned works correlate the size of the training data to the failure rate with a particular level of confidence.

The error bounds for learning described in #OTHEREFR and #OTHEREFR use a generalized term correlated to the size of the hypothesis space to describe the target concept sample complexity.

This is further generalized in #OTHEREFR as a bound that is dependent on the VC-dimension of the model being used.

In contrast to this, #OTHEREFR proposes the formulation of PACbased error bounds through the formulation of the problem as an optimization problem with the objective of minimizing the constraint violation probability.

One of the main contributions of #OTHEREFR is that stochastic perturbations within the input layer, with an underlying probability distribution, are factored into the derived error bounds.

Because the problem investigated in this study can be framed similarly, a similar approach to #OTHEREFR is utilized when deriving the error bounds.

However, because this study attempts to approximate safety constraints using conformal prediction #OTHEREFR the guarantee placed on the constraints being accurate are incorporated into the PAC-based generalized error bounds for the entire system.

To the best of our knowledge, aside from this paper, there are no existing studies on combining multiple types of guarantees when bounding the failure rate of an entire system.

Pac-Based Formal Verification For Out-Of-Distribution Data Detection

Abstract: Cyber-physical systems (CPS) like autonomous vehicles, that utilize learning components, are often sensitive to noise and out-of-distribution (OOD) instances encountered during runtime. As such, safety critical tasks depend upon OOD detection subsystems in order to restore the CPS to a known state or interrupt execution to prevent safety from being compromised. However, it is difficult to guarantee the performance of OOD detectors as it is difficult to characterize the OOD aspect of an instance, especially in high-dimensional unstructured data. To distinguish between OOD data and data known to the learning component through the training process, an emerging technique is to incorporate variational autoencoders (VAE) within systems and apply classification or anomaly detection techniques on their latent spaces. The rationale for doing so is the reduction of the data domain size through the encoding process, which benefits real-time systems through decreased processing requirements, facilitates feature analysis for unstructured data and allows more explainable techniques to be implemented. This study places probably approximately correct (PAC) based guarantees on OOD detection using the encoding process within VAEs to quantify image features and apply conformal constraints over them. This is used to bound the detection error on unfamiliar instances, , with user-defined confidence, 1 − δ. The approach used in this study is to empirically establish these bounds by sampling the latent probability distribution and evaluating the error with respect to the constraint violations that are encountered. The guarantee is then verified using data generated from CARLA, an open-source driving simulator.

Pac Confidence Sets For Deep Neural Networks Via Calibrated Prediction

Abstract: We propose an algorithm combining calibrated prediction and generalization bounds from learning theory to construct confidence sets for deep neural networks with PAC guarantees---i.e., the confidence set for a given input contains the true label with high probability. We demonstrate how our approach can be used to construct PAC confidence sets on ResNet for ImageNet, and on a dynamics model the half-cheetah reinforcement learning problem.


The model is based on sentence-BERT (SBERT) similarity measures #REFR , which specifically targets the STS task.

As demonstrated in previous research #OTHEREFR , document-level models suffer from the loss of details, which affects accuracy.

SBERT, however, has been optimized to treat text at the sentence-level, and not document-level, yielding better results #OTHEREFR than BERT.

CWE input is segmented into sentences and the model computes the cosine similarity between two sentence embeddings.

It was observed that CVE records, in the released dataset, have on average 3.69 sentences.

Alternatively, the 25 CWE inputs is a collate of the name, the description, and the extended description with an average of 8.2 sentences.

Automated Mapping Of Cve Vulnerability Records To Mitre Cwe Weaknesses

Abstract: In recent years, a proliferation of cyber-security threats and diversity has been on the rise culminating in an increase in their reporting and analysis. To counter that, many non-profit organizations have emerged in this domain, such as MITRE and OSWAP, which have been actively tracking vulnerabilities, and publishing defense recommendations in standardized formats. As producing data in such formats manually is very time-consuming, there have been some proposals to automate the process. Unfortunately, a major obstacle to adopting supervised machine learning for this problem has been the lack of publicly available specialized datasets. Here, we aim to bridge this gap. In particular, we focus on mapping CVE records into MITRE CWE Weaknesses, and we release to the research community a manually annotated dataset of 4,012 records for this task. With a human-in-the-loop framework in mind, we approach the problem as a ranking task and aim to incorporate reinforced learning to make use of the human feedback in future work. Our experimental results using fine-tuned deep learning models, namely Sentence-BERT and rankT5, show sizable performance gains over BM25, BERT, and RoBERTa, which demonstrates the need for an architecture capable of good semantic understanding for this task.

Sentence-Bert: Sentence Embeddings Using Siamese Bert-Networks

Abstract: BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. ::: In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. ::: We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.


Federated learning (FL), recently developed and proposed by Google as an emerging distributed machine learning technology, will provide further new technology to support the intelligence of drones #REFR .

However, traditional machine learning techniques require uploading all data to a cloud-based server for training and processing, which represents a considerable challenge for drone swarms #OTHEREFR .

In a first consideration, the data generated by drones may be sensitive, and could be intercepted while uploading the data to the cloud, leading to a privacy breach.

Secondly, drones' large numbers of data can result in impractical delays when uploading, thus creating a time lag for swarms of drones that prevents them from conducting real-time monitoring.

Finally, drones can consume a great amount of energy when training models, meaning there may be related challenges to doing so in terms of energy constraints #OTHEREFR .

Distributed machine learning techniques represent a new solution to address these issues and challenges, whereby drones train machine learning models without sharing raw data.

The concept of federated learning is allowing each drone to train its learning model based on its data.

The parameters of each drone's trained model are then sent to a parameter server to update the model for a new round of training, without sending the raw data to the cloud.

This training model allows for reasonable data security, latency and energy consumption.

However, the highly mobile nature of drones means conventional FL is not well-suited, given their complex working environment.

If the parameter server does not work properly, it will impact the training effectiveness of the whole UAV network #OTHEREFR .

Decentralized Federated Learning Methods For Reducing Communication Cost And Energy Consumption In Uav Networks

Abstract: Unmanned aerial vehicles (UAV) or drones play many roles in a modern smart city such as the delivery of goods, mapping real-time road traffic and monitoring pollution. The ability of drones to perform these functions often requires the support of machine learning technology. However, traditional machine learning models for drones encounter data privacy problems, communication costs and energy limitations. Federated Learning, an emerging distributed machine learning approach, is an excellent solution to address these issues. Federated learning (FL) allows drones to train local models without transmitting raw data. However, existing FL requires a central server to aggregate the trained model parameters of the UAV. A failure of the central server can significantly impact the overall training. In this paper, we propose two aggregation methods: Commutative FL and Alternate FL, based on the existing architecture of decentralised Federated Learning for UAV Networks (DFL-UN) by adding a unique aggregation method of decentralised FL. Those two methods can effectively control energy consumption and communication cost by controlling the number of local training epochs, local communication, and global communication. The simulation results of the proposed training methods are also presented to verify the feasibility and efficiency of the architecture compared with two benchmark methods (e.g. standard machine learning training and standard single aggregation server training). The simulation results show that the proposed methods outperform the benchmark methods in terms of operational stability, energy consumption and communication cost.

Communication-Efficient Learning Of Deep Networks From Decentralized Data

Abstract: Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning. We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.


A similar result was obtained in #REFR , where they argued that a reasonable amount of attention heads could be removed during test time without significant performance loss.

There has been substantial recent research on examining the attention mechanism.

Layer-based attention distribution analysis for 128-token-long inputs was conducted in #OTHEREFR to measure the syntactic ability of attention heads.

One of the findings of #OTHEREFR is that the self-attention heads within the same layer have the similar attention distribution.

According to #OTHEREFR , BERT's initial layers are crucial for capturing word-order information.

In contrast, middle layers are essential for syntactic information #OTHEREFR and the final layer representations are prominent for taskspecific adaptation #OTHEREFR .

However, the relationship between attention weights and model outputs is ambiguous.

For example, #OTHEREFR finds that the attention values have weak correlation with feature importance measures using gradient or feature erasure methods.

They also demonstrate that different sets of attention values learned using adversarial training can result in the same prediction, therefore attention values should not be utilised as an explanation of the model's predictions.

Input-Length-Shortening And Text Generation Via Attention Values

Abstract: Identifying words that impact a task's performance more than others is a challenge in natural language processing. Transformers models have recently addressed this issue by incorporating an attention mechanism that assigns greater attention (i.e., relevance) scores to some words than others. Because of the attention mechanism's high computational cost, transformer models usually have an input-length limitation caused by hardware constraints. This limitation applies to many transformers, including the well-known bidirectional encoder representations of the transformer (BERT) model. In this paper, we examined BERT's attention assignment mechanism, focusing on two questions: (1) How can attention be employed to reduce input length? (2) How can attention be used as a control mechanism for conditional text generation?We investigated these questions in the context of a text classification task. We discovered that BERT's early layers assign more critical attention scores for text classification tasks compared to later layers. We demonstrated that the first layer's attention sums could be used to filter tokens in a given sequence, considerably decreasing the input length while maintaining good test accuracy. We also applied filtering, which uses a computeefficient semantic similarities algorithm, and discovered that retaining approximately 6% of the original sequence is sufficient to obtain 86.5% accuracy. Finally, we showed that we could generate data in a stable manner and indistinguishable from the original one by only using a small percentage (10%) of the tokens with high attention scores according to BERT's first layer.

Are Sixteen Heads Really Better Than One?

Abstract: Multi-headed attention is a driving force behind recent state-of-the-art NLP models. By applying multiple attention mechanisms in parallel, it can express sophisticated functions beyond the simple weighted average. However we observe that, in practice, a large proportion of attention heads can be removed at test time without significantly impacting performance, and that some layers can even be reduced to a single head. Further analysis on machine translation models reveals that the self-attention layers can be significantly pruned, while the encoder-decoder layers are more dependent on multi-headedness.


B. Multiview-Activated Scvp Network Architecture

We follow the setting in NBVNet #REFR to construct the shape size of our input and output.

The function of multiview-activated (MA-)SCVP network is a classic multilabel classification [71] function: #OTHEREFR This function takes a 32 ×32 ×32 occupancy grid and a vector of 32 bits as input, and predicts a vector of 32 bits so that the V * cover can be obtained.

Since convolution is easier to operate on data with equal dimensions, we extract a 32 × 32 × 32 cubic bounding box from our OctoMap M.

We adopt the dynamic resolution of M so that the shape size of 32 × 32 × 32 is sufficient to generalize to different object sizes.

In the real world, we assume o size is not greater than 15 cm, which can be predicted by solving the minimum bounding sphere in the point clouds to obtain the M.

The view state vector V state is the same size as our candidate view space.

A bit in our network output is bound to a certain candidate view because the SCOP is solved in such a fixed candidate view space.

One-Shot View Planning For Fast And Complete Unknown Object Reconstruction

Abstract: Fig. 1: Comparison of surface details, views, and paths to reconstruct an untrained object: reconstructed 3D models (red point clouds), major missing surface details (gray voxels in enlarged areas), local paths (cyan), global paths (purple), views (red-green-blue), and the same initial view (black circle). (a) Results of an iterative learning-based NBV network. (b) Results of the one-shot SCVP network. (c) Results of our novel combined pipeline that selects four NBVs before activating the SCVP network. (d) Results of our novel combined pipeline that selects one NBV before activating the proposed MA-SCVP network. To ensure that sufficient surface details can be illuminated (black arrow), 4-NBV+SCVP requires more views and paths than MA-SCVP in this example (14 views with 4 local paths vs. 13 views with 1 local path).

Supervised Learning Of The Next-Best-View For 3D Object Reconstruction

Abstract: Motivated by the advances in 3D sensing technology and the spreading of low-cost robotic platforms, 3D object reconstruction has become a common task in many areas. Nevertheless, the selection of the optimal sensor pose that maximizes the reconstructed surface is a problem that remains open. It is known in the literature as the next-best-view planning problem. In this paper, we propose a novel next-best-view planning scheme based on supervised deep learning. The scheme contains an algorithm for automatic generation of datasets and an original three-dimensional convolutional neural network (3D-CNN) used to learn the next-best-view. Unlike previous work where the problem is addressed as a search, the trained 3D-CNN directly predicts the sensor pose. We present a comparison of the proposed network against a similar net, and we present several experiments of the reconstruction of unknown objects validating the effectiveness of the proposed scheme.


We use the images rendered by #REFR and follow its split to separate the training and testing sets.

To evaluate the proposed NeRF-LEBM framework and the learning algorithms, we conduct experiments on three datasets.

The Carla dataset is rendered by #OTHEREFR using the Carla Driving Simulator #OTHEREFR .

It contains 10k cars of different shapes, colors and textures.

Each car has one 2D image rendered from one random camera pose.

Another dataset is the ShapeNet #OTHEREFR Car dataset, which contains 2.1k different cars for training and 700 cars for testing.

Each car in the training set has 250 views and we only use 50 views of them for training. Each car in the testing set has 251 views. each image is associated with its camera pose information.

Likelihood-Based Generative Radiance Field With Latent Space Energy-Based Model For 3D-Aware Disentangled Image Representation

Abstract: 1 We propose the NeRF-LEBM, a likelihoodbased top-down 3D-aware 2D image generative model that incorporates 3D representation via Neural Radiance Fields (NeRF) and 2D imaging process via differentiable volume rendering. The model represents an image as a rendering process from 3D object to 2D image and is conditioned on some latent variables that account for object characteristics and are assumed to follow informative trainable energy-based prior models. We propose two likelihood-based learning frameworks to train the NeRF-LEBM: (i) maximum likelihood estimation with Markov chain Monte Carlo-based inference and (ii) variational inference with the reparameterization trick. We study our models in the scenarios with both known and unknown camera poses. Experiments on several benchmark datasets demonstrate that the NeRF-LEBM can infer 3D object structures from 2D images, generate 2D images with novel views and objects, learn from incomplete 2D images, and learn from 2D images with known or unknown camera poses.

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

Abstract: Unsupervised learning with generative models has the potential of discovering rich representations of 3D scenes. While geometric deep learning has explored 3D-structure-aware representations of scene geometry, these models typically require explicit 3D supervision. Emerging neural scene representations can be trained only with posed 2D images, but existing methods ignore the three-dimensional structure of scenes. We propose Scene Representation Networks (SRNs), a continuous, 3D-structure-aware scene representation that encodes both geometry and appearance. SRNs represent scenes as continuous functions that map world coordinates to a feature representation of local scene properties. By formulating the image formation as a differentiable ray-marching algorithm, SRNs can be trained end-to-end from only 2D images and their camera poses, without access to depth or shape. This formulation naturally generalizes across scenes, learning powerful geometry and appearance priors in the process. We demonstrate the potential of SRNs by evaluating them for novel view synthesis, few-shot reconstruction, joint shape and appearance interpolation, and unsupervised discovery of a non-rigid face model.


In practice, the matrix Q can be obtained via (sparse) Cholesky factorization of R U , but it can also be any rectangular matrix such that Q T Q = R U , as pointed out in #REFR Remark 2.7] .

It is for example the case for the partial subsampled randomized Hadamard transform (P-SRHT) described in #OTHEREFR .

Sufficient conditions on k are also available but they are very conservative.

However, numerical experiments from #OTHEREFR showed performances similar to the Gaussian embedding for a given dimension k.

A good practice is to use composed embeddings, for example Θ = Θ 2 Θ 1 with Θ 1 a moderate sized P-SRHT embedding and Θ 2 a small sized Gaussian embedding.

Such matrix may be obtained by Cholesky factorizations of small matrices, which are easy to compute. This is especially important for large scale problems.

Dictionary-Based Model Reduction For State Estimation

Abstract: We consider the problem of state estimation from m linear measurements, where the state u to recover is an element of the manifold M of solutions of a parameter-dependent equation. The state is estimated using a prior knowledge on M coming from model order reduction. Variational approaches based on linear approximation of M, such as PBDW, yields a recovery error limited by the Kolmogorov m-width of M. To overcome this issue, piecewise-affine approximations of M have also be considered, that consist in using a library of linear spaces among which one is selected by minimizing some distance to M. In this paper, we propose a state estimation method relying on dictionary-based model reduction, where a space is selected from a library generated by a dictionary of snapshots, using a distance to the manifold. The selection is performed among a set of candidate spaces obtained from the path of a 1-regularized least-squares problem. Then, in the framework of parameter-dependent operator equations (or PDEs) with affine parameterizations, we provide an efficient offline-online decomposition based on randomized linear algebra, that ensures efficient and stable computations while preserving theoretical guarantees.

Randomized Linear Algebra For Model Reduction. Part I: Galerkin Methods And Error Estimation

Abstract: We propose a probabilistic way for reducing the cost of classical projection-based model order reduction methods for parameter-dependent linear equations. A reduced order model is here approximated from its random sketch, which is a set of low-dimensional random projections of the reduced approximation space and the spaces of associated residuals. This approach exploits the fact that the residuals associated with approximations in low-dimensional spaces are also contained in low-dimensional spaces. We provide conditions on the dimension of the random sketch for the resulting reduced order model to be quasi-optimal with high probability. Our approach can be used for reducing both complexity and memory requirements. The provided algorithms are well suited for any modern computational environment. Major operations, except solving linear systems of equations, are embarrassingly parallel. Our version of proper orthogonal decomposition can be computed on multiple workstations with a communication cost independent of the dimension of the full order model. The reduced order model can even be constructed in a so-called streaming environment, i.e., under extreme memory constraints. In addition, we provide an efficient way for estimating the error of the reduced order model, which is not only more efficient than the classical approach but is also less sensitive to round-off errors. Finally, the methodology is validated on benchmark problems.


To generalize the network Group Norm #REFR with group size 32 is used after each linear layer.

During each epoch we generate 160 point clouds for each component. Thus each training epoch consists of 232000 point clouds.

The network is trained with a batch size of 14, 7 on each GPU, using the Adam optimizer #OTHEREFR , with an initial learning rate of 0.0001.

We use a step scheduler with a step size of 20 and the gamma parameter set to 0.7.

The loss is calculated with cross entropy using a 0.2/0.8 split for segmentation and keypoint loss.

For the keypoint loss only points belonging to the object is used.

Group Norm is used as opposed to Batch Norm as a result of the small batch size.

Dropout is used for the object features, as the network should not overfit to a specific part of the object, and is used after the to concurrent linear layers.

The dropout is set to 40 %, used after the last two linear layers of the object feature and the first two of the combined feature.

Additionally, up to 0.75 % Gaussian noise is applied to the object and scene point clouds, and 10 % position shift is applied to the object point cloud.

The network was trained on a PC environment with two NVIDIA GeForce RTX 2080 GPUs.

Gp3D: Generalized Pose Estimation In 3D Point Clouds: A Case Study On Bin Picking

Abstract: In this paper, we present GP3D, a novel network for generalized pose estimation in 3D point clouds. The method generalizes to new objects by using both the scene point cloud and the object point cloud with keypoint indexes as input. The network is trained to match the object keypoints to scene points. To address the pose estimation of novel objects we also present a new approach for training pose estimation. The typical solution is a single model trained for pose estimation of a specific object in any scenario. This has several drawbacks: training a model for each object is time-consuming, energy consuming, and by excluding the scenario information the task becomes more difficult. In this paper, we present the opposite solution; a scenariospecific pose estimation method for novel objects that do not require retraining. The network is trained on 1500 objects and is able to learn a generalized solution. We demonstrate that the network is able to correctly predict novel objects, and demonstrate the ability of the network to perform outside of the trained class. We believe that the demonstrated method is a valuable solution for many real-world scenarios. Code and trained network will be made available after publication. Index Terms-pose estimation, point cloud, deep learning Pose Estimation Scene Point Cloud (Colored for visualization) Object Point Cloud with Keypoint Indecies

Group Normalization

Abstract: Batch Normalization (BN) is a milestone technique in the development of deep learning, enabling various networks to train. However, normalizing along the batch dimension introduces problems --- BN's error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation. This limits BN's usage for training larger models and transferring features to computer vision tasks including detection, segmentation, and video, which require small batches constrained by memory consumption. In this paper, we present Group Normalization (GN) as a simple alternative to BN. GN divides the channels into groups and computes within each group the mean and variance for normalization. GN's computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes. On ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN and outperforms other normalization variants. Moreover, GN can be naturally transferred from pre-training to fine-tuning. GN can outperform its BN-based counterparts for object detection and segmentation in COCO, and for video classification in Kinetics, showing that GN can effectively replace the powerful BN in a variety of tasks. GN can be easily implemented by a few lines of code in modern libraries.


LPIPS #REFR calculates the cosine distances between the network features of the two images layer by layer and averages them to estimate the perceived distance of the generated image from the ground truth image.

PSNR is numerically related to the mean squared error (MSE) between the ground truth and the reconstructed image, it is used to measure the image reconstruction quality.

SSIM measures the structural similarity between patches of the input images.

As a result, it is more robust to changes in the global illumination than PSNR.

CSIM #OTHEREFR To evaluate the effectiveness of identity preservation, we compute the cosine similarity using embedded vectors created by the pre-trained face recognition model.

AUCON is used to calculate the ratio of the same facial action unit values between the generated images and the driving images.

One-Shot High-Fidelity Talking-Head Synthesis With Deformable Neural Radiance Field

Abstract: Figure 1. Representative results of our method. The first three columns exhibit the source, driving, generated images, respectively. The rest columns show the exploration of the generated images to different yaw angles.

The Unreasonable Effectiveness Of Deep Features As A Perceptual Metric

Abstract: While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on the ImageNet classification task has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called"perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new Full Reference Image Quality Assessment (FR-IQA) dataset of perceptual human judgments, orders of magnitude larger than previous datasets. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by huge margins. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.


The FID passes images from both distributions through the pre-trained inception network #REFR obtaining embeddings for every image, e.g. 2048 dimensional embedding vectors for Inception V3.

The Fréchet Inception Distance (FID) #OTHEREFR is a method for comparing similarity of two image distributions to each other.

It is commonly used between real images and generated images by a GAN as a measure of GAN performance.

For both distribution, the estimated mean embedding vector and the covariance matrix is calculated. The final distance is defined as following:

Spatial Latent Representations In Generative Adversarial Networks For Image Generation Master'S Thesis In Computer Science

Abstract: Generative Adversarial Networks (GANs) are currently state-of-the-art methods in image generation tasks. They generate new images by transforming a latent space into an image data distribution. In the vast majority of GAN architectures, the latent space is defined as a set of vectors of given dimensionality. Such representations are not easily interpretable and do not capture spatial information of image content directly. In this work, we define a family of spatial latent spaces for StyleGAN2, capable of capturing more details and representing images that are out-of-sample in terms of the number and arrangement of object parts, such as an image of multiple faces or a face with more than two eyes. We propose a method for encoding images into our spaces, together with an attribute model capable of performing attribute editing in these spaces. We show that our spaces are effective for image manipulation purposes and encode semantic information well. Our approach can be used on pre-trained generator models, and attribute edition can be done using pre-generated direction vectors making the barrier to entry for experimentation and use extremely low. We propose a regularization method for optimizing latent representations, which equalizes distributions of parts of latent spaces, making representations much closer to generated ones. We use it for encoding images into spatial spaces to obtain significant improvement in quality while keeping semantics and ability to use our attribute model for edition purposes. In total, using our methods gives encoding quality boost even as high as 30% in terms of LPIPS score comparing to standard methods, while keeping semantics. Additionally, we propose a StyleGAN2 training procedure on our spatial latent spaces, together with a custom spatial latent representation distribution to make spatially closer elements in the representation more dependent on each other than farther elements. Such approach improves the FID score by 29% on SpaceNet, and is able to generate consistent images of arbitrary sizes on spatially homogeneous datasets, like satellite imagery.

Rethinking The Inception Architecture For Computer Vision

Abstract: Convolutional networks are at the core of most state of-the-art computer vision solutions for a wide variety of tasks. Since 2014 very deep convolutional networks started to become mainstream, yielding substantial gains in various benchmarks. Although increased model size and computational cost tend to translate to immediate quality gains for most tasks (as long as enough labeled data is provided for training), computational efficiency and low parameter count are still enabling factors for various use cases such as mobile vision and big-data scenarios. Here we are exploring ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization. We benchmark our methods on the ILSVRC 2012 classification challenge validation set demonstrate substantial gains over the state of the art: 21:2% top-1 and 5:6% top-5 error for single frame evaluation using a network with a computational cost of 5 billion multiply-adds per inference and with using less than 25 million parameters. With an ensemble of 4 models and multi-crop evaluation, we report 3:5% top-5 error and 17:3% top-1 error on the validation set and 3:6% top-5 error on the official test set.


Curve GCN #REFR regarded the initial contour as a graph and used a graph convolutional network to predict vertex-wise offsets.

For instance, Polygon RNN [9, 1] employed a CNN-RNN architecture to sequentially trace object boundaries in a given image patch.

Two-stage Deep Snake #OTHEREFR created initial octagon contours using a detector and then iteratively deformed them through a circular convolution network.

Based on that, DANCE #OTHEREFR utilized a segmentwise matching scheme and attentive contour deformation to facilitate learning.

PolyTransform #OTHEREFR generated masks for each object using an off-the-shelf mask-based segmentation pipeline and converted the resulting mask contours into a set of vertices.

Subsequently, the Transformer #OTHEREFR wrapped these vertices to fit the object silhouette better.

It employed a differentiable rendering loss to ensure that masks rendered from the predicted points agreed with the ground-truth masks.

BoundaryFormer #OTHEREFR , on the other hand, applied a differentiable rasterization method to generate masks from polygons, achieving stunning results that are almost comparable to its mask-based counterparts on standard benchmarks.

PolarMask #OTHEREFR and its followups #OTHEREFR adopted a set of rays in the polar coordinate system to represent object contours, which enables an efficient calculation of Intersection-over-Union.

Furthermore, E2EC #OTHEREFR took the vertices and rays to define contours simultaneously.

However, the deep learning-based methods mentioned above require expensive ground-truth masks or polygons, which hinders their practical applicability and extension.

Boxsnake: Polygonal Instance Segmentation With Box Supervision

Abstract: Box-supervised instance segmentation has gained much attention as it requires only simple box annotations instead of costly mask or polygon annotations. However, existing box-supervised instance segmentation models mainly focus on mask-based frameworks. We propose a new endto-end training technique, termed BoxSnake, to achieve effective polygonal instance segmentation using only box annotations for the first time. Our method consists of two loss functions: (1) a point-based unary loss that constrains the bounding box of predicted polygons to achieve coarsegrained segmentation; and (2) a distance-aware pairwise loss that encourages the predicted polygons to fit the object boundaries. Compared with the mask-based weaklysupervised methods, BoxSnake further reduces the performance gap between the predicted segmentation and the bounding box, and shows significant superiority on the Cityscapes dataset. The source code will be available at https://github.com/Yangr116/BoxSnake.

Fast Interactive Object Annotation With Curve-Gcn

Abstract: Manually labeling objects by tracing their boundaries is a laborious process. In Polygon-RNN++, the authors proposed Polygon-RNN that produces polygonal annotations in a recurrent manner using a CNN-RNN architecture, allowing interactive correction via humans-in-the-loop. We propose a new framework that alleviates the sequential nature of Polygon-RNN, by predicting all vertices simultaneously using a Graph Convolutional Network (GCN). Our model is trained end-to-end, and runs in real time. It supports object annotation by either polygons or splines, facilitating labeling efficiency for both line-based and curved objects. We show that Curve-GCN outperforms all existing approaches in automatic mode, including the powerful DeepLab, and is significantly more efficient in interactive mode than Polygon-RNN++. Our model runs at 29.3ms in automatic, and 2.6ms in interactive mode, making it 10x and 100x faster than Polygon-RNN++.


Ultimately, based on the results in Table 4 , we conclude that HieCoAtt #REFR is the most robust VQA model.

Because the plots are monotonously decreasing in accuracy, or, equivalently, monotonously increasing in accuracy decrement, the ranking is effective.

In this figure, "First top 3" represents the first partition, "Second top 3" represents the second partition and so on. models are more robust than non-attention-based ones.

However, when we examine MU and MUA in Table 4 ( 2 ), the non-attention-based model (MU) is more robust than the attention-based model (MUA).

It is worth noting that the only difference between MU and MUA is the attention mechanism.

Meanwhile, in Table 4 ( 1 ), MUA is more robust than MU, indicating that the diversity of BQ candidates affects the robustness of attention-based VQA models in some cases.

The HieCoAtt model employs a co-attention mechanism that repeatedly exploits the text and image information to guide Fig. 7 . Visual Question Answering by Basic Questions (VQABQ) pipeline.

Note that in Module 1 all of the training and validation questions are only encoded by Skip-Thought Question Encoder once for generating the Basic Question Matrix.

That is, the next input of Skip-Thought Question Encoder is only a new main question.

Module 2 is a VQA model which we want to test, and it is the HieCoAtt VQA model in our case.

Regarding the input question of the HieCoAtt model, it is the direct concatenation of a given main question with the corresponding selected basic questions based on the Threshold-based Criterion. "⊕" denotes the direct concatenation of basic questions.

Improving Visual Question Answering Models Through Robustness Analysis And In-Context Learning With A Chain Of Basic Questions

Abstract: Deep neural networks have been critical in the task of Visual Question Answering (VQA), with research traditionally focused on improving model accuracy. Recently, however, there has been a trend towards evaluating the robustness of these models against adversarial attacks. This involves assessing the accuracy of VQA models under increasing levels of noise in the input, which can target either the image or the proposed query question, dubbed the main question. However, there is currently a lack of proper analysis of this aspect of VQA. This work proposes a new method that utilizes semantically related questions, referred to as basic questions, acting as noise to evaluate the robustness of VQA models. It is hypothesized that as the similarity of a basic question to the main question decreases, the level of noise increases. To generate a reasonable noise level for a given main question, a pool of basic questions is ranked based on their similarity to the main question, and this ranking problem is cast as a optimization problem. Additionally, this work proposes a novel robustness measure, , and two basic question datasets to standardize the analysis of VQA model robustness. The experimental results demonstrate that the proposed evaluation method effectively analyzes the robustness of VQA models. Moreover, the experiments show that in-context learning with a chain of basic questions can enhance model accuracy.

Hierarchical Question-Image Co-Attention For Visual Question Answering

Abstract: A number of recent works have proposed attention models for Visual Question Answering (VQA) that generate spatial maps highlighting image regions relevant to answering the question. In this paper, we argue that in addition to modeling "where to look" or visual attention, it is equally important to model "what words to listen to" or question attention. We present a novel co-attention model for VQA that jointly reasons about image and question attention. In addition, our model reasons about the question (and consequently the image via the co-attention mechanism) in a hierarchical fashion via a novel 1-dimensional convolution neural networks (CNN). Our model improves the state-of-the-art on the VQA dataset from 60.3% to 60.5%, and from 61.6% to 63.3% on the COCO-QA dataset. By using ResNet, the performance is further improved to 62.1% for VQA and 65.4% for COCO-QA.


As demonstrated in #REFR , activities in the Opportunity dataset were less structured and closer to real-life scenarios, hence is the most difficult dataset among the three, which resulted in the lowest performance of M S on D S .

The scores were then averaged across the classes to obtain the final precision, recall and F1 scores.

For each of the three dataset, we evaluated the performance metrics with three experiments:

• Table 1 shows the accuracy, precision, recall and F1 score for each of these three models for all the three datasets.

The supervised training of M S on D S yields the highest scores for the PAMAP2 dataset.

Lower performance on the MHEALTH dataset could be partially attributed to the smaller volume of training data and fewer channels of data being used (as stated in Section 5 we only used three accelerometer channels for MHEALTH dataset as opposed to nine IMU channels for the other two datasets).

The same performance trend across the three datasets is also observed in the other two experiments, M S on D ST and M T on D ST .

When comparing the model performance on the target domain, the target domain model M T trained on D ST performs significantly better in all the three datasets than the source domain model being directly applied to the target domain i.e., M S on D T .

This shows that the embedding extractor E T , when trained on D ST , is able to learn similar discriminative information as in the source domain embedding e S for constructing the target domain embeddings e T .

Thus, fine-tuning an existing activity classification model, as part of our unsupervised method, could be potentially sufficient for performing the classification task at a new body location.

However, as the performance of M T on D ST across the three datasets are in the same order as of M S on D S , the potential performance increase from such fine-tuning could be partially limited by the performance of the original supervised model from the source location. Fig.

Activity Classification Using Unsupervised Domain Transfer From Body Worn Sensors

Abstract: Activity classification has become a vital feature of wearable health tracking devices. As innovation in this field grows, wearable devices worn on different parts of the body are emerging. To perform activity classification on a new body location, labeled data corresponding to the new locations are generally required, but this is expensive to acquire. In this work, we present an innovative method to leverage an existing activity classifier, trained on Inertial Measurement Unit (IMU) data from a reference body location (the source domain), in order to perform activity classification on a new body location (the target domain) in an unsupervised way, i.e. without the need for classification labels at the new location. Specifically, given an IMU embedding model trained to perform activity classification at the source domain, we train an embedding model to perform activity classification at the target domain by replicating the embeddings at the source domain. This is achieved using simultaneous IMU measurements at the source and target domains. The replicated embeddings at the target domain are used by a classification model that has previously been trained on the source domain to perform activity classification at the target domain. We have evaluated the proposed methods on three activity classification datasets PAMAP2, MHealth, and Opportunity, yielding high F1 scores of 67.19%, 70.40% and 68.34%, respectively when the source domain is the wrist and the target domain is the torso.

Ensembles Of Deep Lstm Learners For Activity Recognition Using Wearables

Abstract: Recently, deep learning (DL) methods have been introduced very successfully into human activity recognition (HAR) scenarios in ubiquitous and wearable computing. Especially the prospect of overcoming the need for manual feature design combined with superior classification capabilities render deep neural networks very attractive for real-life HAR applications. Even though DL-based approaches now outperform the state-of-the-art in a number of recognition tasks, still substantial challenges remain. Most prominently, issues with real-life datasets, typically including imbalanced datasets and problematic data quality, still limit the effectiveness of activity recognition using wearables. In this paper we tackle such challenges through Ensembles of deep Long Short Term Memory (LSTM) networks. LSTM networks currently represent the state-of-the-art with superior classification performance on relevant HAR benchmark datasets. We have developed modified training procedures for LSTM networks and combine sets of diverse LSTM learners into classifier collectives. We demonstrate that Ensembles of deep LSTM learners outperform individual LSTM networks and thus push the state-of-the-art in human activity recognition using wearables. Through an extensive experimental evaluation on three standard benchmarks (Opportunity, PAMAP2, Skoda) we demonstrate the excellent recognition capabilities of our approach and its potential for real-life applications of human activity recognition.


Namely, RESIT leans a causal structure assuming noise model structural equations, while Lattimore et al. #REFR actually assumes a non-parametric causal graph.

5 , we have not addressed the fact that the used structure learner actually yields a graph on a different parametric level than what is assumed by the reasoning task.

Luckily, because of the composition of both the structural and parametric scale, each stricter assumption is subsumed by a lower level assumption.

As such, from left to right, we can always relax the assumptions (such as allowing non-parametric reasoning based on a noise-model causal structure), but not the other way around.

If, for example, our structure learner yielded a plausible but non-parametric causal structure, we are not guaranteed optimal regret from Lattimore et al. #OTHEREFR .

On our map, that would become clear as we would move from a less strict assumption (plausible causality) to a strict assumption (full causality).

Causal Deep Learning

Abstract: Causality has the potential to truly transform the way we solve a large number of realworld problems. Yet, so far, its potential remains largely unlocked since most work so far requires strict assumptions which do not hold true in practice. To address this challenge and make progress in solving real-world problems, we propose a new way of thinking about causality-we call this causal deep learning. The framework which we propose for causal deep learning spans three dimensions: (1) a structural dimension, which allows incomplete causal knowledge rather than assuming either full or no causal knowledge; (2) a parametric dimension, which encompasses parametric forms which are typically ignored; and finally, (3) a temporal dimension, which explicitly allows for situations which capture exposure times or temporal structure. Together, these dimensions allow us to make progress on a variety of real-world problems by leveraging (sometimes incomplete) causal knowledge and/or combining diverse causal deep learning methods. This new framework also enables researchers to compare systematically across existing works as well as identify promising research areas which can lead to real-world impact.

Causal Bandits: Learning Good Interventions Via Causal Inference

Abstract: We study the problem of using causal models to improve the rate at which good interventions can be learned online in a stochastic environment. Our formalism combines multi-arm bandits and causal inference to model a novel type of bandit feedback that is not exploited by existing approaches. We propose a new algorithm that exploits the causal feedback and prove a bound on its simple regret that is strictly better (in all quantities) than algorithms that do not use the additional causal information.


However, it is argued in works such as #REFR that for certain datasets such as Airport and Disease, one should consider embedding nodes in hyperbolic spaces and perform feature aggregation in the tangent spaces of hyperbolic spaces.

In each row, best performers are highlighted in blue and red for our approach and benchmarks, respectively.

An underlined entry means no noticeable performance improvement over the base model is observed.

In general, we see that our proposed regularized models improve upon their respective base models with significant performance gain in many cases.

Moreover, our method can match up with or even outperform many benchmarks.

2) Hyperbolic models: Base models considered in Section IV-A1 generate embedding of nodes in Euclidean spaces.

Such a consideration is plausible as certain graphs are inherently hyperbolic (measured by δ-hyperbolicity, see #OTHEREFR ).

In this subsection for Airport and Disease datasets, we use hyperbolic versions of their Euclidean counterparts HGCN #OTHEREFR , and HGAT #OTHEREFR as base models.

We also consider the interactive model GIL that combines both Euclidean and hyperbolic approaches #OTHEREFR . The comparison results are shown in Table II .

Again, we see a general improvement by using the proposed regularization, which yields performance comparable with benchmarks.

3) Inductive learning models: In contrast with transductive learning, inductive learning requires one to deal with unseen data outside the training set.

Distributional Signals For Node Classification In Graph Neural Networks

Abstract: In graph neural networks (GNNs), both node features and labels are examples of graph signals, a key notion in graph signal processing (GSP). While it is common in GSP to impose signal smoothness constraints in learning and estimation tasks, it is unclear how this can be done for discrete node labels. We bridge this gap by introducing the concept of distributional graph signals. In our framework, we work with the distributions of node labels instead of their values and propose notions of smoothness and non-uniformity of such distributional graph signals. We then propose a general regularization method for GNNs that allows us to encode distributional smoothness and non-uniformity of the model output in semi-supervised node classification tasks. Numerical experiments demonstrate that our method can significantly improve the performance of most base GNN models in different problem settings.

Hyperbolic Graph Convolutional Neural Networks

Abstract: Graph convolutional neural networks (GCNs) embed nodes in a graph into Euclidean space, which has been shown to incur a large distortion when embedding real-world graphs with scale-free or hierarchical structure. Hyperbolic geometry offers an exciting alternative, as it enables embeddings with much smaller distortion. However, extending GCNs to hyperbolic geometry presents several unique challenges because it is not clear how to define neural network operations, such as feature transformation and aggregation, in hyperbolic space. Furthermore, since input features are often Euclidean, it is unclear how to transform the features into hyperbolic embeddings with the right amount of curvature. Here we propose Hyperbolic Graph Convolutional Neural Network (HGCN), the first inductive hyperbolic GCN that leverages both the expressiveness of GCNs and hyperbolic geometry to learn inductive node representations for hierarchical and scale-free graphs. We derive GCNs operations in the hyperboloid model of hyperbolic space and map Euclidean input features to embeddings in hyperbolic spaces with different trainable curvature at each layer. Experiments demonstrate that HGCN learns embeddings that preserve hierarchical structure, and leads to improved performance when compared to Euclidean analogs, even with very low dimensional embeddings: compared to state-of-the-art GCNs, HGCN achieves an error reduction of up to 63.1% in ROC AUC for link prediction and of up to 47.5% in F1 score for node classification, also improving state-of-the art on the Pubmed dataset.


Effective methods to gain adversarial robustness usually involve adversarial training #REFR , which leverages adversarial examples to train models.

Adversarial examples and adversarial training.

Adversarial examples deceive machine learning models by adding adversarial perturbations, often imperceptible to human, to source images, leading to incorrect classification results #OTHEREFR .

White-box adversarial attacks #OTHEREFR maximize the loss of a source image with gradient descent on the defending model to add adversarial perturbations onto an image to maximize its loss on the model.

Adversarial training algorithms thus solve the min-max problem of minimizing the loss function for most adversarial examples within a perturbation budget, typically bounded in p .

Recent years have thus observed an arms race between adversarial attack strategies and defense mechanisms #OTHEREFR .

Data poisoning attacks manipulate the training of a deep learning model by injecting malicious and poisoned examples into its training set #OTHEREFR .

Data poisoning methods #OTHEREFR achieve their malicious objectives by stealthily replacing a portion of training data, and successful attacks can be triggered with specially-crafted prescribed inputs.

Learning The Unlearnable: Adversarial Augmentations Suppress Unlearnable Example Attacks

Abstract: Unlearnable example attacks are data poisoning techniques that can be used to safeguard public data against unauthorized training of deep learning models. These methods add stealthy perturbations to the original image, thereby making it difficult for deep learning models to learn from these training data effectively. Current research suggests that adversarial training can, to a certain degree, mitigate the impact of unlearnable example attacks, while common data augmentation methods are not effective against such poisons. Adversarial training, however, demands considerable computational resources and can result in non-trivial accuracy loss. In this paper, we introduce the UEraser method, which outperforms current defenses against different types of state-of-the-art unlearnable example attacks through a combination of effective data augmentation policies and loss-maximizing adversarial augmentations. In stark contrast to the current SOTA adversarial training methods, UEraser uses adversarial augmentations, which extends beyond the confines of p perturbation budget assumed by current unlearning attacks and defenses. It also helps to improve the model's generalization ability, thus protecting against accuracy loss. UEraser wipes out the unlearning effect with error-maximizing data augmentations, thus restoring trained model accuracies. Interestingly, UEraser-Lite, a fast variant without adversarial augmentations, is also highly effective in preserving clean accuracies. On challenging unlearnable CIFAR-10, CIFAR-100, SVHN, and ImageNet-subset datasets produced with various attacks, it achieves results that are comparable to those obtained during clean training. We also demonstrate the efficacy of UEraser against possible adaptive attacks. Our code is open source and available to the deep learning community 1 .

Towards Deep Learning Models Resistant To Adversarial Attacks

Abstract: Recent work has demonstrated that neural networks are vulnerable to adversarial examples, i.e., inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network. In fact, some of the latest findings suggest that the existence of adversarial attacks may be an inherent weakness of deep learning models. To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization. This approach provides us with a broad and unifying view on much of the prior work on this topic. Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal. In particular, they specify a concrete security guarantee that would protect against any adversary. These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks. They also suggest the notion of security against a first-order adversary as a natural and broad security guarantee. We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models.

adversarial training
adversarial robustness


For instance, training a BERT-base model without hyperparameter tuning, which is 1590 times smaller than ChatGPT, requires a CO2 emission of 650kg, which is comparable to flying from New York to San Francisco for one passenger #REFR .

Question: Do both sentences mean the same thing? Answer:" PERTURBED OUTPUT: "No, the two sentences do not mean the same thing.

The first sentence says that he asked for six pizzas and soda, while the second sentence says that he asked for six pepperoni pizzas and two six-packs of soft drinks.

The type of pizza and the specific drinks requested are different in the two sentences." TASK: SNLI, Semantic-BECEL ORIGINAL QUESTION: "An enthusiastic boy pushes a shopping card containing a smiling boy. Question: A boy is pushing a shopping cart.

True, False or Neither? Answer:" ORIGINAL OUTPUT: "True" PERTURBED QUESTION: "An enthusiastic boy pushes a shopping card containing a smiling boy. Question: A young man is pushing a shopping cart.

True, False or Neither? Answer:" PERTURBED OUTPUT: "Neither (the description does not mention the age or gender of the person pushing the shopping cart, only that the person inside the cart is a smiling boy.) " et al., 2021).

A simple expectation of CO2 emission for re-training ChatGPT is 1033t, while a human is responsible for 5t CO2 emission per year.

Therefore, it is desirable to enlarge our viewpoint beyond LLMs to implement sustainable remedies that can fundamentally solve the inconsistency problem, particularly in a modern society facing the global climate crisis.

Consistency Analysis Of Chatgpt

Abstract: ChatGPT, a question-and-answer dialogue system based on a large language model, has gained huge popularity since its introduction. Its positive aspects have been reported through many media platforms, and some analyses even showed that ChatGPT achieved a decent grade in professional exams, including the law, medical, and finance domains, adding extra support to the claim that AI now can assist and, even, replace humans in industrial fields. Others, however, doubt its reliability and trustworthiness. In this paper, we investigate Chat-GPT's trustworthiness regarding logically consistent behaviours. Our findings suggest that, although ChatGPT seems to achieve an improved language understanding ability, it still fails to generate logically correct predictions frequently. Hence, while it is true that Chat-GPT is an impressive and promising new technique, we conclude that its usage in real-world applications without thorough human inspection requires further consideration, especially for risk-sensitive areas.

Energy And Policy Considerations For Deep Learning In Nlp

Abstract: Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data. These models have obtained notable gains in accuracy across many NLP tasks. However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption. As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware. In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP. Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.


We introduced least fixed formulae that use recursion on second-order function symbols and provided logical characterizations of SpanL and TotP, answering an open question of #REFR . Furthermore, we determined logics that capture SpanPSPACE and FPSPACE.

Inspired by the two-step semantics developed in the context of weighted logics, we introduced two-step semantics that enriches the existing framework of quantitative logics, i.e. logics for expressing counting problems.

Some of the quantitative logics we introduced in this work naturally capture classes of functions that count different valid outputs of space-restricted transducers.

The logic that captures TotP over finite ordered structures was defined in a more complicated way that is related to the properties of TotP problems: recursion of the logic expresses self-reducibility and the restricted form of the recursion captures the easy-decision property.

It would be interesting to investigate whether TotP is captured by a simpler, more elegant logic.

The two-step semantics that we propose in this work is noteworthy for reasons beyond its primary objective.

It can be generalized to map formulae to elements of any structure S equipped with operations ∪ and •, instead of solely sets of strings. Conversely, it can also be specialized.

Counting Computations With Formulae: Logical Characterisations Of Counting Complexity Classes

Abstract: We present quantitative logics with two-step semantics based on the framework of quantitative logics introduced by Arenas et al. (2020) and the two-step semantics defined in the context of weighted logics by Gastin & Monmege (2018). We show that some of the fragments of our logics augmented with a least fixed point operator capture interesting classes of counting problems. Specifically, we answer an open question in the area of descriptive complexity of counting problems by providing logical characterizations of two subclasses of #P, namely SpanL and TotP, that play a significant role in the study of approximable counting problems. Moreover, we define logics that capture FPSPACE and SpanPSPACE, which are counting versions of PSPACE.

Descriptive Complexity For Counting Complexity Classes

Abstract: Descriptive Complexity has been very successful in characterizing complexity classes of decision problems in terms of the properties definable in some logics. However, descriptive complexity for counting complexity classes, such as FP and #P, has not been systematically studied, and it is not as developed as its decision counterpart. In this paper, we propose a framework based on Weighted Logics to address this issue. Specifically, by focusing on the natural numbers we obtain a logic called Quantitative Second Order Logics (QSO), and show how some of its fragments can be used to capture fundamental counting complexity classes such as FP, #P and FPSPACE, among others. We also use QSO to define a hierarchy inside #P, identifying counting complexity classes with good closure and approximation properties, and which admit natural complete problems. Finally, we add recursion to QSO, and show how this extension naturally captures lower counting complexity classes such as #L.


However, these works focus on channel models such as binary input additive white Gaussian noise (BIAWGN) that do not match the ET-QKD channel #REFR .

Hence, baseline NB-LDPC codes with large field sizes are not favorable in QKD applications requiring low latency, such as in #OTHEREFR , #OTHEREFR .

key rate trade-off, the LDPC codes used previously in the IR step of ET-QKD protocols have not fully utilized the properties of the ET-QKD channel.

For example, #OTHEREFR used a standard LDPC ensemble without optimization.

Similarly, spatially-coupled (SC) LDPC codes, irregular repeat accumulate (IRA) codes, SC-IRA codes, and multi-edge-type (MET) codes have been discussed for the continuous-variable (CV) QKD #OTHEREFR , #OTHEREFR .

A unique property of the ET-QKD problem considered in this paper is that the key rate of the system is closely dependent on both the rate of the code and the frame error rate (FER) performance. Fig.

2 shows the FER and key rates obtained by a random LDPC code for different values of rate.

From this graph, we see that increasing the code rate can improve the key rate even at the cost of higher FER, a phenomenon we see in both binary and non-binary LDPC codes.

Additionally, the maximum in the key rate occurs for a relatively large value of FER (∼ 5%).

While the conventional code design approach is to minimize the FER to a very small value for a given rate, in this case, the goal is to jointly optimize both the rate and the FER to achieve the largest key rate.

Non-Binary Ldpc Code Design For Energy-Time Entanglement Quantum Key Distribution

Abstract: In energy-time entanglement Quantum Key Distribution (QKD), two users extract a shared secret key from the arrival times (discretized as symbols) of entangled photon pairs. In prior work, Zhou et al. proposed a multi-level coding (MLC) scheme that splits the observed symbols into bit layers and utilizes binary Low-Density Parity-Check (LDPC) codes for reconciliation of the symbols. While binary LDPC codes offer low latency for key generation, splitting the symbols into bits results in a loss of key generation rate due to error propagation. Additionally, existing LDPC codes do not fully utilize the properties of the QKD channel to optimize the key rates. In this paper, we mitigate the above issues by first generalizing the MLC scheme to a nonbinary(NB) MLC scheme that has layers with non-binary symbols and utilizes NB-LDPC codes. We show the NB-MLC scheme offers flexibility in system design. Additionally, we show that the NB-MLC scheme with a small symbol size per layer offers the best trade-off between latency and key rate. We then propose a framework to jointly optimize the rate and degree profile of the NB-LDPC codes that is tailored towards the QKD channel resulting in higher key rates than prior work.

Efficient Information Reconciliation For Energy-Time Entanglement Quantum Key Distribution

Abstract: Graph based codes such as low density parity check (LDPC) codes have been shown promising for the information reconciliation phase in quantum key distribution (QKD). However, existing graph coding schemes have not fully utilized the properties of the QKD channel. In this work, we first investigate the channel statistics for discrete variable (DV) QKD based on energy-time entangled photons. We then establish a so-called balanced modulation scheme that is promising for this channel. Based on the modulation, we propose a joint local-global graph coding scheme that is expected to achieve good error-correction performance.