In summary, the conventional UPW model is valid for most practical cases in terms of the received power, but the accurate near-field modelling needs to be considered when the IRS size becomes significantly large, especially for the asymptotic analysis, which is consistent with the conclusions in #REFR . Fig.

The transmit SNR isP = 90 dB, and the BS and user are assumed to be located at q = [10, 0, 0] T m and p = [100, 0, 0] T m, respectively.

It is firstly observed that for UPA-based XL-IRS, the derived bounds in Lemma 1 are sufficiently tight for the SNR estimation. In particular, Figs.

7(b) and 7(c) validate our closed-form bounds in #OTHEREFR and #OTHEREFR .

Furthermore, as the IRS size L becomes large, the SNR in model in #OTHEREFR is approximately consistent with our developed near-field model when the IRS size is not very large.

However, as the IRS size exceeds a certain threshold, such two models exhibit drastically different scaling laws, i.e., converging to a constant value versus increasing unboundedly.

8 plots the SNR versus the link distance r q between the BS and the IRS center, where the square IRS size is set as L y = L z 8 m.

We compare the resulting SNR with the following expressions, i.e., the summation form in #OTHEREFR , generic lower-and upper-bounds in #OTHEREFR , and the square power scaling law under the conventional UPW model in #OTHEREFR .

The BS direction is set as (θ q , φ q ) = π 3 , π 6 , and the user is located at (r p , θ p , φ p ) = 200 m, 3π 4 , − π 5 .

It is observed that the SNR bounds given in Theorem 1 are rather accurate, and the conventional UPW model in general over-estimates the SNR values when taking into account the directional gain patterns of IRS's reflecting elements, as illustrated in Fig. 8(b) and 8(c). In particular, Fig.

8(c) reveals that the SNR does not necessarily monotonically decrease with the distance from the BS to the XL-IRS, because it also depends on the gain pattern of each element.

Near-Field Modelling And Performance Analysis For Extremely Large-Scale Irs Communications

Abstract: Intelligent reflecting surface (IRS) is an emerging technology for wireless communications, thanks to its powerful capability to engineer the radio environment. However, in practice, this benefit is attainable only when the passive IRS is of sufficiently large size, for which the conventional uniform plane wave (UPW)-based far-field model may become invalid. In this paper, we pursue a near-field modelling and performance analysis for wireless communications with extremely large-scale IRS (XL-IRS). By taking into account the directional gain pattern of IRS's reflecting elements and the variations in signal amplitude across them, we derive both the lower-and upper-bounds of the resulting signal-to-noise ratio (SNR) for the generic uniform planar array (UPA)-based XL-IRS. Our results reveal that, instead of scaling quadratically and unboundedly with the number of reflecting elements M as in the conventional UPW-based model, the SNR under the new non-uniform spherical wave (NUSW)-based model increases with M with a diminishing return and eventually converges to a certain limit. To gain more insights, we

Power Scaling Laws And Near-Field Behaviors Of Massive Mimo And Intelligent Reflecting Surfaces

Abstract: Large arrays might be the solution to the capacity problems in wireless communications. The signal-to-noise ratio (SNR) grows linearly with the number of array elements $N$ when using Massive MIMO receivers/relays. Moreover, intelligent reflecting surfaces (IRSs) have recently attracted attention since their SNR grows as $N^2$, which seems like a major benefit. In this paper, we use a deterministic propagation model for a planar array of arbitrary size, to demonstrate that the mentioned SNR behaviors, and associated power scaling laws, only apply in the far-field. They cannot be used to study the regime where $N\to\infty$. We derive an exact channel gain expression that captures the near-field behavior and use it to revisit the power scaling laws. We derive new finite asymptotic SNR limits but also conclude that these are unlikely to be approached in practice. We further prove that an IRS setup cannot achieve a higher SNR than the corresponding Massive MIMO setups, despite its faster SNR growth. The IRS typically must have a much larger array size to achieve the same SNR. Finally, we show that an optimized IRS can be interpreted as a reconfigurable lens and that it is generally suboptimal to operate it as an "anomalous" mirror.


Continually Training Approximation Models

This can result in overfitting to certain orientations, and catastrophic forgetting #REFR for others that may soon be ranked highly.

To cope with such data drift, MadEye employs continual learning (every 400-500 ms) to update the model's weights using the latest query results on orientations sent to the server for full workload inference.

While continual learning has been applied to edge video analytics #OTHEREFR , MadEye requires several alterations from prior efforts.

The main challenge is that within each retraining window, samples are only available for the orientations that MadEye's camera-side component recently visited and deemed worthy of backend inference.

Since orientations are typically best for short total times ( §2.3), there is often severe imbalance in the orientations covered by new training samples.

For instance, with perfect rankings, the average 2-minute window sees only 9.3% of orientations get sent to the backend.

To deal with this, MadEye retrieves the most recent historical training samples from each orientation and uses this to balance the dataset.

As we will discuss in §3.3, we find that orientation shifts are often spatially localized, with changes to distant orientations happening over longer timescales.

Thus, MadEye pads the data samples for neighboring orientations (up to 3 away from the latest one) to match the count for the most popular orientation in the retraining window.

The remaining orientations use an exponentially declining number of samples based on their distance from the latest orientation.

Madeye: Boosting Live Video Analytics Accuracy With Adaptive Camera Configurations

Abstract: Camera orientations (i.e., rotation and zoom) govern the content that a camera captures in a given scene, which in turn heavily influences the accuracy of live video analytics pipelines. However, existing analytics approaches leave this crucial adaptation knob untouched, instead opting to only alter the way that captured images from fixed orientations are encoded, streamed, and analyzed. We present MadEye, a camera-server system that automatically and continually adapts orientations to maximize accuracy for the workload and resource constraints at hand. To realize this using commodity pan-tilt-zoom (PTZ) cameras, MadEye embeds (1) a search algorithm that rapidly explores the massive space of orientations to identify a fruitful subset at each time, and (2) a novel knowledge distillation strategy to efficiently (with only camera resources) select the ones that maximize workload accuracy. Experiments on diverse workloads show that MadEye boosts accuracy by 2.9-25.7% for the same resource usage, or achieves the same accuracy with 2-3.7× lower resource costs.

Measuring Catastrophic Forgetting In Neural Networks

Abstract: Deep neural networks are used in many state-of-the-art systems for machine perception. Once a network is trained to do a specific task, e.g., bird classification, it cannot easily be trained to do new tasks, e.g., incrementally learning to recognize additional bird species or learning an entirely different task such as flower recognition. When new tasks are added, typical deep neural networks are prone to catastrophically forgetting previous tasks. Networks that are capable of assimilating new information incrementally, much like how humans form new memories over time, will be more efficient than re-training the model from scratch each time a new task needs to be learned. There have been multiple attempts to develop schemes that mitigate catastrophic forgetting, but these methods have not been directly compared, the tests used to evaluate them vary considerably, and these methods have only been evaluated on small-scale problems (e.g., MNIST). In this paper, we introduce new metrics and benchmarks for directly comparing five different mechanisms designed to mitigate catastrophic forgetting in neural networks: regularization, ensembling, rehearsal, dual-memory, and sparse-coding. Our experiments on real-world images and sounds show that the mechanism(s) that are critical for optimal performance vary based on the incremental training paradigm and type of data being used, but they all demonstrate that the catastrophic forgetting problem has yet to be solved.

certain orientations
catastrophic forgetting


Based on the results of the bitrate consumption and compression performance presented in Table R1 , it can be found that QVRF only has an impact on the compression of the latent representation, which requires a fixed number of bits for the side information. #REFR as the benchmark. Table R1 .

Results of bitrate consumption and compression performance on compressing kodim01 using the single model of QVRF with Cheng et al. #OTHEREFR for discrete bitrates.

Qvrf: A Quantization-Error-Aware Variable Rate Framework For Learned Image Compression

Abstract: Learned image compression has exhibited promising compression performance, but variable bitrates over a wide range remain a challenge. State-of-the-art variable rate methods compromise the loss of model performance and require numerous additional parameters. In this paper, we present a Quantization-error-aware Variable Rate Framework (QVRF) that utilizes a univariate quantization regulator a to achieve wide-range variable rates within a single model. Specifically, QVRF defines a quantization regulator vector coupled with predefined Lagrange multipliers to control quantization error of all latent representation for discrete variable rates. Additionally, the reparameterization method makes QVRF compatible with a round quantizer. Exhaustive experiments demonstrate that existing fixed-rate VAE-based methods equipped with QVRF can achieve wide-range continuous variable rates within a single model without significant performance degradation. Furthermore, QVRF outperforms contemporary variable-rate methods in rate-distortion performance with minimal additional parameters.

Learned Image Compression With Discretized Gaussian Mixture Likelihoods And Attention Modules

Abstract: Image compression is a fundamental research field and many well-known compression standards have been developed for many decades. Recently, learned compression methods exhibit a fast development trend with promising results. However, there is still a performance gap between learned compression algorithms and reigning compression standards, especially in terms of widely used PSNR metric. In this paper, we explore the remaining redundancy of recent learned compression algorithms. We have found accurate entropy models for rate estimation largely affect the optimization of network parameters and thus affect the rate-distortion performance. Therefore, in this paper, we propose to use discretized Gaussian Mixture Likelihoods to parameterize the distributions of latent codes, which can achieve a more accurate and flexible entropy model. Besides, we take advantage of recent attention modules and incorporate them into network architecture to enhance the performance. Experimental results demonstrate our proposed method achieves a state-of-the-art performance compared to existing learned compression methods on both Kodak and high-resolution datasets. To our knowledge our approach is the first work to achieve comparable performance with latest compression standard Versatile Video Coding (VVC) regarding PSNR. More importantly, our approach generates more visually pleasant results when optimized by MS-SSIM. This project page is at this https URL https://github.com/ZhengxueCheng/Learned-Image-Compression-with-GMM-and-Attention


Our results however do not directly contradict the results of #REFR itself, since it does not compare with the fast tools identified here and stands to a large degree on parametric and random benchmarks.

Although the mediocre performance of the other tools can be partially explained by their focus on a different kind of a problem or a dated underlying technology, and each of them is respectable in its own right, a point can be made against relying on them as a baseline in comparisons of tools for solving our kind of problem.

MONA, opti-mised for a different settings (complex alphabets of bit-vectors with many minterms), is held back by the implicit determinisation, and, in our case, probably by the overhead of the symbolic representation.

It also frequently runs out of the 32-bit address space for BDD nodes. Similarly for BRICS, which also always determinises.

The low performance of BISIM is surprising relative to the good results of the up-to algorithms reported in #OTHEREFR .

It is more consistent with #OTHEREFR where up-to algorithms were not wining against antichains on the more practical examples.

There is also always the possibility that we have prepared the input in a way not ideal for the tool.

For instance, transformation to the separated AFA, required by BISIM, is not entirely trivial.

Further investigation of this and a comparison with some other implementation of the up-to techniques seems to be needed.

The lack of a raw speed of JALTIMPACT on BRE and AFA emptiness is expectable considering that it is meant for a different kind of systems, AFA over data words.

The stable trends shown in the graphs suggest that an implementation of an interpolation-based abstraction refinement optimised for BRE and AFA emptiness might have a potential.

Reasoning About Regular Properties: A Comparative Study

Abstract: Several new algorithms for deciding emptiness of Boolean combinations of regular languages and of languages of alternating automata (AFA) have been proposed recently, especially in the context of analysing regular expressions and in string constraint solving. The new algorithms demonstrated a significant potential, but they have never been systematically compared, neither among each other nor with the state-of-the art implementations of existing (non)deterministic automata-based methods. In this paper, we provide the first such comparison as well as an overview of the existing algorithms and their implementations. We collect a diverse benchmark mostly originating in or related to practical problems from string constraint solving, analysing LTL properties, and regular model checking, and evaluate collected implementations on it. The results reveal the best tools and hint on what the best algorithms and implementation techniques are. Roughly, although some advanced algorithms are fast, such as antichain algorithms and reductions to IC3/PDR, they are not as overwhelmingly dominant as sometimes presented and there is no clear winner. The simplest NFA-based technology may be actually the best choice, depending on the problem source and implementation style. Our findings should be highly relevant for development of these techniques as well as for related fields such as string constraint solving.

A Symbolic Decision Procedure For Symbolic Alternating Finite Automata

Abstract: We introduce Symbolic Alternating Finite Automata (s-AFA) as an expressive, succinct, and decidable model for describing sets of finite sequences over arbitrary alphabets. Boolean operations over s-AFAs have linear complexity, which is in sharp contrast with the quadratic cost of intersection and union for non-alternating symbolic automata. Due to this succinctness, emptiness and equivalence checking are PSpace-hard. We introduce an algorithm for checking the equivalence of two s-AFAs based on bisimulation up to congruence. This algorithm allows us to exploit the power of SAT and SMT solvers to efficiently search the state space of the s-AFAs. We evaluate our decision procedure on two verification and security applications: 1) checking satisfiability of linear temporal logic formulas over finite traces, and 2) checking equivalence of Boolean combinations of regular expressions. Our experiments show that our technique often outperforms existing techniques and it can be beneficial in both such applications.


In #REFR , GP was combined with a local search but this is harder to re-implement in a nondeterministic robotic scenario.

Evolutionary approaches to learn BTs were first applied to computer games #OTHEREFR , #OTHEREFR , #OTHEREFR - #OTHEREFR but later years have also seen it applied to robotic applications #OTHEREFR , #OTHEREFR , #OTHEREFR - #OTHEREFR .

Authors in #OTHEREFR , #OTHEREFR successfully use Grammatical Evolution (GE).

With GE, the design of the grammar requires domain knowledge and the engineering effort grows with the complexity of the task.

Further, the disconnect between genotype and phenotype can make analysis and implementing heuristics difficult and hinders locality #OTHEREFR , #OTHEREFR .

In #OTHEREFR , #OTHEREFR , #OTHEREFR , structural and dynamic constraints were implemented in the GP, speeding up the learning by preventing the generation of undesirable trees. Similar constraints are also used in our implementation.

Most notably, #OTHEREFR presented a method to combine the results from a planner with a GP algorithm.

An adaptation of that method is what we use in the proposed approach of this paper to seed the GP with the information gathered in the demonstrations.

There has been some previous work on combining LfD and BTs #OTHEREFR , #OTHEREFR , #OTHEREFR - #OTHEREFR .

The method proposed in #OTHEREFR , #OTHEREFR , and #OTHEREFR learns a Decision Tree (DT) to map from state space to action space.

The DT is then converted into a BT using the fact that BTs generalize DTs #OTHEREFR .

A Framework For Learning Behavior Trees In Collaborative Robotic Applications

Abstract: In modern industrial collaborative robotic applications, it is desirable to create robot programs automatically, intuitively, and time-efficiently. Moreover, robots need to be controlled by reactive policies to face the unpredictability of the environment they operate in. In this paper we propose a framework that combines a method that learns Behavior Trees (BTs) from demonstration with a method that evolves them with Genetic Programming (GP) for collaborative robotic applications. The main contribution of this paper is to show that by combining the two learning methods we obtain a method that allows non-expert users to semi-automatically, time-efficiently, and interactively generate BTs. We validate the framework with a series of manipulation experiments. The BT is fully learnt in simulation and then transferred to a real collaborative robot.

Learning Of Behavior Trees For Autonomous Agents

Abstract: Definition of an accurate system model for Automated Planner (AP) is often impractical, especially for real-world problems. Conversely, off-the-shelf planners fail to scale up and are domain dependent. These drawbacks are inherited from conventional transition systems such as Finite State Machines (FSMs) that describes the action-plan execution generated by the AP. On the other hand, Behavior Trees (BTs) represent a valid alternative to FSMs presenting many advantages in terms of modularity, reactiveness, scalability and domain-independence. In this paper, we propose a model-free AP framework using Genetic Programming (GP) to derive an optimal BT for an autonomous agent to achieve a given goal in unknown (but fully observable) environments. We illustrate the proposed framework using experiments conducted with an open source benchmark Mario AI for automated generation of BTs that can play the game character Mario to complete a certain level at various levels of difficulty to include enemies and obstacles.


Specifically, we use a FiLM layer #REFR at every 3 WaveNet layers to fuse the condition information processed by the second Q-K-V attention in the prompting mechanism in the diffusion model.

The pitch and duration predictor share the same architecture of 30-layer 1D convolution with ReLU activation and layer normalization, 10 Q-K-V attention layers for in-context learning, which have 512 hidden dimensions and 8 attention heads and are placed every 3 1D convolution layers.

Specifically, we set the dropout to 0.5 in the duration predictor and 0.2 in the pitch predictor.

For the speech prompt encoder, we use a 6-layer Transformer with 512 hidden size, which has the same architecture as the phoneme encoder.

As for the m query tokens in the first Q-K-V attention in the prompting mechanism in the diffusion model (as shown in Figure 3 ), we set the token number m to 32 and the hidden dimension to 512.

The diffusion model contains 40 WaveNet layers #OTHEREFR , which consist of 1D dilated convolution layers with 3 kernel size, 1024 filter size, and 2 dilation size.

The hidden size in WaveNet is 512, and the dropout rate is 0.2.

More details of the model configurations are shown in Appendix A.

Naturalspeech 2: Latent Diffusion Models Are Natural And Zero-Shot Speech And Singing Synthesizers

Abstract: Scaling text-to-speech (TTS) to large-scale, multi-speaker, and in-the-wild datasets is important to capture the diversity in human speech such as speaker identities, prosodies, and styles (e.g., singing). Current large TTS systems usually quantize speech into discrete tokens and use language models to generate these tokens one by one, which suffer from unstable prosody, word skipping/repeating issue, and poor voice quality. In this paper, we develop NaturalSpeech 2, a TTS system that leverages a neural audio codec with residual vector quantizers to get the quantized latent vectors and uses a diffusion model to generate these latent vectors conditioned on text input. To enhance the zero-shot capability that is important to achieve diverse speech synthesis, we design a speech prompting mechanism to facilitate in-context learning in the diffusion model and the duration/pitch predictor. We scale NaturalSpeech 2 to large-scale datasets with 44K hours of speech and singing data and evaluate its voice quality on unseen speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, robustness, and voice quality in a zero-shot setting, and performs novel zero-shot singing synthesis with only a speech prompt. Audio samples are available at https://speechresearch.github.io/naturalspeech2.

Film: Visual Reasoning With A General Conditioning Layer

Abstract: We introduce a general-purpose conditioning method for neural networks called FiLM: Feature-wise Linear Modulation. FiLM layers influence neural network computation via a simple, feature-wise affine transformation based on conditioning information. We show that FiLM layers are highly effective for visual reasoning - answering image-related questions which require a multi-step, high-level process - a task which has proven difficult for standard deep learning methods that do not explicitly model reasoning. Specifically, we show on visual reasoning tasks that FiLM layers 1) halve state-of-the-art error for the CLEVR benchmark, 2) modulate features in a coherent manner, 3) are robust to ablations and architectural modifications, and 4) generalize well to challenging, new data from few examples or even zero-shot.


The second model is Attention UNet #REFR , a U-Net model with attention gates introduced in the skipped connections.

We first evaluated our proposed methods on PI-CAI validation sets.

For comparable methods, we report results from 3 other state-of-the-art CNN segmentation models and 1 state-of-the-art transformer model.

The first one is the basic UNet structure #OTHEREFR without the attention mechanism.

Attention gates allow the network learning to suppress irrelevant regions in an input image while highlighting salient features useful for a specific task.

The third model is DynUNet implemented in MONAI, which followed nnUNet architecture proposed by #OTHEREFR .

The transformer model we compare to is Swin UNETR #OTHEREFR which has a Swin transformer encoder and UNet-like architecture.

Swin UNETR achieved state-of-the-art results in Brain Tumor Segmentation (BraTS) and Beyond the Cranial Vault (BTCV) datasets.

We facilitated a fair comparison by maintaining an identical preprocessing, augmentation, tuning and train-validation pipeline for each candidate system in a given experiment.

Cross-Shaped Windows Transformer With Self-Supervised Pretraining For Clinically Significant Prostate Cancer Detection In Bi-Parametric Mri

Abstract: Multiparametric magnetic resonance imaging (mpMRI) has demonstrated promising results in prostate cancer (PCa) detection using deep convolutional neural networks (CNNs). Recently, transformers have achieved competitive performance compared to CNNs in computer vision. Large-scale transformers need abundant annotated data for training, which are difficult to obtain in medical imaging. Self-supervised learning can effectively leverage unlabeled data to extract useful semantic representations without annotation and its associated costs. This can improve model performance on downstream tasks with limited labelled data and increase generalizability. We introduce a novel end-to-end Cross-Shaped windows (CSwin) transformer UNet model, CSwin UNet, to detect clinically significant prostate cancer (csPCa) in prostate bi-parametric MR imaging (bpMRI) and demonstrate the effectiveness of our proposed selfsupervised pre-training framework. Using a large prostate bpMRI dataset with 1500 patients, we first pretrain CSwin transformer using multi-task self-supervised learning to improve data-efficiency and network generalizability. We then finetuned using lesion annotations to perform csPCa detection. Five-fold cross validation shows that self-supervised CSwin UNet achieves 0.888±0.010 AUC and 0.545±0.060 Average Precision (AP), significantly outperforming four state-of-the-art models (Swin UNETR, DynUNet, Attention UNet, UNet). Using a separate bpMRI dataset with 158 patients, we evaluated our method's robustness to external hold-out data. Self-supervised CSwin UNet achieves 0.79 AUC and 0.45 AP, still outperforming all other comparable methods and demonstrating generalization to a dataset shift.

Attention Gated Networks: Learning To Leverage Salient Regions In Medical Images

Abstract: Graphical abstract Figure. No Caption available. ABSTRACT We propose a novel attention gate (AG) model for medical image analysis that automatically learns to focus on target structures of varying shapes and sizes. Models trained with AGs implicitly learn to suppress irrelevant regions in an input image while highlighting salient features useful for a specific task. This enables us to eliminate the necessity of using explicit external tissue/organ localisation modules when using convolutional neural networks (CNNs). AGs can be easily integrated into standard CNN models such as VGG or U‐Net architectures with minimal computational overhead while increasing the model sensitivity and prediction accuracy. The proposed AG models are evaluated on a variety of tasks, including medical image classification and segmentation. For classification, we demonstrate the use case of AGs in scan plane detection for fetal ultrasound screening. We show that the proposed attention mechanism can provide efficient object localisation while improving the overall prediction performance by reducing false positives. For segmentation, the proposed architecture is evaluated on two large 3D CT abdominal datasets with manual annotations for multiple organs. Experimental results show that AG models consistently improve the prediction performance of the base architectures across different datasets and training sizes while preserving computational efficiency. Moreover, AGs guide the model activations to be focused around salient regions, which provides better insights into how model predictions are made. The source code for the proposed AG models is publicly available.


Finally, a new set learning module #REFR aggregates subgraph representations obtained after a global pooling operation into a single one used in downstream tasks.

We adapt the binarization threshold to obtain a set of explanations with K edges, where K ranges from 5% to 75% of the initial number of edges in the original graph.

An example of these two strategies is presented in Figure 2 , on the synthetic dataset BA-2Motifs where the five nodes cycle is responsible for the prediction.

In the third step, we insert the backbone GNN into the DSS-GNN model introduced in ESAN in #OTHEREFR .

The backbone works as the encoder that processes each explanation subgraph independently.

A new message-passing module preprocesses the aggregation of the explanations working as an information-sharing module for the subgraphs.

Besides being used by the information-sharing module, the subgraph obtained by aggregating all the explanations is used to explain the model's prediction.

We would like to stress the fact that this explanation, rather than being computed just from a post-hoc explainer, is used to train the model.

Combining Stochastic Explainers And Subgraph Neural Networks Can Increase Expressivity And Interpretability

Abstract: Subgraph-enhanced graph neural networks (SGNN) can increase the expressive power of the standard message-passing framework. This model family represents each graph as a collection of subgraphs, generally extracted by random sampling or with hand-crafted heuristics. Our key observation is that by selecting "meaningful" subgraphs, besides improving the expressivity of a GNN, it is also possible to obtain interpretable results. For this purpose, we introduce a novel framework that jointly predicts the class of the graph and a set of explanatory sparse subgraphs, which can be analyzed to understand the decision process of the classifier. We compare the performance of our framework against standard subgraph extraction policies, like random node/edge deletion strategies. The subgraphs produced by our framework allow to achieve comparable performance in terms of accuracy, with the additional benefit of providing explanations.

On Learning Sets Of Symmetric Elements

Abstract: Learning from unordered sets is a fundamental learning setup, which is attracting increasing attention. Research in this area has focused on the case where elements of the set are represented by feature vectors, and far less emphasis has been given to the common case where set elements themselves adhere to certain symmetries. That case is relevant to numerous applications, from deblurring image bursts to multi-view 3D shape recognition and reconstruction. ::: In this paper, we present a principled approach to learning sets of general symmetric elements. We first characterize the space of linear layers that are equivariant both to element reordering and to the inherent symmetries of elements, like translation in the case of images. We further show that networks that are composed of these layers, called Deep Sets for Symmetric elements layers (DSS), are universal approximators of both invariant and equivariant functions. DSS layers are also straightforward to implement. Finally, we show that they improve over existing set-learning architectures in a series of experiments with images, graphs, and point-clouds.

subgraph representations
new set learning


We used the script from to tokenize the data and used SentencePiece #REFR to segment the tokens.

We followed the scripts in FairSeq to preprocess the data.

The WMT DE→EN dataset is larger, with 1.4M sentence pairs.

We followed the preprocessing steps outlined in Takase and Kiyono (2021) by tokenizing the data with Moses tokenizer and then processing it with BPE #OTHEREFR .

The model was trained on the WMT-14 training set and evaluated on the test set from years 2010 to 2016, following Takase and Kiyono (2021) .

The OPUS-100 dataset is a large-scale multilingual dataset containing 100 languages and approximately 55M sentence pairs.

All data processing scripts are available in the Appendix G.

Model Our model is implemented using the FairSeq framework.

We follow the convention of using the same model size as previous works.

Notably, our method does not introduce additional parameters to the vanilla Transformer network.

We trained our models using the Adam (Kingma and Ba, 2014) optimizer with β = (0.9, 0.98), = and used the invert_sqrt learning rate scheduler with warm up, unless specified otherwise For detailed training hyper-parameters, please refer to the Appendix G.

Residual: Transformer With Dual Residual Connections

Abstract: Transformer networks have become the preferred architecture for many tasks due to their state-of-the-art performance. However, the optimal way to implement residual connections in Transformer, which are essential for effective training, is still debated. Two widely used variants are the Post-Layer Normalization (Post-LN) and Pre-Layer Normalization (Pre-LN) Transformers, which apply layer normalization after each residual block's output or before each residual block's input, respectively. While both variants enjoy their advantages, they also suffer from severe limitations: Post-LN causes gradient vanishing issue that hinders training deep Transformers, and Pre-LN causes representation collapse issue that limits model capacity. In this paper, we propose ResiDual, a novel Transformer architecture with Pre-Post-LN (PPLN), which fuses the connections in Post-LN and Pre-LN together, and inherits their advantages while avoids their limitations. We conduct both theoretical analyses and empirical experiments to verify the effectiveness of ResiDual. Theoretically, we prove that ResiDual has a lower bound on the gradient to avoid the vanishing issue due to the residual connection from Pre-LN. Moreover, ResiDual also has diverse model representations to avoid the collapse issue due to the residual connection from Post-LN. Empirically, ResiDual outperforms both Post-LN and Pre-LN on several machine translation benchmarks across different network depths and data sizes. Thanks to the good theoretical and empirical performance, ResiDual Transformer can serve as a foundation architecture for different AI models (e.g., large language models). Our code is available at https://github.com/microsoft/ResiDual.

Sentencepiece: A Simple And Language Independent Subword Tokenizer And Detokenizer For Neural Text Processing

Abstract: This paper describes SentencePiece, a language-independent subword tokenizer and detokenizer designed for Neural-based text processing, including Neural Machine Translation. It provides open-source C++ and Python implementations for subword units. While existing subword segmentation tools assume that the input is pre-tokenized into word sequences, SentencePiece can train subword models directly from raw sentences, which allows us to make a purely end-to-end and language independent system. We perform a validation experiment of NMT on English-Japanese machine translation, and find that it is possible to achieve comparable accuracy to direct subword training from raw sentences. We also compare the performance of subword training and segmentation with various configurations. SentencePiece is available under the Apache 2 license at https://github.com/google/sentencepiece.


This means that D JS (p, q) is equal to the average amount of information in one randomly sampled word token about which of the two distributions it was sampled from #REFR .

The JSD has a number of properties that are useful for its interpretation as a distance.

It is symmetric, non-negative, and equal to 0 if and only if p = q.

Furthermore, D JS (p, q) satisfies the triangle inequality and is thus a metric #OTHEREFR .

Additionally, the JSD between distributions p and q is equivalent to the mutual information of variables sampled from p and q.

In this paper, we predominately considered a generalisation of the JSD whereby H in Equation (6) is replaced by the generalised entropy of order α #OTHEREFR

This generalisation, first introduced in Ref.

#OTHEREFR , yields a spectrum of divergence measures D α parameterised by α.

When α = 1, we recover the usual Jensen-Shannon divergence D JS .

Quantifying The Dissimilarity Of Texts

Abstract: Quantifying the dissimilarity of two texts is an important aspect of a number of natural language processing tasks, including semantic information retrieval, topic classification, and document clustering. In this paper, we compared the properties and performance of different dissimilarity measures D using three different representations of texts-vocabularies, word frequency distributions, and vector embeddings-and three simple tasks-clustering texts by author, subject, and time period. Using the Project Gutenberg database, we found that the generalised Jensen-Shannon divergence applied to word frequencies performed strongly across all tasks, that D's based on vector embedding representations led to stronger performance for smaller texts, and that the optimal choice of approach was ultimately task-dependent. We also investigated, both analytically and numerically, the behaviour of the different D's when the two texts varied in length by a factor h. We demonstrated that the (natural) estimator of the Jaccard distance between vocabularies was inconsistent and computed explicitly the h-dependency of the bias of the estimator of the generalised Jensen-Shannon divergence applied to word frequencies. We also found numerically that the Jensen-Shannon divergence and embedding-based approaches were robust to changes in h, while the Jaccard distance was not.

Similarity Of Symbol Frequency Distributions With Heavy Tails

Abstract: Quantifying the similarity between symbolic sequences is a traditional problem in Information Theory which requires comparing the frequencies of symbols in different sequences. In numerous modern applications, ranging from DNA over music to texts, the distribution of symbol frequencies is characterized by heavy-tailed distributions (e.g., Zipf's law). The large number of low-frequency symbols in these distributions poses major difficulties to the estimation of the similarity between sequences, e.g., they hinder an accurate finite-size estimation of entropies. Here we show analytically how the systematic (bias) and statistical (fluctuations) errors in these estimations depend on the sample size~$N$ and on the exponent~$\gamma$ of the heavy-tailed distribution. Our results are valid for the Shannon entropy $(\alpha=1)$, its corresponding similarity measures (e.g., the Jensen-Shanon divergence), and also for measures based on the generalized entropy of order $\alpha$. For small $\alpha$'s, including $\alpha=1$, the errors decay slower than the $1/N$-decay observed in short-tailed distributions. For $\alpha$ larger than a critical value $\alpha^* = 1+1/\gamma \leq 2$, the $1/N$-decay is recovered. We show the practical significance of our results by quantifying the evolution of the English language over the last two centuries using a complete $\alpha$-spectrum of measures. We find that frequent words change more slowly than less frequent words and that $\alpha=2$ provides the most robust measure to quantify language change.


Technical Evolution Of Gpt-Series Models

In 2017, the Transformer model #REFR was introduced by Google, and the OpenAI team quickly adapted their language modeling work to this new neural network architecture.

Overall, the research of OpenAI on LLMs can be roughly divided into the following stages 10 .

According to one interview with Ilya Sutskever #OTHEREFR (a co-founder and chief scientist of OpenAI), the idea of approaching intelligent systems with language models was already explored in the early days of Ope-nAI, while it was attempted with recurrent neural networks (RNN) #OTHEREFR .

With the advent of Transformer, OpenAI developed two initial GPT models, namely GPT-1 #OTHEREFR and GPT-2 #OTHEREFR , which can considered as the foundation to more powerful models subsequently i.e., GPT-3 and GPT-4.

They released the first GPT model in 2018, i.e., GPT-1 #OTHEREFR , and coined the abbreviation term GPT as the model name, standing for Generative Pre-Training.

GPT-1 was developed based on a generative, decoder-only Transformer architecture, and adopted a hybrid approach of unsupervised pretraining and supervised fine-tuning.

GPT-1 has set up the core architecture for the GPT-series models and established the underlying principle to model natural language text, i.e., predicting the next word.

Following a similar architecture of GPT-1, GPT-2 #OTHEREFR increased the parameter scale to 1.5B, which was trained with a large webpage dataset WebText.

A Survey Of Large Language Models

Abstract: Ever since the Turing Test was proposed in the 1950s, humans have explored the mastering of language intelligence by machine. Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable artificial intelligence (AI) algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pretraining Transformer models over large-scale corpora, showing strong capabilities in solving various natural language processing (NLP) tasks. Since the researchers have found that model scaling can lead to an improved model capacity, they further investigate the scaling effect by increasing the parameter scale to an even larger size. Interestingly, when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance improvement, but also exhibit some special abilities (e.g., incontext learning) that are not present in small-scale language models (e.g., BERT). To discriminate the language models in different parameter scales, the research community has coined the term large language models (LLM) for the PLMs of significant size (e.g., containing tens or hundreds of billions of parameters). Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the launch of ChatGPT (a powerful AI chatbot developed based on LLMs), which has attracted widespread attention from society. The technical evolution of LLMs has been making an important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. Considering this rapid technical progress, in this survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular, we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides, we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions. This survey provides an up-to-date review of the literature on LLMs, which can be a useful resource for both researchers and engineers.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


Rehearsal based methods #REFR save a few exemplars from the previous dataset and replay them or use them to constrain the model during the new training sessions.

Incremental learning aims to develop methods that can learn new knowledge from new data while not forgetting previous knowledge learned from the previous training stages.

The existing methods can mainly be categorized as three types: distillation based, dynamic model based and rehearsal based #OTHEREFR .

Distillation based methods focus on how to limit the change of the model by applying a loss on the weights directly #OTHEREFR , or on the output features #OTHEREFR and probabilities #OTHEREFR .

Dynamic model based methods #OTHEREFR extend the architecture of the network to learn new knowledge from the new incoming data distribution.

Previously, incremental learning mainly focused on image classification problems.

Recently, the community also developed incremental learning algorithms for other problems such as image generation #OTHEREFR , segmentation #OTHEREFR , object detection #OTHEREFR , video classification #OTHEREFR .

But to the best of our knowledge, there is no work for incremental learning of counting problems yet.

Density Map Distillation For Incremental Object Counting

Abstract: We investigate the problem of incremental learning for object counting, where a method must learn to count a variety of object classes from a sequence of datasets. A naïve approach to incremental object counting would suffer from catastrophic forgetting, where it would suffer from a dramatic performance drop on previous tasks. In this paper, we propose a new exemplar-free functional regularization method, called Density Map Distillation (DMD). During training, we introduce a new counter head for each task and introduce a distillation loss to prevent forgetting of previous tasks. Additionally, we introduce a cross-task adaptor that projects the features of the current backbone to the previous backbone. This projector allows for the learning of new features while the backbone retains the relevant features for previous tasks. Finally, we set up experiments of incremental learning for counting new objects. Results confirm that our method greatly reduces catastrophic forgetting and outperforms existing methods.

Icarl: Incremental Classifier And Representation Learning

Abstract: A major open problem on the road to artificial intelligence is the development of incrementally learning systems that learn about more and more concepts over time from a stream of data. In this work, we introduce a new training strategy, iCaRL, that allows learning in such a class-incremental way: only the training data for a small number of classes has to be present at the same time and new classes can be added progressively. iCaRL learns strong classifiers and a data representation simultaneously. This distinguishes it from earlier works that were fundamentally limited to fixed data representations and therefore incompatible with deep learning architectures. We show by experiments on CIFAR-100 and ImageNet ILSVRC 2012 data that iCaRL can learn many classes incrementally over a long period of time where other strategies quickly fail.


To investigate the effectiveness of our method, we mainly employ the cascade MVS- Net as the baseline and use two branches on the basis of Cas-MVSNet #REFR to predict the depth and signed distance, respectively.

In this section, we introduce the detailed structure of the proposed RA-MVSNet.

As shown in Figure 2 , the overall framework mainly consists of three parts, including cost volume construction, the multi-scale depth map and signed distance prediction, and handling of ground truth.

Since our proposed point-to-surface distance supervision employs an additional branch to calculate the signed distance of the sampling points around the surface through cost volume, it is easy to be added into the existing learning-based MVSNet scheme with slight changes.

Multi-View Stereo Representation Revist: Region-Aware Mvsnet

Abstract: Deep learning-based multi-view stereo has emerged as a powerful paradigm for reconstructing the complete geometrically-detailed objects from multi-views. Most of the existing approaches only estimate the pixel-wise depth value by minimizing the gap between the predicted point and the intersection of ray and surface, which usually ignore the surface topology. It is essential to the textureless regions and surface boundary that cannot be properly reconstructed. To address this issue, we suggest to take advantage of point-to-surface distance so that the model is able to perceive a wider range of surfaces. To this end, we predict the distance volume from cost volume to estimate the signed distance of points around the surface. Our proposed RA-MVSNet is patch-awared, since the perception range is enhanced by associating hypothetical planes with a patch of surface. Therefore, it could increase the completion of textureless regions and reduce the outliers at the boundary. Moreover, the mesh topologies with fine details can be generated by the introduced distance volume. Comparing to the conventional deep learning-based multiview stereo methods, our proposed RA-MVSNet approach obtains more complete reconstruction results by taking advantage of signed distance supervision. The experiments on both the DTU and Tanks & Temples datasets demonstrate that our proposed approach achieves the state-of-the-art results.

Cascade Cost Volume For High-Resolution Multi-View Stereo And Stereo Matching

Abstract: The deep multi-view stereo (MVS) and stereo matching approaches generally construct 3D cost volumes to regularize and regress the output depth or disparity. These methods are limited when high-resolution outputs are needed since the memory and time costs grow cubically as the volume resolution increases. In this paper, we propose a both memory and time efficient cost volume formulation that is complementary to existing multi-view stereo and stereo matching approaches based on 3D cost volumes. First, the proposed cost volume is built upon a standard feature pyramid encoding geometry and context at gradually finer scales. Then, we can narrow the depth (or disparity) range of each stage by the depth (or disparity) map from the previous stage. With gradually higher cost volume resolution and adaptive adjustment of depth (or disparity) intervals, the output is recovered in a coarser to fine manner. ::: We apply the cascade cost volume to the representative MVS-Net, and obtain a 23.1% improvement on DTU benchmark (1st place), with 50.6% and 74.2% reduction in GPU memory and run-time. It is also the state-of-the-art learning-based method on Tanks and Temples benchmark. The statistics of accuracy, run-time and GPU memory on other representative stereo CNNs also validate the effectiveness of our proposed method.


GIN is a well-known typical GNN that was expected to achieve the ability as the Weisfeiler-Lehman graph isomorphism test, but a graph-level readout function of GIN was specially designed to produce the embedding of the entire graph for graph classification tasks #REFR .

• GCN #OTHEREFR , GAT #OTHEREFR , GraphSAGE #OTHEREFR , and snowball GCN (SBGCN) #OTHEREFR : There are typical GNN models that were not specifically designed for graph classification.

GCN is based on a variant of CNNs which operate directly on graphs. GAT is a GNN architecture using masked self-attentional layers.

GraphSAGE is based on randomly edges sampling to enhance the ability of capturing global structure.

SBGCN is a GCN architecture using a snowball structure to capture multi-scale information.

• DiffPool #OTHEREFR , GIN #OTHEREFR , PGCN #OTHEREFR , and MRGNN #OTHEREFR : There are graph neural networks which have a special design for graph classification.

DiffPool is a differentiable graph pooling module that could generate hierarchical representations for graph classification.

PGCN introduced the polynomial graph convolution layer to independently exploit neighbouring nodes at different topological distances, and used subsequent readout layers to generate graph representation.

MRGNN used a shallow readout function to generate an unsupervised graph representation.

• GraphTrans #OTHEREFR , Graphormer #OTHEREFR , and Spectral Attention Network (SAN) #OTHEREFR : They are three well-known graph-transformer based models.

GraphTrans applied graph convolution operation to introduce local patterns and structure information of graphs.

Transformer And Snowball Graph Convolution Learning For Biomedical Graph Classification

Abstract: Graph or network has been widely used for describing and modeling complex systems in biomedicine. Deep learning methods, especially graph neural networks (GNNs), have been developed to learn and predict with such structured data. In this paper, we proposed a novel transformer and snowball encoding networks (TSEN) for biomedical graph classification, which introduced transformer architecture with graph snowball connection into GNNs for learning whole-graph representation. TSEN combined graph snowball connection with graph transformer by snowball encoding layers, which enhanced the power to capture multi-scale information and global patterns to learn the whole-graph features. On the other hand, TSEN also used snowball graph convolution as position embedding in transformer structure, which was a simple yet effective method for capturing local patterns naturally. Results of experiments using four graph classification datasets demonstrated that TSEN outperformed the state-of-the-art typical GNN models and the graph-transformer based GNN models. Keywords Graph neural networks • Transformer • Snowball graph convolutional networks 1 Introduction Graph or network is pervasive in describing and modeling complex systems in biomedicine, such as molecular graphs, multi-omics graphs, and pharmaceutical graphs. Recently, a growing number of deep learning methods, especially graph neural networks (GNNs) have been developed for learning and predicting on biomedical graph data, such as predicting the function of proteins in an interaction network, and predicting the toxicity or biological activity profile of a candidate drug [1, 2, 3]. Graph classification is one type of the most prominent learning tasks of GNNs, which typically learns a graph-level representation for graph classification in an end-to-end fashion. To learn an effective representation, GNNs were expected to capture the global patterns as well as the local patterns and their interaction [4]. There have been various attempts to enhance the expressive power of GNNs, for example, enlarging the receptive field [5], using effective graph pooling operations [6, 7], and relieving the over-smoothing caused by increased network depth [8, 9]. Compared with data from other fields, biomedical graphs are usually complex and noisy, and the data size is usually limited, since collecting biomedical graph data usually requires time consuming and laborious wet experiments [2]. In the view of biomedical graph data, it is difficult to develop targeted graph-level representation learning approaches to produce compact vector representations which can be optimized for the classification task. Recently, transformer [10] has been shown excel performance in nature language processing (NLP), such as Bert [11] and GPT [12], and this architecture was also introduced for graph structure data in biomedicine [13]. To date, transformer has been shown to be good at capturing the long-range dependency and global information in graph [14]. And, most researchers tried to properly incorporate structural information of graphs into GNN models with transformer, such

How Powerful Are Graph Neural Networks?

Abstract: Graph Neural Networks (GNNs) are an effective framework for representation learning of graphs. GNNs follow a neighborhood aggregation scheme, where the representation vector of a node is computed by recursively aggregating and transforming representation vectors of its neighboring nodes. Many GNN variants have been proposed and have achieved state-of-the-art results on both node and graph classification tasks. However, despite GNNs revolutionizing graph representation learning, there is limited understanding of their representational properties and limitations. Here, we present a theoretical framework for analyzing the expressive power of GNNs to capture different graph structures. Our results characterize the discriminative power of popular GNN variants, such as Graph Convolutional Networks and GraphSAGE, and show that they cannot learn to distinguish certain simple graph structures. We then develop a simple architecture that is provably the most expressive among the class of GNNs and is as powerful as the Weisfeiler-Lehman graph isomorphism test. We empirically validate our theoretical findings on a number of graph classification benchmarks, and demonstrate that our model achieves state-of-the-art performance.


In particular, it was shown in #REFR that a self-attention on CNN features was effective in camera pose regression.

Since Transformer #OTHEREFR made a great success in the literature on natural language process, its self-attention, one of the essential elements in Transformer, has been utilized in many studies for computer vision applications.

We use the same self-attention, but its input consists of image features computed from an input image and point cloud features computed from an input 2D point cloud.

The input of the self-attention f i , which is a column vector, is first projected to generate query W q f i , key W k f i , and value W v f i by three learnable projections.

Then, the value is weighted based on the normalized correlations between the query and the key.

The correlations are calculated using the softmax function φ, and these procedures can be represented as

Fusionloc: Camera-2D Lidar Fusion Using Multi-Head Self-Attention For End-To-End Serving Robot Relocalization

Abstract: As technology advances in autonomous mobile robots, mobile service robots have been actively used more and more for various purposes. Especially, serving robots have been not surprising products anymore since the COVID-19 pandemic. One of the practical problems in operating serving a robot is that it often fails to estimate its pose on a map that it moves around. Whenever the failure happens, servers should bring the serving robot to its initial location and reboot it manually. In this paper, we focus on end-to-end relocalization of serving robots to address the problem. It is to predict robot pose directly from only the onboard sensor data using neural networks. In particular, we propose a deep neural network architecture for the relocalization based on camera-2D LiDAR sensor fusion. We call the proposed method FusionLoc. In the proposed method, the multi-head selfattention complements different types of information captured by the two sensors to regress the robot pose. Our experiments on a dataset collected by a commercial serving robot demonstrate that FusionLoc can provide better performances than previous endto-end relocalization methods taking only a single image or a 2D LiDAR point cloud as well as a straightforward fusion method concatenating their features.

Atloc: Attention Guided Camera Localization

Abstract: Deep learning has achieved impressive results in camera localization, but current single-image techniques typically suffer from a lack of robustness, leading to large outliers. To some extent, this has been tackled by sequential (multi-images) or geometry constraint approaches, which can learn to reject dynamic objects and illumination conditions to achieve better performance. In this work, we show that attention can be used to force the network to focus on more geometrically robust objects and features, achieving state-of-the-art performance in common benchmark, even if using only a single image as input. Extensive experimental evidence is provided through public indoor and outdoor datasets. Through visualization of the saliency maps, we demonstrate how the network learns to reject dynamic objects, yielding superior global camera pose regression performance. The source code is avaliable at this https URL.


B. Differentiable Rendering-Based Camera Pose Estimation

Specifically, we first train the PointRend model on synthetic data generated by the SAPIEN #REFR robot simulator and then fine-tune it on real data to enhance its generalization ability.

L(ξ cb ) = (min(1, l π(exp(ξ cb )T bl l)) − M ) 2 , (3)

The above loss function is fully differentiable, we could use the gradient descent method with PyTorch #OTHEREFR Autograd to perform the optimization.

To obtain the robot arm mask M as the supervision for the differentiable rendering-based optimization, we leverage the off-the-shelf learning-based segmentation network PointRend #OTHEREFR to predict the mask from the RGB image.

This obviates the need for additional manual annotation efforts, making the entire pipeline fully automated.

There exist various methods for initializing the camera pose, such as traditional marker-based techniques #OTHEREFR , learning-based methods #OTHEREFR , or random initialization of the object pose within the frustum of the camera #OTHEREFR .

In our work, we adopt the PVNet #OTHEREFR to perform the pose initialization.

Given an RGB image of the robot arm with the initial joint pose, PVNet estimates the pre-defined 2D keypoint locations (obtained with farthest point sampling on the 3D object model) through pixelwise voting and subsequently solves the transformation using PnP #OTHEREFR .

By leveraging pixel-wise voting, PVNet can tackle the challenges of occlusion and truncation, which are often encountered in real-world hand-eye calibration scenarios.

Easyhec: Accurate And Automatic Hand-Eye Calibration Via Differentiable Rendering And Space Exploration

Abstract: Hand-eye calibration is a critical task in robotics, as it directly affects the efficacy of critical operations such as manipulation and grasping. Traditional methods for achieving this objective necessitate the careful design of joint poses and the use of specialized calibration markers, while most recent learning-based approaches using solely pose regression are limited in their abilities to diagnose inaccuracies. In this work, we introduce a new approach to hand-eye calibration called EasyHeC, which is markerless, white-box, and offers comprehensive coverage of positioning accuracy across the entire robot configuration space. We introduce two key technologies: differentiable rendering-based camera pose optimization and consistency-based joint space exploration, which enables accurate end-to-end optimization of the calibration process and eliminates the need for the laborious manual design of robot joint poses. Our evaluation demonstrates superior performance in synthetic and real-world datasets, enhancing downstream manipulation tasks by providing precise camera poses for locating and interacting with objects. The code is available at the project page: https://ootts.github.io/easyhec/.

Sapien: A Simulated Part-Based Interactive Environment

Abstract: Building home assistant robots has long been a pursuit for vision and robotics researchers. To achieve this task, a simulated environment with physically realistic simulation, sufficient articulated objects, and transferability to the real robot is indispensable. Existing environments achieve these requirements for robotics simulation with different levels of simplification and focus. We take one step further in constructing an environment that supports household tasks for training robot learning algorithm. Our work, SAPIEN, is a realistic and physics-rich simulated environment that hosts a large-scale set for articulated objects. Our SAPIEN enables various robotic vision and interaction tasks that require detailed part-level understanding.We evaluate state-of-the-art vision algorithms for part detection and motion attribute recognition as well as demonstrate robotic interaction tasks using heuristic approaches and reinforcement learning algorithms. We hope that our SAPIEN can open a lot of research directions yet to be explored, including learning cognition through interaction, part motion discovery, and construction of robotics-ready simulated game environment.


Ii. Blockchain-Based Process Orchestration

Multichain is similar to Chorchain, but it is also capable of generating smart contracts for Hyperledger Fabric #REFR .

Lorikeet #OTHEREFR is a model-driven engineering approach that integrates assets into business processes.

Lorikeet extends the BPMN 2.0 specification with support for asset registries and also transforms models into Solidity smart contracts.

The smart contracts handle the orchestration of the process as well as interactions with the tokens.

Chorchain #OTHEREFR is a tool that takes a BPMN choreography and generates an Ethereum smart contract that can be used to execute the model. ChorChain also includes a dedicated modelling tool.

The same authors also released two additional but related tools: Multi-Chain #OTHEREFR and FlexChain #OTHEREFR .

FlexChain can only produce Solidity smart contracts, but the user can also define a ruleset for each choreography.

If a condition in the ruleset is met, then an off-chain processor will perform its underlying action.

Our analysis showed that the process state, as well as the process trace, are easily recoverable from the process manager smart contracts for all the tools above.

The Baseline protocol #OTHEREFR is a developing open standard that allows enterprises to synchronize complex, multi-party business processes on distributed ledger technologies.

Business process workflows in Baseline are formed as state machines. The standard includes some essential and optional privacyrelated requirements.

Blockchain-Based, Confidentiality-Preserving Orchestration Of Collaborative Workflows

Abstract: Business process collaboration between independent parties can be challenging, especially if the participants do not have complete trust in each other. Tracking actions and enforcing the activity authorizations of participants via blockchain-hosted smart contracts is an emerging solution to this lack of trust, with most state-of-the-art approaches generating the orchestrating smart contract logic from BPMN models. However, as a significant drawback in comparison to centralized business process orchestration, smart contract state typically leaks potentially sensitive information about the state of the collaboration. We describe a novel approach where the process manager smart contract only stores cryptographic commitments to the state and checks zero-knowledge proofs on update proposals. We cover a representative subset of BPMN, support message passing commitments between participants and provide an opensource end-to-end implementation. Under our approach, no party external to the collaboration can gain trustable knowledge of the current state of a process instance (barring collusion with a participant), even if it has full access to the blockchain history.

Hyperledger Fabric: A Distributed Operating System For Permissioned Blockchains

Abstract: Hyperledger Fabric is a modular and extensible open-source system for deploying and operating permissioned blockchains. Fabric is currently used in more than 400 prototypes and proofs-of-concept of distributed ledger technology, as well as several production systems, across different industries and use cases. Starting from the premise that there are no"one-size-fits-all"solutions, Fabric is the first truly extensible blockchain system for running distributed applications. It supports modular consensus protocols, which allows the system to be tailored to particular use cases and trust models. Fabric is also the first blockchain system that runs distributed applications written in general-purpose programming languages, without systemic dependency on a native cryptocurrency. This stands in sharp contrast to existing blockchain platforms for running smart contracts that require code to be written in domain-specific languages or rely on a cryptocurrency. Furthermore, it uses a portable notion of membership for realizing the permissioned model, which may be integrated with industry-standard identity management. To support such flexibility, Fabric takes a novel approach to the design of a permissioned blockchain and revamps the way blockchains cope with non-determinism, resource exhaustion, and performance attacks. This paper describes Fabric, its architecture, the rationale behind various design decisions, its security model and guarantees, its most prominent implementation aspects, as well as its distributed application programming model. We further evaluate Fabric by implementing and benchmarking a Bitcoin-inspired digital currency. We show that Fabric achieves end-to-end throughput of more than 3500 transactions per second in certain popular deployment configurations, with sub-second latency.


However, the use of graph neural network for global relationship modeling incurs high computing and memory costs, and slow gradient decline #REFR .

Deep learning has emerged as a significant breakthrough in the field of machine learning, providing automatic feature learning from data #OTHEREFR .

HSI classification, in particular, benefits from deep learning models due to their ability to handle complex and nonlinear relationships between input data and output classes #OTHEREFR , leading to improved classification accuracy compared to traditional machine learning methods #OTHEREFR , #OTHEREFR .

To fully utilize the spectral-spatial features of HSI, joint spectral-spatial HSI feature extraction methods based on deep learning, such as recurrent neural network #OTHEREFR , deep residual network #OTHEREFR , #OTHEREFR , capsule networks #OTHEREFR , and transformer #OTHEREFR , have been proposed.

While these methods have yielded satisfactory results, challenges remain, such as the inability to capture global relationships between samples.

To address this issue, researchers have proposed using graph neural network to model the relationship between samples #OTHEREFR .

In this paper, we propose using a diffusion model to model the global relationship of samples and verify the effectiveness of this method through extensive comparative experiments.

Spectraldiff: Hyperspectral Image Classification With Spectral-Spatial Diffusion Models

Abstract: Hyperspectral image (HSI) classification is an important topic in the field of remote sensing, and has a wide range of applications in Earth science. HSIs contain hundreds of continuous bands, which are characterized by high dimension and high correlation between adjacent bands. The high dimension and redundancy of HSI data bring great difficulties to HSI classification. In recent years, a large number of HSI feature extraction and classification methods based on deep learning have been proposed. However, their ability to model the global relationships among samples in both spatial and spectral domains is still limited. In order to solve this problem, an HSI classification method with spectral-spatial diffusion models is proposed. The proposed method realizes the reconstruction of spectralspatial distribution of the training samples with the forward and reverse spectral-spatial diffusion process, thus modeling the global spatial-spectral relationship between samples. Then, we use the spectral-spatial denoising network of the reverse process to extract the unsupervised diffusion features. Features extracted by the spectral-spatial diffusion models can achieve cross-sample perception from the reconstructed distribution of the training samples, thus obtaining better classification performance. Experiments on three public HSI datasets show that the proposed method can achieve better performance than the state-of-the-art methods. The source code and the pre-trained spectral-spatial diffusion model will be publicly available at https://github.com/chenning0115/SpectralDiff. Index Terms-Deep neural network, hyperspectral image (HSI) classification, latent representation, feature extraction, diffusion model, deep generative model. I. INTRODUCTION H YPERSPECTRAL imaging is a cutting-edge technology that enables the acquisition of high-resolution spectral information of imaged objects. By integrating spatial and spectral reflectance information, each pixel in a hyperspectral

Band Attention Convolutional Networks For Hyperspectral Image Classification

Abstract: Redundancy and noise exist in the bands of hyperspectral images (HSIs). Thus, it is a good property to be able to select suitable parts from hundreds of input bands for HSIs classification methods. In this letter, a band attention module (BAM) is proposed to implement the deep learning based HSIs classification with the capacity of band selection or weighting. The proposed BAM can be seen as a plug-and-play complementary component of the existing classification networks which fully considers the adverse effects caused by the redundancy of the bands when using convolutional neural networks (CNNs) for HSIs classification. Unlike most of deep learning methods used in HSIs, the band attention module which is customized according to the characteristics of hyperspectral images is embedded in the ordinary CNNs for better performance. At the same time, unlike classical band selection or weighting methods, the proposed method achieves the end-to-end training instead of the separated stages. Experiments are carried out on two HSI benchmark datasets. Compared to some classical and advanced deep learning methods, numerical simulations under different evaluation criteria show that the proposed method have good performance. Last but not least, some advanced CNNs are combined with the proposed BAM for better performance.


ANN #REFR uses axial attention to capture remote subsurface context, etc. (3) Transformer-based decoder.

It mainly uses convolution to model contextual relationships, expand the perceptive field, extract local detail features, fuse multi-scale information, etc.

For example, PSPNet #OTHEREFR utilizes Pyramid pooling techniques to model contextual features and expand the perceptive field.

Some works #OTHEREFR , #OTHEREFR , #OTHEREFR use top-down connectivity to fuse multi-scale spatial and semantic information to enhance pixel representation and multi-scale object recognition. (2) Attention-based decoder.

It means that the decoder uses the attention technique to obtain the similarity between pixels, and provides a focus on the region of interest.

OCRNet #OTHEREFR uses attention to compute the similarity of pixel features to individual class features, and CCNet #OTHEREFR uses Criss-Cross Attention to capture longrange dependencies and model rich contextual features.

Some researchers have incorporated the transformer into decoder designs as a result of its ability to model global contextual features, capture long-range dependencies, and provide flexible prototype representations.

For example, Segmenter #OTHEREFR uses transformer to build global contextual relations and embeds the prototypes of categories into transformer to learn together with feature maps.

MaskFormer #OTHEREFR uses transformer's cross-attention to let the mask token learn the feature map's category representation. (4) MLP-based decoder.

Recently, the MLP-based decoder has received more and more attention, mainly because of two aspects, the lightweight and efficient property of the MLP architecture, and the development of the transformer backbone, which offers global extraction capabilities for the encoder.

SegFormer #OTHEREFR then proposed a lightweight multi-scale fusion MLP architecture that uses the transformer's backbone to arXiv:2305.00671v1 [cs.CV] 1 May 2023 achieve state-of-the-art performance on various datasets.

Prseg: A Lightweight Patch Rotate Mlp Decoder For Semantic Segmentation

Abstract: The lightweight MLP-based decoder has become increasingly promising for semantic segmentation. However, the channel-wise MLP cannot expand the receptive fields, lacking the context modeling capacity, which is critical to semantic segmentation. In this paper, we propose a parametric-free patch rotate operation to reorganize the pixels spatially. It first divides the feature map into multiple groups and then rotates the patches within each group. Based on the proposed patch rotate operation, we design a novel segmentation network, named PRSeg, which includes an off-the-shelf backbone and a lightweight Patch Rotate MLP decoder containing multiple Dynamic Patch Rotate Blocks (DPR-Blocks). In each DPR-Block, the fully connected layer is performed following a Patch Rotate Module (PRM) to exchange spatial information between pixels. Specifically, in PRM, the feature map is first split into the reserved part and rotated part along the channel dimension according to the predicted probability of the Dynamic Channel Selection Module (DCSM), and our proposed patch rotate operation is only performed on the rotated part. Extensive experiments on ADE20K, Cityscapes and COCO-Stuff 10K datasets prove the effectiveness of our approach. We expect that our PRSeg can promote the development of MLPbased decoder in semantic segmentation.

Asymmetric Non-Local Neural Networks For Semantic Segmentation

Abstract: The non-local module works as a particularly useful technique for semantic segmentation while criticized for its prohibitive computation and GPU memory occupation. In this paper, we present Asymmetric Non-local Neural Network to semantic segmentation, which has two prominent components: Asymmetric Pyramid Non-local Block (APNB) and Asymmetric Fusion Non-local Block (AFNB). APNB leverages a pyramid sampling module into the non-local block to largely reduce the computation and memory consumption without sacrificing the performance. AFNB is adapted from APNB to fuse the features of different levels under a sufficient consideration of long range dependencies and thus considerably improves the performance. Extensive experiments on semantic segmentation benchmarks demonstrate the effectiveness and efficiency of our work. In particular, we report the state-of-the-art performance of 81.3 mIoU on the Cityscapes test set. For a 256x128 input, APNB is around 6 times faster than a non-local block on GPU while 28 times smaller in GPU running memory occupation. Code is available at: this https URL.


In our work, the random label noises can be either (1) drawn from probability distributions before training steps (but re-sampled by mini-batch sampling of SGD) or (2) realized by the random variables per training iteration #REFR .

In this work, we aim at investigating the influence of random label noises to the implicit regularization under mini-batch sampling of SGD.

To simplify our research, we assume the training dataset as a set of vectors D = {x 1 , x 2 , x 3 , . . .

, x N }.The labelỹ i for every vector x i ∈ D is the noisy response of the true neural network f * (x) such that y i = y i + ε i , y i = f * (x i ), and E[ε i ] = 0, var [

where the label noise ε i is assumed to be an independent zero-mean random variable.

Thus learning is to estimate θ in f (x, θ) for approximating f * (x),

Stochastic Gradient Descent With Random Label Noises: Doubly Stochastic Models And Inference Stabilizer

Abstract: Random label noises (or observational noises) widely exist in practical machine learning settings. While previous studies primarily focus on the affects of label noises to the performance of learning, our work intends to investigate the implicit regularization effects of the label noises, under mini-batch sampling settings of stochastic gradient descent (SGD), with assumptions that label noises are unbiased. Specifically, we analyze the learning dynamics of SGD over the quadratic loss with unbiased label noises, where we model the dynamics of SGD as a stochastic differentiable equation (SDE) with two diffusion terms (namely a Doubly Stochastic Model). While the first diffusion term is caused by mini-batch sampling over the (label-noiseless) loss gradients as many other works on SGD [1, 2], our model investigates the second noise term of SGD dynamics, which is caused by mini-batch sampling over the label noises, as an implicit regularizer. Our theoretical analysis finds such implicit regularizer would favor some convergence points that could stabilize model outputs against perturbation of parameters (namely inference stability). Though similar phenomenon have been investigated in [3], our work

Co-Teaching: Robust Training Of Deep Neural Networks With Extremely Noisy Labels

Abstract: Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of clean labels and then those of noisy labels. Therefore in this paper, we propose a new deep learning paradigm called Co-teaching for combating with noisy labels. Namely, we train two deep neural networks simultaneously, and let them teach each other given every mini-batch: firstly, each network feeds forward all data and selects some data of possibly clean labels; secondly, two networks communicate with each other what data in this mini-batch should be used for training; finally, each network back propagates the data selected by its peer network and updates itself. Empirical results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models.


However, those attention models do not explicitly work at conserving the modalityspecific knowledge, missing the point raised by #REFR .

The module refactors the channels of each modality regarding how the intra-and inter-modality channels correlate. Based on the MMTM concept and inspired by H. #OTHEREFR

(2020) that used the split operation to improve the dynamic channel selection, Su et al. (2020) presented the MSAF approach.

The dynamic channel refactoring in such multimodal models may allow for fine-grained feature selection and limit corruption impact. Unlike previous approaches, modality attention #OTHEREFR , later updated by #OTHEREFR

(2020) , provides soft attention weights for each modality to balance modality importance in the final embedding based on their discriminating capabilities.

Again, such attention sounds to be a great approach to tackling punctually corrupted data.

Some transformers architectures tackle this aspect, conserving modality-specific knowledge through modality-specific streams and selfattention, and modality-shared knowledge thanks to modality-shared streams and cross-attention #OTHEREFR .

However, transformer architectures are known to be complex and heavy Han et al.

(2020) , which do not align with video-surveillance challenges, requiring close to real-time algorithms. Multimodal person ReID.

Most approaches for person ReID #OTHEREFR focus on the unimodal (RGB) (H.

#OTHEREFR and cross-modal #OTHEREFR settings. Few only focused on combining multimodal information. For example, J. #OTHEREFR used the contour information. used pose information.

Fusion For Visual-Infrared Person Reid In Real-World Surveillance Using Corrupted Multimodal Data -First Version Submitted To Ijcv

Abstract: Visible-infrared person re-identification (V-I ReID) seeks to match images of individuals captured over a distributed network of RGB and IR cameras. The task is challenging due to the significant differences between V and I modalities, especially under real-world conditions, where images are corrupted by, e.g, blur, noise, and weather. Despite their practical relevance, deep learning (DL) models for multimodal V-I ReID remain far less investigated than for single and cross-modal V to I settings. Moreover, state-of-art V-I ReID models cannot leverage corrupted modality information to sustain a high level of accuracy. In this paper, we propose an efficient model for multimodal V-I ReID-named Multimodal Middle Stream Fusion (MMSF)-that preserves modality-specific knowledge for improved robustness to corrupted multimodal images. In addition, three state-of-art attention-based multimodal fusion models are adapted to address corrupted multimodal data in V-I ReID, allowing for dynamic balancing of the importance of each modality. The literature typically reports ReID performance using clean datasets, but more recently, evaluation protocols have been proposed to assess the robustness of ReID models under challenging real-world scenarios, using data with realistic corruptions. However, these protocols are limited to unimodal V settings. For realistic evaluation of multimodal (and cross-modal) V-I person ReID models, we propose new challenging corrupted datasets for scenarios where V and I cameras are co-located (CL) and not co-located (NCL). Finally, the benefits of our Masking and Local Multimodal Data Augmentation (ML-MDA) strategy are explored to improve the robustness of ReID models to multimodal corruption. Our experiments on clean and corrupted versions of the SYSU-MM01, RegDB, and ThermalWORLD datasets indicate the multimodal V-I ReID models that are more likely to perform well in realworld operational conditions. In particular, our ML-MDA is an important strategy for a V-I person ReID system to sustain high accuracy and robustness when processing corrupted multimodal images. The multimodal ReID models provide the best accuracy and complexity trade-off under both CL and NCL settings and compared to state-of-art unimodal ReID systems, except for the Thermal-WORLD dataset due to its low-quality I. Our MMSF model outperforms every method under CL and NCL camera scenarios. GitHub code: https://github.com/art2611/MREiD-UCD-CCD.git.

Multimodal Machine Learning: A Survey And Taxonomy

Abstract: Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.

attention models
modalityspecific knowledge


Furthermore, we find out the recommendation performance can be improved by averaging the predictions yielded from different random seeds #REFR .

4) 7: end for 8: Predict h 0 via (h 1 , 1) (Eq.

16) 9: Obtain probability distribution ←ˆ( |h 0 ) (Eq.

8) 10: return However, a major problem with Algorithm 2 is the lengthened diffusion steps.

As we have to start from h until we reach h 0 , the time complexity of the inference procedure is increased by times.

To address this issue, we notice that is trained to directly predict h 0 based on any h (1≤ ≤ ), so it is able to directly predict h 0 from h without the need of the intermediate diffusion steps.

This is probably because different recommendation Obtain probability distribution ←ˆ( |h 0 ) (Eq. 8)

Algorithm 3 greatly saves computational cost by directly predicting h 0 from h .

As we normally set the size of random seeds list as a small number (i.e., 10 by default), the additional computational cost incurred by using different random seeds can be minimal.

Sequential Recommendation With Diffusion Models

Abstract: Generative models, such as Variational Auto-Encoder (VAE) and Generative Adversarial Network (GAN), have been successfully applied in sequential recommendation. These methods require sampling from probability distributions and adopt auxiliary loss functions to optimize the model, which can capture the uncertainty of user behaviors and alleviate exposure bias. However, existing generative models still suffer from the posterior collapse problem or the model collapse problem, thus limiting their applications in sequential recommendation. To tackle the challenges mentioned above, we leverage a new paradigm of the generative models, i.e., diffusion models, and present sequential recommendation with diffusion models (DiffRec), which can avoid the issues of VAE-and GAN-based models and show better performance. While diffusion models are originally proposed to process continuous image data, we design an additional transition in the forward process together with a transition in the reverse process to enable the processing of the discrete recommendation data. We also design a different noising strategy that only noises the target item instead of the whole sequence, which is more suitable for sequential recommendation. Based on the modified diffusion process, we derive the objective function of our framework using a simplification technique and design a denoise sequential recommender to fulfill the objective function. As the lengthened diffusion steps substantially increase the time complexity, we propose an efficient training strategy and an efficient inference strategy to reduce training and inference cost and improve recommendation diversity. Extensive experiment results on three public benchmark datasets verify the effectiveness of our approach and show that DiffRec outperforms the state-of-theart sequential recommendation models. 1 1 Our code will be available upon acceptance.

Large Scale Gan Training For High Fidelity Natural Image Synthesis

Abstract: Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal. To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities specific to such scale. We find that applying orthogonal regularization to the generator renders it amenable to a simple "truncation trick," allowing fine control over the trade-off between sample fidelity and variety by reducing the variance of the Generator's input. Our modifications lead to models which set the new state of the art in class-conditional image synthesis. When trained on ImageNet at 128x128 resolution, our models (BigGANs) achieve an Inception Score (IS) of 166.5 and Frechet Inception Distance (FID) of 7.4, improving over the previous best IS of 52.52 and FID of 18.6.

recommendation performance
different random seeds


We infer that it is due to the impact of the computation method of ROUGE score as it is an n-gram-based metric #REFR .

We infer that it is due to the impact of the simplification module in KIS-mBART.

Table 2 presents the results of all models on SPEK-TRUM.

We find a similar pattern that SIMCSUM outperforms all baselines.

We also compute the statistical significance of these results with the same procedure.

The SPEKTRUM results are on the lower side compared to the WIKIPEDIA results due to zero-shot adaptability, especially for R.

The SPEKTRUM summaries have higher FRE scores compared to WIKIPEDIA.

Interestingly, we find that all baselines perform lower than the GOLD summaries.

However, the SIMCSUM score is similar to the GOLD summaries.

Comparing the performance of mBART and KIS-mBART, KIS-mBART performs slightly lower than mBART for all scores except Rbecause only WIKIPEDIA is used for fine-tuning of both models in KIS-mBART. Human Evaluation.

We compare the SIMCSUM and mBART outputs for analyzing linguistic qualities because SIMCSUM's architecture is based on mBART.

Simcsum: Joint Learning Of Simplification And Cross-Lingual Summarization For Cross-Lingual Science Journalism

Abstract: Cross-lingual science journalism generates popular science stories of scientific articles different from the source language for a nonexpert audience. Hence, a cross-lingual popular summary must contain the salient content of the input document, and the content should be coherent, comprehensible, and in a local language for the targeted audience. We improve these aspects of cross-lingual summary generation by joint training of two high-level NLP tasks, simplification and cross-lingual summarization. The former task reduces linguistic complexity, and the latter focuses on cross-lingual abstractive summarization. We propose a novel multi-task architecture-SIMC-SUM consisting of one shared encoder and two parallel decoders jointly learning simplification and cross-lingual summarization. We empirically investigate the performance of SIMC-SUM by comparing it with several strong baselines over several evaluation metrics and by human evaluation. Overall, SIMCSUM demonstrates statistically significant improvements over the state-of-the-art on two non-synthetic cross-lingual scientific datasets. Furthermore, we conduct an in-depth investigation into the linguistic properties of generated summaries and an error analysis.

Better Summarization Evaluation With Word Embeddings For Rouge

Abstract: ROUGE is a widely adopted, automatic evaluation measure for text summarization. While it has been shown to correlate well with human judgements, it is biased towards surface lexical similarities. This makes it unsuitable for the evaluation of abstractive summarization, or summaries with substantial paraphrasing. We study the effectiveness of word embeddings to overcome this disadvantage of ROUGE. Specifically, instead of measuring lexical overlaps, word embeddings are used to compute the semantic similarity of the words used in summaries instead. Our experimental results show that our proposal is able to achieve better correlations with human judgements when measured with the Spearman and Kendall rank coefficients.


In the field of remote sensing image segmentation, UNet has a very important position #REFR .

Many elements in remote sensing images have some connections, such as rivers and buildings often appearing together, and roads and vehicles often appearing together.

Deep learning #OTHEREFR has shown amazing results in many fields.

UNet was mainly used for medical image segmentation when it was proposed.

After several years of development, it has also been widely used in the field of remote sensing, and has many different variants, which have been applied to different types of tasks.

The dataset I am using is the FloodNet Dataset #OTHEREFR , which is a multi category segmented remote sensing image dataset.

The data is collected with a small UAS platform, DJI Mavic Pro quadcopters, after Hurricane Harvey. The whole dataset has 2343 images.

The classic Unet network is divided into two parts: encoder and decoder, and the encoding and decoding processes are connected through residual connections to avoid information loss during the encoding process.

Deep Attention Unet: A Network Model With Global Feature Perception Ability

Abstract: Remote sensing image segmentation is a specific task of remote sensing image interpretation. A good remote sensing image segmentation algorithm can provide guidance for environmental protection, agricultural production, and urban construction. This paper proposes a new type of UNet image segmentation algorithm based on channel self attention mechanism and residual connection called. In my experiment, the new network model improved mIOU by 2.48% compared to traditional UNet on the FoodNet dataset. The image segmentation algorithm proposed in this article enhances the internal connections between different items in the image, thus achieving better segmentation results for remote sensing images with occlusion.

U-Net: Convolutional Networks For Biomedical Image Segmentation

Abstract: There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .