To reduce the variance of the backward process across different views, we use the same initialization noise x T and diffusion noises {ϵ t } #REFR t=T to generate all images in the same video.

We use the following two solutions to address the issue. Stochastic conditioning.

We find that using stochastic conditioning #OTHEREFR slightly improves the temporal flickering.

Specifically, at each step in the backward process described in (3), instead of using the previous frame x i−1 , we randomly sample the source view image x j from the set of prior frames x j ∼ Uniform({x k , • • • , x i−1 }).

Such a strategy encourages the diffusion model to be guided by all the previous frames, thus improving the temporal consistency. Fixing noises in the backward process.

The noises introduced during the backward process illustrated in (3) also contribute to the temporal inconsistency.

Neverthe-less, we observe noticeable artifacts if we fix all diffusion noises {ϵ t } 1 t=T during the backward process, as demonstrated in Figure 3 .

In practice, fixing the diffusion noises {ϵ t } t ′ t=T to a certain backward step t ′ alleviates the issue and improves the temporal consistency. #OTHEREFR

Consistent View Synthesis With Pose-Guided Diffusion Models

Abstract: https://poseguided-diffusion.github.io/ Outputs Input Figure 1. Consistent view synthesis via pose-guided diffusion model. (top) Given an input image and a sequence of camera poses, we present a pose-guided diffusion model to generate a sequence of frames that are photorealistic and 3D consistent. (bottom) Our proposed method can synthesize diverse sequences from the same set of inputs.

Neural Point-Based Graphics

Abstract: We present a new point-based approach for modeling the appearance of real scenes. The approach uses a raw point cloud as the geometric representation of a scene, and augments each point with a learnable neural descriptor that encodes local geometry and appearance. A deep rendering network is learned in parallel with the descriptors, so that new views of the scene can be obtained by passing the rasterizations of a point cloud from new viewpoints through this network. The input rasterizations use the learned descriptors as point pseudo-colors. We show that the proposed approach can be used for modeling complex scenes and obtaining their photorealistic views, while avoiding explicit surface estimation and meshing. In particular, compelling results are obtained for scene scanned using hand-held commodity RGB-D sensors as well as standard RGB cameras even in the presence of objects that are challenging for standard mesh-based modeling.


We use similar parameters setup to those in #REFR . Each audio clip contains 16,000 samples.

The first FC layer contains 128 hidden units, whereas the output FC layer consists of 80 units.

As for training, the batch size used is 64, and Adam Optimizer is applied with 1e-4 learning rate.

For checkpoint selection, an early stop is applied regarding the validation loss.

The α is sampled from a uniform distribution of interval [0, 0.2].

For downstream tasks in our experiments, Waveglow is selected as a neural vocoder.

The batch size used is 8 with Adam Optimizer of learning rate 1e-4.

For ground-truth acoustic feature extraction, Fast Fourier Transform (FFT) window size is selected to be 1024 with a hop size equal to 16 milliseconds (256 for 16000 sampling rate, and 128 for 8000 sampling rate).

As for the acoustic model, Tacotron2 #OTHEREFR is utilized to predict acoustic features.

The Prenet dimension is set to 256, and the number of embedding dimensions is selected to be 512 with a dimension of attention recurrent neural network (RNN) equal to 1024.

The dimension of decoder RNN is set to 1024 with the dimension of Postnet embedding equal to 512.

Sar: Self-Supervised Anti-Distortion Representation For End-To-End Speech Model

Abstract: In recent Text-to-Speech (TTS) systems, a neural vocoder often generates speech samples by solely conditioning on acoustic features predicted from an acoustic model. However, there are always distortions existing in the predicted acoustic features, compared to those of the groundtruth, especially in the common case of poor acoustic modeling due to lowquality training data. To overcome such limits, we propose a Self-supervised learning framework to learn an Anti-distortion acoustic Representation (SAR) to replace human-crafted acoustic features by introducing distortion prior to an auto-encoder pretraining process. The learned acoustic representation from the proposed framework is proved anti-distortion compared to the most commonly used mel-spectrogram through both objective and subjective evaluation.

Waveglow: A Flow-Based Generative Network For Speech Synthesis

Abstract: In this paper we propose WaveGlow: a flow-based network capable of generating high quality speech from mel-spectrograms. WaveGlow combines insights from Glow [1] and WaveNet [2] in order to provide fast, efficient and high-quality audio synthesis, without the need for auto-regression. WaveGlow is implemented using only a single network, trained using only a single cost function: maximizing the likelihood of the training data, which makes the training procedure simple and stable. Our PyTorch implementation produces audio samples at a rate of more than 500 kHz on an NVIDIA V100 GPU. Mean Opinion Scores show that it delivers audio quality as good as the best publicly available WaveNet implementation. All code will be made publicly available online [3].


Note that e (t) u,i essentially models the associations between clusters since it implicitly computes the interaction between cluster embeddings, analogous to the positional encoding #REFR .

Let v u denote the query node and set N u denotes its neighborhood, then we encode such dynamics into representation s

v (t) i;l is the i th column of V (t) l , e (t)

u,i is the attention coefficient which can be implemented in various ways (e.g., SA #OTHEREFR , GAT #OTHEREFR and GATv2 #OTHEREFR ).

Easydgl: Encode, Train And Interpret For Continuous-Time Dynamic Graph Learning

Abstract: Dynamic graphs arise in various real-world applications, and it is often welcomed to model the dynamics directly in continuous time domain for its flexibility. This paper aims to design an easy-to-use pipeline (termed as EasyDGL which is also due to its implementation by DGL toolkit) composed of three key modules with both strong fitting ability and interpretability. Specifically the proposed pipeline which involves encoding, training and interpreting: i) a temporal point process (TPP) modulated attention architecture to endow the continuous-time resolution with the coupled spatiotemporal dynamics of the observed graph with edge-addition events; ii) a principled loss composed of task-agnostic TPP posterior maximization based on observed events on the graph, and a task-aware loss with a masking strategy over dynamic graph, where the covered tasks include dynamic link prediction, dynamic node classification and node traffic forecasting; iii) interpretation of the model outputs (e.g., representations and predictions) with scalable perturbation-based quantitative analysis in the graph Fourier domain, which could more comprehensively reflect the behavior of the learned model. Extensive experimental results on public benchmarks show the superior performance of our EasyDGL for time-conditioned predictive tasks, and in particular demonstrate that EasyDGL can effectively quantify the predictive power of frequency content that a model learn from the evolving graph data.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


The initial learning rate is set as 2.5 × 10 −4 and is decreased using the polynomial decay with power of 0.9 as mentioned in #REFR , and the maximum iteration number is Table 2 : Performance gap between the adapted model and the fully-supervised(oracle)model.

To train the proposed adaptation model, in each train batch, we first forward the source image X s to optimize the segmentation network for L s seg in (9) and generate the output P s .

For the target image X t , we obtain the segmentation output P t and pass it along with P s for optimizing L OT in (7).

For the multi-level training objective in (10), we simply repeat the same procedure for each adaptation module.

Our method is implemented with the PyTorch library on Nvidia GPU GTX 2080Ti with 12GB memory.

To train the segmentation network, we use the Stochastic Gradient Descent(SGD) optimizer with Nesterov acceleration where the momentum is 0.9 and the weight decay is 5 × 10 −4 .

We first compare results with state-of-the-art methods using the VGG based model, and then show our result using the ResNet one. Our baseline model is AdaptSegNet #OTHEREFR .

Other previous state-of-the-art methods include CDA #OTHEREFR , CyCADA #OTHEREFR .

Domain Adaptive Semantic Segmentation By Optimal Transport

Abstract: Scene segmentation is widely used in the field of autonomous driving for environment perception, and semantic scene segmentation (3S) has received a great deal of attention due to the richness of the semantic information it contains. It aims to assign labels to pixels in an image, thus enabling automatic image labeling. Current approaches are mainly based on convolutional neural networks (CNN), but they rely on a large number of labels. Therefore, how to use a small size of labeled data to achieve semantic segmentation becomes more and more important. In this paper, we propose a domain adaptation (DA) framework based on optimal transport (OT) and attention mechanism to address this issue. Concretely, first we generate the output space via CNN due to its superiority of feature representation. Second, we utilize OT to achieve a more robust alignment of source and target domains in output space, where the OT plan defines a well attention mechanism to improve the adaptation of the model. In particular, with OT, the number of network parameters has been reduced and the network has been better interpretable. Third, to better describe the multi-scale property of features, we construct a multi-scale segmentation network to perform domain adaptation. Finally, in order to verify the performance of our proposed method, we conduct experimental comparison with three benchmark and four SOTA methods on three scene datasets, and the mean intersection-over-union (mIOU) has been significant improved, and visualization results under multiple domain adaptation scenarios also show that our proposed method has better performance than compared semantic segmentation methods.

Deeplab: Semantic Image Segmentation With Deep Convolutional Nets, Atrous Convolution, And Fully Connected Crfs

Abstract: In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First , we highlight convolution with upsampled filters, or ‘atrous convolution’, as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second , we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third , we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed “DeepLab” system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7 percent mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online.


Finally, #REFR found that minimizing a stochastic network with a regularizer is equivalent to minimizing the cross-entropy over deterministic DNNs with multiplicative noise.

As a result, the mutual information between the input and the DNN representation in such objectives would be infinite, resulting in ill-posed optimization problems.

To avoid this problem, stochastic DNNs with variational bounds could be used, where the output of the deterministic network is used as the parameters of the conditional distribution #OTHEREFR ; Shwartz-Ziv and Alemi, 2020). Dubois et al.

(2021) assumed that the randomness of data augmentation among the two views is the source of stochasticity in the network.

Other work assumed a random input, but without making any assumptions about the properties of the distribution of the network's output, to analyze the objective and relied on general lower bounds (Wang and Isola, 2020; Zimmermann et al., 2021) . For supervised learning, #OTHEREFR

(2018) introduced an auxiliary (noisy) DNN by injecting additive noise into the model and demonstrated that the resulting model is a good proxy for the original (deterministic) DNN in terms of both performance and representation.

All of these methods assume that the source of randomness comes from the DNN, contradicting common practice.

An Information-Theoretic Perspective On Variance-Invariance-Covariance Regularization

Abstract: In this paper, we provide an information-theoretic perspective on Variance-Invariance-Covariance Regularization (VICReg) for self-supervised learning. To do so, we first demonstrate how information-theoretic quantities can be obtained for deterministic networks as an alternative to the commonly used unrealistic stochastic networks assumption. Next, we relate the VICReg objective to mutual information maximization and use it to highlight the underlying assumptions of the objective. Based on this relationship, we derive a generalization bound for VICReg, providing generalization guarantees for downstream supervised learning tasks and present new self-supervised learning methods, derived from a mutual information maximization objective, that outperform existing methods in terms of performance. This work provides a new information-theoretic perspective on self-supervised learning and Variance-Invariance-Covariance Regularization in particular and guides the way for improved transfer learning via information-theoretic self-supervised learning objectives.

Information Dropout: Learning Optimal Representations Through Noisy Computation

Abstract: The cross-entropy loss commonly used in deep learning is closely related to the defining properties of optimal representations, but does not enforce some of the key properties. We show that this can be solved by adding a regularization term, which is in turn related to injecting multiplicative noise in the activations of a Deep Neural Network, a special case of which is the common practice of dropout. We show that our regularized loss function can be efficiently minimized using Information Dropout, a generalization of dropout rooted in information theoretic principles that automatically adapts to the data and can better exploit architectures of limited capacity. When the task is the reconstruction of the input, we show that our loss function yields a Variational Autoencoder as a special case, thus providing a link between representation learning, information theory and variational inference. Finally, we prove that we can promote the creation of optimal disentangled representations simply by enforcing a factorized prior, a fact that has been observed empirically in recent work. Our experiments validate the theoretical intuitions behind our method, and we find that Information Dropout achieves a comparable or better generalization performance than binary dropout, especially on smaller models, since it can automatically adapt the noise to the structure of the network, as well as to the test sample.


To obtain knowledge for view images, facts (i.e., knowledge described by language descriptions) are retrieved from the knowledge base constructed on the Visual Genome dataset #REFR .

Second, the knowledge improves the generalization ability of the agent.

As the alignment between the instruction and the navigable candidate is learned in limited-seen environments, leveraging knowledge benefits the alignment in the unseen environment, as there is no specific regularity for target object arrangement. Third, knowledge increases the capability of VLN models.

As rich conceptual information is injected into VLN models, the correlations among numerous concepts are learned.

The learned correlations are able to benefit visual and language alignment, especially for tasks with high-level instructions.

In this work, we incorporate knowledge into the VLN task.

The retrieved facts by CLIP #OTHEREFR provide rich and complementary information for visual view images.

And then, a knowledge enhanced reasoning model (KERM) which leverages knowledge for sufficient interaction and better alignment between vision and language information is proposed.

Especially, the proposed KERM consists of a purification module, a fact-aware interaction module, and an instruction-guided aggregation module.

The purification model aims to extract key information in the fact representations, the visual region representations, and the historical representations respectively guided by the instruction.

The fact-aware interaction module allows visual and historical representations to obtain the interaction of the facts with cross-attention encoders.

Kerm: Knowledge Enhanced Reasoning For Vision-And-Language Navigation

Abstract: Vision-and-language navigation (VLN) is the task to enable an embodied agent to navigate to a remote location following the natural language instruction in real scenes. Most of the previous approaches utilize the entire features or object-centric features to represent navigable candidates. However, these representations are not efficient enough for an agent to perform actions to arrive the target location. As knowledge provides crucial information which is complementary to visible content, in this paper, we propose a Knowledge Enhanced Reasoning Model (KERM) to leverage knowledge to improve agent navigation ability. Specifically, we first retrieve facts (i.e., knowledge described by language descriptions) for the navigation views based on local regions from the constructed knowledge base. The retrieved facts range from properties of a single object (e.g., color, shape) to relationships between objects (e.g., action, spatial position), providing crucial information for VLN. We further present the KERM which contains the purification, fact-aware interaction, and instruction-guided aggregation modules to integrate visual, history, instruction, and fact features. The proposed KERM can automatically select and gather crucial and relevant cues, obtaining more accurate action prediction. Experimental results on the REVERIE, R2R, and SOON datasets demonstrate the effectiveness of the proposed method. The source code is available at https://github.com/XiangyangLi20/KERM .

Visual Genome: Connecting Language And Vision Using Crowdsourced Dense Image Annotations

Abstract: Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in an image. When asked "What vehicle is the person riding?", computers will need to identify the objects in an image as well as the relationships riding(man, carriage) and pulling(horse, carriage) to answer correctly that "the person is riding a horse-drawn carriage." In this paper, we present the Visual Genome dataset to enable the modeling of such relationships. We collect dense annotations of objects, attributes, and relationships within each image to learn these models. Specifically, our dataset contains over 108K images where each image has an average of $$35$$35 objects, $$26$$26 attributes, and $$21$$21 pairwise relationships between objects. We canonicalize the objects, attributes, relationships, and noun phrases in region descriptions and questions answer pairs to WordNet synsets. Together, these annotations represent the densest and largest dataset of image descriptions, objects, attributes, relationships, and question answer pairs.


It provides high persistence guarantees, higher read and write bandwidth and lower latencies than most persistent block devices #REFR .

The emergence of modern NVMM is a great opportunity to implement known designs and adapt them, or invent new ones.

We tried these two approaches with caching mechanisms for a file system stored in secondary storage.

Indeed, caching data for slower tier storage devices (SSD or HDD) is a great use case for NVMM.

In this study, we target applications that require a high level of data consistency, which would highly solicit a regular disk with frequent calls to fsync.

For such applications, we propose a persistent cache able to give fast persistence guarantees without having to wait for a slow secondary storage.

Nvmm Cache Design: Logging Vs. Paging

Abstract: Modern NVMM is closing the gap between DRAM and persistent storage, both in terms of performance and features. Having both byte addressability and persistence on the same device gives NVMM an unprecedented set of features, leading to the following question: How should we design an NVMMbased caching system to fully exploit its potential? We build two caching mechanisms, NVPages and NVLog, based on two radically different design approaches. NVPages stores memory pages in NVMM, similar to the Linux page cache (LPC). NVLog uses NVMM to store a log of pending write operations to be submitted to the LPC, while it ensures reads with a small DRAM cache. Our study shows and quantifies advantages and flaws for both designs.

Basic Performance Measurements Of The Intel Optane Dc Persistent Memory Module

Abstract: Scalable nonvolatile memory DIMMs will finally be commercially available with the release of the Intel Optane DC Persistent Memory Module (or just "Optane DC PMM"). This new nonvolatile DIMM supports byte-granularity accesses with access times on the order of DRAM, while also providing data storage that survives power outages. This work comprises the first in-depth, scholarly, performance review of Intel's Optane DC PMM, exploring its capabilities as a main memory device, and as persistent, byte-addressable memory exposed to user-space applications. This report details the technologies performance under a number of modes and scenarios, and across a wide variety of macro-scale benchmarks. Optane DC PMMs can be used as large memory devices with a DRAM cache to hide their lower bandwidth and higher latency. When used in this Memory (or cached) mode, Optane DC memory has little impact on applications with small memory footprints. Applications with larger memory footprints may experience some slow-down relative to DRAM, but are now able to keep much more data in memory. When used under a file system, Optane DC PMMs can result in significant performance gains, especially when the file system is optimized to use the load/store interface of the Optane DC PMM and the application uses many small, persistent writes. For instance, using the NOVA-relaxed NVMM file system, we can improve the performance of Kyoto Cabinet by almost 2x. Optane DC PMMs can also enable user-space persistence where the application explicitly controls its writes into persistent Optane DC media. In our experiments, modified applications that used user-space Optane DC persistence generally outperformed their file system counterparts. For instance, the persistent version of RocksDB performed almost 2x faster than the equivalent program utilizing an NVMM-aware file system.