On the one hand, our average flow is generally consistent with the flow estimated from RAFT #REFR , which approximates to the ground truth displacements.

(4) but estimates the residual part in the final interpolation result. As shown in Tab.

2f, the results of this variant underperform the original setting, which indicates we need to compensate details for each frame candidate separately. Besides, if we replace the convolution operators in Eqn.

#OTHEREFR with an average operation, the performance will be degraded (see Tab. 2f).

This indicates that it is important for our AMT to perform an adaptive fusion and refinement. Discussion.

For further discussion, we visualize the mean and deviation of three estimated flow pairs. The results are shown in Fig. 6 .

On the other hand, we observe that the major diversities of flows are at the motion boundaries and in the regions with rich textures.

This indicates that these regions need to involve more potential pixel candidates for reconstruction.

Through these visualizations, we see that our method generate promising task-oriented flows, generally consistent with the ground truth optical flows but diverse in local details.

Amt: All-Pairs Multi-Field Transforms For Efficient Frame Interpolation

Abstract: We present All-Pairs Multi-Field Transforms (AMT), a new network architecture for video frame interpolation. It is based on two essential designs. First, we build bidirectional correlation volumes for all pairs of pixels, and use the predicted bilateral flows to retrieve correlations for updating both flows and the interpolated content feature. Second, we derive multiple groups of fine-grained flow fields from one pair of updated coarse flows for performing backward warping on the input frames separately. Combining these two designs enables us to generate promising taskoriented flows and reduce the difficulties in modeling large motions and handling occluded areas during frame interpolation. These qualities promote our model to achieve stateof-the-art performance on various benchmarks with high efficiency. Moreover, our convolution-based model competes favorably compared to Transformer-based models in terms of accuracy and efficiency. Our code is available at https://github.com/MCG-NKU/AMT.

Raft: Recurrent All-Pairs Field Transforms For Optical Flow

Abstract: We introduce Recurrent All-Pairs Field Transforms (RAFT), a new deep network architecture for optical flow. RAFT extracts per-pixel features, builds multi-scale 4D correlation volumes for all pairs of pixels, and iteratively updates a flow field through a recurrent unit that performs lookups on the correlation volumes. RAFT achieves state-of-the-art performance, with strong cross-dataset generalization and high efficiency in inference time, training speed, and parameter count. Code is available \url{https://github.com/princeton-vl/RAFT}.


For the synthetic data, we can easily obtain strand maps and depth maps from 3D strand models assisted by mature rendering techniques #REFR .

To enable learning-based single-view 3D hair modeling, HairStep needs to be firstly extracted from both synthetic 3D hair data and real images for training and testing.

But it is infeasible to extract strand maps from real images via existing approaches.

Thus, we use a learning-based approach and annotate a dataset HiSa to provide supervision.

We collect 1,250 clear portrait images with various hairstyles from the Internet.

The statistics of the hairstyles, gender and race are given in the supplementary material.

Hairstep: Transfer Synthetic To Real Using Strand And Depth Maps For Single-View 3D Hair Modeling

Abstract: Our experiments show that HairStep narrows the domain gap between synthetic and real and achieves state-of-theart performance on single-view 3D hair reconstruction.

Soft Rasterizer: A Differentiable Renderer For Image-Based 3D Reasoning

Abstract: Rendering bridges the gap between 2D vision and 3D scenes by simulating the physical process of image formation. By inverting such renderer, one can think of a learning approach to infer 3D information from 2D images. However, standard graphics renderers involve a fundamental discretization step called rasterization, which prevents the rendering process to be differentiable, hence able to be learned. Unlike the state-of-the-art differentiable renderers, which only approximate the rendering gradient in the back propagation, we propose a truly differentiable rendering framework that is able to (1) directly render colorized mesh using differentiable functions and (2) back-propagate efficient supervision signals to mesh vertices and their attributes from various forms of image representations, including silhouette, shading and color images. The key to our framework is a novel formulation that views rendering as an aggregation function that fuses the probabilistic contributions of all mesh triangles with respect to the rendered pixels. Such formulation enables our framework to flow gradients to the occluded and far-range vertices, which cannot be achieved by the previous state-of-the-arts. We show that by using the proposed renderer, one can achieve significant improvement in 3D unsupervised single-view reconstruction both qualitatively and quantitatively. Experiments also demonstrate that our approach is able to handle the challenging tasks in image-based shape fitting, which remain nontrivial to existing differentiable renderers.


Accordingly, non-empty voxels with the same horizontal location could project to the corresponding pillars with sparse pooling #REFR .

= (l j , w j ) ∈ N 2 } Np j=1 and {f p j ∈ R Dp } Np j=1

for pillar indices with their corresponding features (N v ≥ N p due to many-to-one voxel-pillar relationship).

With the consistent occupancy in BEV guaranteed, for a certain X-Y coordinate containing a non-empty pillar, we could find the corresponding voxels by comparing the horizontal vector of voxel indices (l i , w i ) with pillar indice p j = (l j , w j ):

The collection of c j forms the voxel-to-pillar index {c j } Np j=1 .

Conversely, non-empty pillars could also scatter to the matching voxels with the inverse index, which we represent as sparse broadcasting. Sparse Voxel-Pillar Fusion.

Equipped with sparse pooling and broadcasting, we deploy the SFL. As illustrated in Fig.

2 , given the pair of sparse voxel and pillar features in a stage, SFL first calculates the voxel-to-pillar index {c j } Np j=1 . Then, sparse features are interacted by two pathways.

For the voxel-to-pillar connection, voxel features are aggregated in the vertical direction by sparse maxpooling to form the pillar-wise features in shape of (N p , D v ).

These pillar-wise features are spatially consistent with original sparse features, while distinct in feature dimension.

Vpfusion: Towards Robust Vertical Representation Learning For 3D Object Detection

Abstract: Efficient point cloud representation is a fundamental element of Lidar-based 3D object detection. Recent gridbased detectors usually divide point clouds into voxels or pillars and construct single-stream networks in Bird's Eye View. However, these point cloud encoding paradigms underestimate the point representation in the vertical direction, which cause the loss of semantic or fine-grained information, especially for vertical sensitive objects like pedestrian and cyclists. In this paper, we propose an explicit vertical multi-scale representation learning framework, VPFusion, to combine the complementary information from both voxel and pillar streams. Specifically, VPFusion first builds upon a sparse voxel-pillar-based backbone. The backbone divides point clouds into voxels and pillars, then encodes features with 3D and 2D sparse convolution simultaneously. Next, we introduce the Sparse Fusion Layer (SFL), which establishes a bidirectional pathway for sparse voxel and pillar features to enable the interaction between them. Additionally, we present the Dense Fusion Neck (DFN) to effectively combine the dense feature maps from voxel and pillar branches with multi-scale. Extensive experiments on the large-scale Waymo Open Dataset and nuScenes Dataset demonstrate that VPFusion surpasses the singlestream baselines by a large margin and achieves state-ofthe-art performance with real-time inference speed.

Fast Graph Representation Learning With Pytorch Geometric

Abstract: We introduce PyTorch Geometric, a library for deep learning on irregularly structured input data such as graphs, point clouds and manifolds, built upon PyTorch. In addition to general graph data structures and processing methods, it contains a variety of recently published methods from the domains of relational learning and 3D data processing. PyTorch Geometric achieves high data throughput by leveraging sparse GPU acceleration, by providing dedicated CUDA kernels and by introducing efficient mini-batch handling for input examples of different size. In this work, we present the library in detail and perform a comprehensive comparative study of the implemented methods in homogeneous evaluation scenarios.


Figure 1: DSTC11 Track 3 introduced a spoken version of MultiWOZ 2.1 #REFR with user utterances voiced by crowdworkers.

However, state-of-the-art systems trained on chats face robustness issues when dealing with spoken inputs #OTHEREFR .

In a TOD system, the role of DST is to predict at each turn and based on the dialogue history the current belief state, i.e. a condensed and updated representation of the user needs.

DST plays a central role as the system relies on the belief state to decide which action to take next.

The belief state is typically frame-based and represented as a list of <slot, value> pairs. 1 Our code is made available at https://github.com/ Orange-OpenSource/olisia-dstc11.

While both cascade and end-to-end approaches have been well studied for Spoken Language Understanding (SLU, Serdyuk et al.

(2018) ), there has been little recent work on spoken DST.

Considering the entire dialogue context, as opposed to only the current turn, requires tricky strategies for end-to-end systems #OTHEREFR .

In order to leverage state-of-the-art models, a cascade approach with separate ASR and DST components is thus preferred.

However, these two components do not benefit from joint optimization and often lack integration.

Olisia: A Cascade System For Spoken Dialogue State Tracking

Abstract: Though Dialogue State Tracking (DST) is a core component of spoken dialogue systems, recent work on this task mostly deals with chat corpora, disregarding the discrepancies between spoken and written language. In this paper, we propose OLISIA, a cascade system which integrates an Automatic Speech Recognition (ASR) model and a DST model. We introduce several adaptations in the ASR and DST modules to improve integration and robustness to spoken conversations. With these adaptations, our system ranked first in DSTC11 Track 3, a benchmark to evaluate spoken DST. We conduct an in-depth analysis of the results and find that normalizing the ASR outputs and adapting the DST inputs through data augmentation, along with increasing the pre-trained models size all play an important role in reducing the performance discrepancy between written and spoken conversations. 1

Multiwoz 2.1: A Consolidated Multi-Domain Dialogue Dataset With State Corrections And State Tracking Baselines

Abstract: MultiWOZ 2.0 (Budzianowski et al., 2018) is a recently released multi-domain dialogue dataset spanning 7 distinct domains and containing over 10,000 dialogues. Though immensely useful and one of the largest resources of its kind to-date, MultiWOZ 2.0 has a few shortcomings. Firstly, there is substantial noise in the dialogue state annotations and dialogue utterances which negatively impact the performance of state-tracking models. Secondly, follow-up work (Lee et al., 2019) has augmented the original dataset with user dialogue acts. This leads to multiple co-existent versions of the same dataset with minor modifications. In this work we tackle the aforementioned issues by introducing MultiWOZ 2.1. To fix the noisy state annotations, we use crowdsourced workers to re-annotate state and utterances based on the original utterances in the dataset. This correction process results in changes to over 32% of state annotations across 40% of the dialogue turns. In addition, we fix 146 dialogue utterances by canonicalizing slot values in the utterances to the values in the dataset ontology. To address the second problem, we combined the contributions of the follow-up works into MultiWOZ 2.1. Hence, our dataset also includes user dialogue acts as well as multiple slot descriptions per dialogue state slot. We then benchmark a number of state-of-the-art dialogue state tracking models on the MultiWOZ 2.1 dataset and show the joint state tracking performance on the corrected state annotations. We are publicly releasing MultiWOZ 2.1 to the community, hoping that this dataset resource will allow for more effective models across various dialogue subproblems to be built in the future.


Unlike previous works (e.g., #REFR ), we observe that imposing a low-deformation loss reduces performance (Sec 4.4).

From the mesh M i , we project keypointŝ k i and encourage their proximity to target 2D keypoints k i via the loss:

We run HRNet #OTHEREFR on each frame to produce k i for peoplesnapshot and use the given keypoints for ZJU Mocap.

Mesh regularization losses. We add some regularization to the mesh representation.

This is especially important for the free-form per-vertex offsets D.

This is because loose clothing need not necessarily correspond to a low deformation from the underlying SMPL model.

We only encourage normal consistency of adjacent faces in the mesh.

Let M D be the mesh generated from the SMPL parameters (β, 0, D, I, 0) and let M 0 be generated from the SMPL parameters (β, 0, 0, I, 0).

Let f j be the j th face of M D and f j be the j th face of M 0 .

With some abuse of notation, for two faces f j and f k , we denote |f j ∩ f k | as the number of vertices that are shared between both faces. The normal consistency loss is then given as:

Mesh Strikes Back: Fast And Efficient Human Reconstruction From Rgb Videos

Abstract: Human reconstruction and synthesis from monocular RGB videos is a challenging problem due to clothing, occlusion, texture discontinuities and sharpness, and framespecific pose changes. Many methods employ deferred rendering, NeRFs and implicit methods to represent clothed humans, on the premise that mesh-based representations cannot capture complex clothing and textures from RGB, silhouettes, and keypoints alone. We provide a counter viewpoint to this fundamental premise by optimizing a SMPL+D mesh and an efficient, multi-resolution texture representation using only RGB images, binary silhouettes and sparse 2D keypoints. Experimental results demonstrate that our approach is more capable of capturing geometric details compared to visual hull, mesh-based methods. We show competitive novel view synthesis and improvements in novel pose synthesis compared to NeRF-based methods, which introduce noticeable, unwanted artifacts. By restricting the solution space to the SMPL+D model combined with differentiable rendering, we obtain dramatic speedups in compute, training times (up to 24x) and inference times (up to 192x). Our method therefore can be used as is or as a fast initialization to NeRF-based methods.

Video Based Reconstruction Of 3D People Models

Abstract: This paper describes how to obtain accurate 3D body models and texture of arbitrary people from a single, monocular video in which a person is moving. Based on a parametric body model, we present a robust processing pipeline achieving 3D model fits with 5mm accuracy also for clothed people. Our main contribution is a method to nonrigidly deform the silhouette cones corresponding to the dynamic human silhouettes, resulting in a visual hull in a common reference frame that enables surface reconstruction. This enables efficient estimation of a consensus 3D shape, texture and implanted animation skeleton based on a large number of frames. We present evaluation results for a number of test subjects and analyze overall performance. Requiring only a smartphone or webcam, our method enables everyone to create their own fully animatable digital double, e.g., for social VR applications or virtual try-on for online fashion shopping.


We refer #REFR for more details about the internal architecture of Transformer model. Encoder for Table and Persona.

S2S module is utilized to encode original tabular data X i into a latent representation z S2S (X i ), and AE module is used to encode its aligned context y i and unpaired persona information u i into latent representation z AE (y i ) and z AE (u i ) respectively.

Then, we utilize a latent space fusion technique to distill persona information into z f use .

Besides, a contrastive-based discriminator is employed to guarantee the style consistency.

Both S2S and AE modules are initially parameterized by the Transformer-based model #OTHEREFR .

We focus on the interaction among latent variables, and omit the details inside encoder and decoder.

The encoder modules in autoencoders are used to encode y i and u i into latent representations z AE (y i ) and z AE (u i ) separately.

Note that although the autoencoders for y i and u i have a shared decoder, their encoders are separate.

The auto-encoder module empowers the model with the ability of generating personalized content, especially in the situation that strong supervision towards personalized context is absent.

It can help force the S 2 P-CPD model to memorize the persona information.

Therefore, we combine z AE (y i ) and z AE (u i ), defined as follows:

Towards Zero-Shot Personalized Table-To-Text Generation With Contrastive Persona Distillation

Abstract: Existing neural methods have shown great potentials towards generating informative text from structured tabular data as well as maintaining high content fidelity. However, few of them shed light on generating personalized expressions, which often requires well-aligned persona-table-text datasets that are difficult to obtain. To overcome these obstacles, we explore personalized table-to-text generation under a zeroshot setting, by assuming no well-aligned persona-table-text triples are required during training. To this end, we firstly collect a set of unpaired persona information and then propose a semi-supervised approach with contrastive persona distillation (S 2 P-CPD) to generate personalized context. Specifically, tabular data and persona information are firstly represented as latent variables separately. Then, we devise a latent space fusion technique to distill persona information into the table representation. Besides, a contrastive-based discriminator is employed to guarantee the style consistency between the generated context and its corresponding persona. Experimental results on two benchmarks demonstrate S 2 P-CPD's ability on keeping both content fidelity and personalized expressions.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


Assumption 2 indicates that all communities have approximately the same size, which can be realized by the Girvan-Newman algorithm #REFR .

where N max and N min are the maximum and minimum numbers of nodes in each community, respectively, and ε > 0 is an arbitrarily small constant.

3) We partition the nodes into K subsets according to the number of communities to which they belong. Let

where D is the degree distribution of nodes and A i is the event such that a certain node v belongs to i communities.

Assumption 3 enforces nodes not to belong overly to either only a single community or multiple (≥ 2) communities.

Note that many real-world networks (e.g., Ego-Facebook #OTHEREFR and citation networks #OTHEREFR ) tend to almost follow Assumption 3.

Now, we are ready to show the following two theorems.

Graph Neural Network-Aided Exploratory Learning For Community Detection With Unknown Topology

Abstract: In social networks, the discovery of community structures has received considerable attention as a fundamental problem in various network analysis tasks. However, due to privacy concerns or access restrictions, the network structure is often unknown, thereby rendering established community detection approaches ineffective without costly network topology acquisition. To tackle this challenge, we present META-CODE, a novel end-to-end solution for detecting overlapping communities in networks with unknown topology via exploratory learning aided by easy-to-collect node metadata. Specifically, META-CODE consists of three iterative steps in addition to the initial network inference step: 1) node-level community-affiliation embeddings based on graph neural networks (GNNs) trained by our new reconstruction loss, 2) network exploration via communityaffiliation-based node queries, and 3) network inference using an edge connectivity-based Siamese neural network model from the explored network. Through comprehensive evaluations using five real-world datasets, we demonstrate that META-CODE exhibits (a) its superiority over benchmark community detection methods, (b) empirical evaluations as well as theoretical findings to see the effectiveness of our node query, (c) the influence of each module, and (d) its computational efficiency.

Community Structure In Social And Biological Networks

Abstract: A number of recent studies have focused on the statistical properties of networked systems such as social networks and the Worldwide Web. Researchers have concentrated particularly on a few properties that seem to be common to many networks: the small-world property, power-law degree distributions, and network transitivity. In this article, we highlight another property that is found in many networks, the property of community structure, in which network nodes are joined together in tightly knit groups, between which there are only looser connections. We propose a method for detecting such communities, built around the idea of using centrality indices to find community boundaries. We test our method on computer-generated and real-world graphs whose community structure is already known and find that the method detects this known structure with high sensitivity and reliability. We also apply the method to two networks whose community structure is not well known—a collaboration network and a food web—and find that it detects significant and informative community divisions in both cases.


Similar to our work, most of these prior works have focused on next-day predictions. However, there are examples of hourly predictions #REFR .

#OTHEREFR modeled river segments using an LSTM network with graph convolutions.

One segment in the river network corresponds to a distance that the water flows during approximately one day.

Input features include daily average precipitation, daily average air temperature, date of the year, solar radiation, shade fraction, potential evapotranspiration, elevation, length, slope, and width of each segment.

Models were trained using a physics-informed setup where a traditional flow model acted as a teacher for the machine learning model.

LSTM networks have also been used for post-processing the output from the NWM #OTHEREFR .

Others have also employed convolutional neural networks (CNNs) for stream forecasting #OTHEREFR , #OTHEREFR , #OTHEREFR .

However, in contrast to us, these works do not incorporate spatial data from satellites or GIS, but instead model only the much lower-dimensional data (in single coordinates or very small neighborhoods, not entire areas as in our setup) provided as a feature vector for each time step, similar to the models using LSTMs.

More broadly, deep learning has been used for many related tasks, such as groundwater level estimation #OTHEREFR , water quality estimation #OTHEREFR , and rainfall-runoff #OTHEREFR .

While some of the above mentioned works on stream flow estimation include information about the near environment (such as elevation and slope), none of them use detailed spatial information inputs as is proposed in our work.

The use of fully convolutional neural networks to encode this information, in combination with traditional inputs such as rainfall and temperature, has the potential of representing more complex relationships and can result in a more detailed view of the near environment.

Fully Convolutional Networks For Dense Water Flow Intensity Prediction In Swedish Catchment Areas

Abstract: Intensifying climate change will lead to more extreme weather events, including heavy rainfall and drought. Accurate stream flow prediction models which are adaptable and robust to new circumstances in a changing climate will be an important source of information for decisions on climate adaptation efforts, especially regarding mitigation of the risks of and damages associated with flooding. In this work we propose a machine learning-based approach for predicting water flow intensities in inland watercourses based on the physical characteristics of the catchment areas, obtained from geospatial data (including elevation and soil maps, as well as satellite imagery), in addition to temporal information about past rainfall quantities and temperature variations. We target the one-day-ahead regime, where a fully convolutional neural network model receives spatio-temporal inputs and predicts the water flow intensity in every coordinate of the spatial input for the subsequent day. To the best of our knowledge, we are the first to tackle the task of dense water flow intensity prediction; earlier works have considered predicting flow intensities at a sparse set of locations at a time. An extensive set of model evaluations and ablations are performed, which empirically justify our various design choices. Code and preprocessed data have been made publicly available at https://github.com/ aleksispi/fcn-water-flow.

The Proper Care And Feeding Of Camels: How Limited Training Data Affects Streamflow Prediction

Abstract: Accurate streamflow prediction largely relies on historical records of both meteorological data and streamflow measurements. For many regions around the world, however, such data are only scarcely or not at all available. To select an appropriate model for a region with a given amount of historical data, it is therefore indispensable to know a model's sensitivity to limited training data, both in terms of geographic diversity and different spans of time. In this study, we provide decision support for tree- and LSTM-based models. We feed the models meteorological measurements from the CAMELS dataset, and individually restrict the training period length and the number of basins used in training. Our findings show that tree-based models provide more accurate predictions on small datasets, while LSTMs are superior given sufficient training data. This is perhaps not surprising, as neural networks are known to be data-hungry; however, we are able to characterize each model's strengths under different conditions, including the "breakeven point" when LSTMs begin to overtake tree-based models.


For downstream dense prediction tasks like detection and segmentation, we need to rearrange the output points by position after each stage to satisfy the needs of most detection and segmentation heads (e.g., Mask-RCNN #REFR ).

To reduce the points number, we evenly select some anchors in space, and the nearest k points are concatenated and fused by a linear projection.

Note that this reduction can be achieved by a convolutional operation if all points are arranged in order and k is properly set (i.e., 4 and 9), like in ViT #OTHEREFR .

For clarity on the centers and anchors stated previously, we strongly suggest the readers check appendix § B.

For classification, we average all points of the last block's output and use a FC layer for classification.

That is, Context Cluster offers remarkable flexibility in classification, but is limited to a compromise between dense prediction tasks' requirements and our model configurations.

We expect innovative detection & segmentation heads (like DETR #OTHEREFR ) can seamlessly integrate with our method.

Image As Set Of Points

Abstract: What is an image and how to extract latent features? Convolutional Networks (ConvNets) consider an image as organized pixels in a rectangular shape and extract features via convolutional operation in local region; Vision Transformers (ViTs) treat an image as a sequence of patches and extract features via attention mechanism in a global range. In this work, we introduce a straightforward and promising paradigm for visual representation, which is called Context Clusters. Context clusters (CoCs) view an image as a set of unorganized points and extract features via simplified clustering algorithm. In detail, each point includes the raw feature (e.g., color) and positional information (e.g., coordinates), and a simplified clustering algorithm is employed to group and extract deep features hierarchically. Our CoCs are convolution-and attention-free, and only rely on clustering algorithm for spatial interaction. Owing to the simple design, we show CoCs endow gratifying interpretability via the visualization of clustering process. Our CoCs aim at providing a new perspective on image and visual representation, which may enjoy broad applications in different domains and exhibit profound insights. Even though we are not targeting SOTA performance, COCs still achieve comparable or even better results than ConvNets or ViTs on several benchmarks. Codes are available at: https://github.com/ma-xu/Context-Cluster.

Mask R-Cnn

Abstract: We present a conceptually simple, flexible, and general framework for object instance segmentation. Our approach efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small overhead to Faster R-CNN, running at 5 fps. Moreover, Mask R-CNN is easy to generalize to other tasks, e.g., allowing us to estimate human poses in the same framework. We show top results in all three tracks of the COCO suite of challenges, including instance segmentation, bounding-box object detection, and person keypoint detection. Without bells and whistles, Mask R-CNN outperforms all existing, single-model entries on every task, including the COCO 2016 challenge winners. We hope our simple and effective approach will serve as a solid baseline and help ease future research in instance-level recognition. Code has been made available at: https://github.com/facebookresearch/Detectron .


Thanks to the residual upsampling, depth edges are less dispersed comparing to the results using conventional upsampling #REFR as demonstrated in Fig. 9 . A RMU consists of modulation and update parts.

All the results are evaluated on their testing splits and are capped at 80m per standard practice. RMU and Residual Upsampling.

As shown in Table 4 , the full model outperforms the baseline by a large margin. The Gordon et al.

#OTHEREFR Distilled Semantics #OTHEREFR Lite-HR-Depth #OTHEREFR Monodepth2 #OTHEREFR Lee et al.

#OTHEREFR HR-Depth #OTHEREFR PackNet #OTHEREFR RM-Depth proposed components are effective in improving the depth accuracy.

By removing either the residual upsampling or RMU, the depth error is increased.

When the modulation part is removed, the depth error is increased.

This indicates that the depth improvement is largely benefited by the modulation since it adaptively modifies the feed-forward behavior of the encoder. Object Motion.

The full model performs the best among Figure 9 . Depth map predictions using different upsamplings. all the variants as summarized in Table 5 .

The proposed components are effective in improving depth accuracy on non-rigid scenes.

When warping is disabled, the source images are not warped towards the target image.

Rm-Depth: Unsupervised Learning Of Recurrent Monocular Depth In Dynamic Scenes *

Abstract: Unsupervised methods have showed promising results on monocular depth estimation. However, the training data must be captured in scenes without moving objects. To push the envelope of accuracy, recent methods tend to increase their model parameters. In this paper, an unsupervised learning framework is proposed to jointly predict monocular depth and complete 3D motion including the motions of moving objects and camera. (1) Recurrent modulation units are used to adaptively and iteratively fuse encoder and decoder features. This not only improves the singleimage depth inference but also does not overspend model parameters. (2) Instead of using a single set of filters for upsampling, multiple sets of filters are devised for the residual upsampling. This facilitates the learning of edgepreserving filters and leads to the improved performance. (3) A warping-based network is used to estimate a motion field of moving objects without using semantic priors. This breaks down the requirement of scene rigidity and allows to use general videos for the unsupervised learning. The motion field is further regularized by an outlier-aware training loss. Despite the depth model just uses a single image in test time and 2.97M parameters, it achieves state-of-the-art results on the KITTI and Cityscapes benchmarks. * This research work is not for commercial use unless a prior arrangement has been made with the author. 1 The words, ego-motion, camera motion and pose, are interchangeably used throughout the paper.

Real-Time Single Image And Video Super-Resolution Using An Efficient Sub-Pixel Convolutional Neural Network

Abstract: Recently, several models based on deep neural networks have achieved great success in terms of both reconstruction accuracy and computational performance for single image super-resolution. In these methods, the low resolution (LR) input image is upscaled to the high resolution (HR) space using a single filter, commonly bicubic interpolation, before reconstruction. This means that the super-resolution (SR) operation is performed in HR space. We demonstrate that this is sub-optimal and adds computational complexity. In this paper, we present the first convolutional neural network (CNN) capable of real-time SR of 1080p videos on a single K2 GPU. To achieve this, we propose a novel CNN architecture where the feature maps are extracted in the LR space. In addition, we introduce an efficient sub-pixel convolution layer which learns an array of upscaling filters to upscale the final LR feature maps into the HR output. By doing so, we effectively replace the handcrafted bicubic filter in the SR pipeline with more complex upscaling filters specifically trained for each feature map, whilst also reducing the computational complexity of the overall SR operation. We evaluate the proposed approach using images and videos from publicly available datasets and show that it performs significantly better (+0.15dB on Images and +0.39dB on Videos) and is an order of magnitude faster than previous CNN-based methods.


This flexible framework incorporates well with the reward-shaping techniques, such as entropy regularization, curiosity #REFR , etc.

Our work unifies the SF-GPI and value composition to the continuous concurrent composition framework and allows reconstructing task policy from a set of primitives.

The proposed method was extended to composition at the action component level.

We demonstrate in the Pointmass environment that our multi-task agents can reconstruct the task policy from a set of primitives in real time and transfer the skills to solve unseen tasks while the single-task performance is competitive with SAC.

In addition, the task-agnostic property should benefit the autotelic framework #OTHEREFR where agents can set goals and curriculum for themselves #OTHEREFR .

However, the primary concern at this stage is whether the proposed approach can scale to higher dimensional problems. Additionally, two important topics are left as future works. First, look for the corresponding value composition for DAC.

A good starting point might be thinking of the MSF composition with weights evaluated by GPE. Second, the optimality of each composition method.

One might start with bounding the loss incurred by the policy and value composition.

Max Planck Institute for Intelligent Systems, 72076 Tübingen, Germany.2 Institute of Flight Mechanics and Controls, University of Stuttgart, 70569 Stuttgart, Germany. yutang.liu@tuebingen.mpg.de, aamir.ahmad@ifr.uni-stuttgart.de

Multi-Task Reinforcement Learning In Continuous Control With Successor Feature-Based Concurrent Composition

Abstract: Deep reinforcement learning (DRL) frameworks are increasingly used to solve high-dimensional continuouscontrol tasks in robotics. However, due to the lack of sample efficiency, applying DRL for online learning is still practically infeasible in the robotics domain. One reason is that DRL agents do not leverage the solution of previous tasks for new tasks. Recent work on multi-tasking DRL agents based on successor features has proven to be quite promising in increasing sample efficiency. In this work, we present a new approach that unifies two prior multi-task RL frameworks, SF-GPI and value composition, for the continuous control domain. We exploit compositional properties of successor features to compose a policy distribution from a set of primitives without training any new policy. Lastly, to demonstrate the multi-tasking mechanism, we present a new benchmark for multi-task continuous control environment based on Raisim. This also facilitates large-scale parallelization to accelerate the experiments. Our experimental results in the Pointmass environment show that our multi-task agent has single task performance on par with soft actor critic (SAC) and the agent can successfully transfer to new unseen tasks where SAC fails. We provide our code as open-source at https://github.com/robot-perception-group/ concurrent_composition for the benefit of the community.

Curiosity-Driven Exploration By Self-Supervised Prediction

Abstract: In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life. We formulate curiosity as the error in an agent's ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model. Our formulation scales to high-dimensional continuous state spaces like images, bypasses the difficulties of directly predicting pixels, and, critically, ignores the aspects of the environment that cannot affect the agent. The proposed approach is evaluated in two environments: VizDoom and Super Mario Bros. Three broad settings are investigated: 1) sparse extrinsic reward, where curiosity allows for far fewer interactions with the environment to reach the goal; 2) exploration with no extrinsic reward, where curiosity pushes the agent to explore more efficiently; and 3) generalization to unseen scenarios (e.g. new levels of the same game) where the knowledge gained from earlier experience helps the agent explore new places much faster than starting from scratch. Demo video and code available at this https URL


Therefore, for this case, the improvement in the information rate supported by 3-Group NOMA transmissions and 2-Group NOMA transmissions is given by #REFR and (22), respectively.

10) CASE X (l n = l m > l f ) : In this case, a total of l n transmissions will be required by 3-Group IC-NOMA scheme, with l f 3-Group NOMA transmissions and l m − l f 2-Group NOMA transmissions. The transmitted messages are

S IC−N OMA k =        √ αP sn k + √ βP sm k + √ γP s f k , k ∈ [l f ] and √ α 1 P sn k + (1 − α 1 )P sm k , k = {l f + 1, • • • ln}

with α < β < γ, α + β + γ=1 and α 1 < 0.5

The information rate for 3-Group NOMA transmissions will be the same as #OTHEREFR .

The information rate for 2-Group NOMA transmissions will be the same as (21).

11) CASE XI ( l m > l f = l n ) : In this case, a total of l m transmissions will be required by 3-Group IC-NOMA scheme, with l n 3-Group NOMA transmissions and l m −l f index-coded transmissions. The transmitted messages are

S IC−N OMA k =        √ αP sn k + √ βP sm k + √ γP s f k , k ∈ [ln]

Design And Analysis Of Index Codes For 3-Group Noma In Vehicular Adhoc Networks

Abstract: Index coding (IC) is a source coding technique employed to improve spectral utilisation, where the source node aims to satisfy users' demands by making minimum transmissions. Non-orthogonal multiple access (NOMA) is integral to the radio access technique used in 5G networks. Index-coded NOMA (IC-NOMA) transmission scheme in Vehicular Adhoc Networks (VANETs) involves applying NOMA principles on index-coded data to avoid network congestion and to improve spectral efficiency compared to conventional IC systems. In this work, a spectral efficient transmission scheme called 3-Group IC-NOMA is proposed, and an innovative index code design that fits with NOMA decoding principles to obtain improved spectral efficiency is developed. Through exhaustive analytical studies, we demonstrate that the proposed transmission scheme always supports higher rates than the conventional IC systems and requires less power to achieve an information rate at least as good as conventional IC systems.

A Survey On Non-Orthogonal Multiple Access For 5G Networks: Research Challenges And Future Trends

Abstract: Non-orthogonal multiple access (NOMA) is an essential enabling technology for the fifth generation (5G) wireless networks to meet the heterogeneous demands on low latency, high reliability, massive connectivity, improved fairness, and high throughput. The key idea behind NOMA is to serve multiple users in the same resource block, such as a time slot, subcarrier, or spreading code. The NOMA principle is a general framework, and several recently proposed 5G multiple access schemes can be viewed as special cases. This survey provides an overview of the latest NOMA research and innovations as well as their applications. Thereby, the papers published in this special issue are put into the content of the existing literature. Future research challenges regarding NOMA in 5G and beyond are also discussed.

3-Group NOMA transmissions
2-Group NOMA transmissions


In contrast to prior works that require complex structures, the proposed method requires only simple convolutional neural networks (CNNs) for restoration and a single deformable convolution #REFR to align the features of the current and reference frame.

However, the first step has the limitation that the restoration is based on general datasets and does not utilize any video-specific information.

Hence, we further enhance the video with the guidance of a reference image.

The reference image is an uncompressed frame, which drives the enhancement process to be video-specific, leading to improved restoration for both background and textures.

To better utilize the reference image, we let the network derive a confidence map indicating which pixels to use for the enhancement.

Through the confidence map, we can efficiently exclude parts of the reference image that are uncorrelated to the current frame, i.e., components that may rather increase the distortion due to the misguided information.

Experiments show that the proposed method achieves comparable performance for highresolution benchmarks while requiring practical decoding times when the HEVC is employed in Fig. 1 .

For the case of using the VVC, we still obtain significant coding gains over the VVC, leading to higher performance compared to other neural codecs. In summary, the main contributions are as follows:

• We propose a novel hybrid video compression method, consisting of a conventional video codec, a lossless image codec, and a reference-guided restoration network.

• The compressed video is enhanced in a two-step procedure, where the first step restores the details lost by conventional video encoding, and the second step uses video-specific information from the reference image for further enhancement.

• Our method requires only 38 ms for decoding a 2K frame on NVIDIA 1080 Ti while achieving comparable or better performance than top-tier methods.

Lightweight Hybrid Video Compression Framework Using Reference-Guided Restoration Network

Abstract: Recent deep-learning-based video compression methods brought coding gains over conventional codecs such as AVC and HEVC. However, learning-based codecs generally require considerable computation time and model complexity. In this paper, we propose a new lightweight hybrid video codec consisting of a conventional video codec (HEVC or VVC), a lossless image codec, and our new restoration network. Precisely, our encoder consists of a conventional video encoder and a lossless image encoder, transmitting a lossy-compressed video bitstream along with a losslessly-compressed reference frame. The decoder is constructed with corresponding video/image decoders and a new restoration network, which enhances the compressed video in two-step processes. In the first step, a network trained with a video dataset restores the details lost by the conventional encoder. Then, we further boost the video quality with the guidance of a reference image, which is a losslessly compressed video frame. The reference image provides video-specific information, which can be utilized to better restore the details of a compressed video. Experimental results show that when the HEVC is employed as a baseline in our framework, the overall coding gain is comparable to recent top-tier neural codecs while requiring much less encoding time and lower complexity. When combined with the VVC, our method brings significant gain over the VVC, thus achieving state-of-the-art coding performance.

Deformable Convolutional Networks

Abstract: Convolutional neural networks (CNNs) are inherently limited to model geometric transformations due to the fixed geometric structures in their building modules. In this work, we introduce two new modules to enhance the transformation modeling capability of CNNs, namely, deformable convolution and deformable RoI pooling. Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from the target tasks, without additional supervision. The new modules can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back-propagation, giving rise to deformable convolutional networks. Extensive experiments validate the performance of our approach. For the first time, we show that learning dense spatial transformation in deep CNNs is effective for sophisticated vision tasks such as object detection and semantic segmentation. The code is released at https://github.com/msracver/Deformable-ConvNets.

simple convolutional neural
single deformable convolution


It is known #REFR that the (i, j) th element of matrix A −1 is +1 if branch j is directed along path from i to slack bus, -1 if it is directed against, and 0 otherwise.

Lemma 2 (Norm values of r j , s j and t j ). We have, for all branches j,

r j 2 = |H j ∩ N g |, s j 2 = |H j ∩ N l |, t j 2 = |H j ∩ N l |.

Recall that branch j shares its index with its downstream bus j.

We use the properties of matrix F in this proof. From Lemma 1, F = A − .

By the construction of G, A −1 only has entries -1 and 0.

Thus, A − has (i, j) th element -1 if branch i is falls in the path from bus j to the origin and 0 otherwise.

Therefore, the i th row of A − collects all buses in H i .

The result follows by separating generator buses and placing their coefficients in r j , and load buses in s j and t j .

On Lindistflow Model Congestion Pricing: Bounding The Changes In Power Tariffs

Abstract: The optimal power flow (OPF) problem is an important mathematical program that aims at obtaining the best operating point of an electric power grid. The optimization problem typically minimizes the total generation cost subject to certain physical constraints of the system. The so-called linearized distribution flow (LinDistFlow) model leverages a set of linear equations to approximate the nonlinear AC power flows. In this paper, we consider an OPF problem based on the LinDistFlow model for a single-phase radial power network. We derive closed-form solutions to the marginal values of both real and reactive power demands. We also derive upper bounds on the congestion price (a.k.a. 'shadow price'), which denotes the change in marginal demand prices when the apparent power flow limits of certain lines are binding at optimum. Various cases of our result are discussed while simulations are carried out on a 141-bus radial power network.

Structure Learning In Power Distribution Networks

Abstract: Traditionally power distribution networks are either not observable or only partially observable. This complicates development and implementation of new smart grid technologies, such as those related to demand response, outage detection and management, and improved load-monitoring. Here, inspired by proliferation of the metering technology, we discuss statistical estimation problems in structurally loopy but operationally radial distribution grids consisting in learning operational layout of the network from measurements, e.g. voltage data, which are either already available or can be made available with a relatively minor investment. Our newly suggested algorithms apply to a wide range of realistic scenarios. The algorithms are also computationally efficient -- polynomial in time -- which is proven theoretically and illustrated computationally on a number of test cases. The technique developed can be applied to detect line failures in real time as well as to understand the scope of possible adversarial attacks on the grid.


We also have two control explanations to which we compare our experimental explanations: one where there are no privacy protections (deterministic control) and another that includes a high-level explanation of DP that does not mention , which is adapted from Xiong et al. #REFR (Xiong et al. control).

We use a 3 × 4 × 2 between-subjects study design where each respondent sees one explanation, computed using a particular explanation method (ODDS-TEXT, ODDS-VIS, SAMPLE REPORTS) and privacy budget ( ∈ {.1, .5, 2, 4}), in a given scenario type (optional, mandatory).

Deterministic Control: The worst-case situation for a hypothetical employee in our vignette is that no privacy protection is applied.

In this case, the risk of the negative consequence in the scenario is deterministic (i.e., respondents can expect deterministic outcomes in their manager's beliefs): If the respondent participates/answers NO, their manager will believe they responded NO with probability 1 and if they do the opposite, their manager will believe they responded NO with probability 0.

Not only is this "explanation" a worst case, but prior work #OTHEREFR on risk communication also suggests that it will be the simplest for respondents to understand (objective risk comprehension) and may give them the greatest self-efficacy because of its determinism, in contrast to our probabilistic explanations.

Hence, we use this deterministic setting as a control: to do so, we include the same scenario text as in the experimental conditions but omit the stylized description of DP and explanation of privacy guarantees.

What Are The Chances? Explaining The Epsilon Parameter In Differential Privacy

Abstract: Differential privacy (DP) is a mathematical privacy notion increasingly deployed across government and industry. With DP, privacy protections are probabilistic: they are bounded by the privacy budget parameter,. Prior work in health and computational science finds that people struggle to reason about probabilistic risks. Yet, communicating the implications of to people contributing their data is vital to avoiding privacy theater-presenting meaningless privacy protection as meaningful-and empowering more informed data-sharing decisions. Drawing on best practices in risk communication and usability, we develop three methods to convey probabilistic DP guarantees to end users: two that communicate odds and one offering concrete examples of DP outputs. We quantitatively evaluate these explanation methods in a vignette survey study (n = 963) via three metrics: objective risk comprehension, subjective privacy understanding of DP guarantees, and self-efficacy. We find that odds-based explanation methods are more effective than (1) output-based methods and (2) state-of-the-art approaches that gloss over information about. Further, when offered information about , respondents are more willing to share their data than when presented with a state-of-the-art DP explanation; this willingness to share is sensitive to values: as privacy protections weaken, respondents are less likely to share data. Keywords differential privacy • usable security & privacy • privacy budgets * The author conducted part of this work while visiting Columbia University. † The author conducted part of this work while visiting the Simons Institute for the Theory of Computing at UC Berkeley. ‡ The author contributed equally to advising this work.

Towards Effective Differential Privacy Communication For Users' Data Sharing Decision And Comprehension

Abstract: Differential privacy protects an individual's privacy by perturbing data on an aggregated level (DP) or individual level (LDP). We report four online human-subject experiments investigating the effects of using different approaches to communicate differential privacy techniques to laypersons in a health app data collection setting. Experiments 1 and 2 investigated participants' data disclosure decisions for low-sensitive and high-sensitive personal information when given different DP or LDP descriptions. Experiments 3 and 4 uncovered reasons behind participants' data sharing decisions, and examined participants' subjective and objective comprehensions of these DP or LDP descriptions. When shown descriptions that explain the implications instead of the definition/processes of DP or LDP technique, participants demonstrated better comprehension and showed more willingness to share information with LDP than with DP, indicating their understanding of LDP's stronger privacy guarantee compared with DP.


Inspired by InfoNCE loss #REFR , we propose a hyperedge-level contrastive loss to enhance the Fig. 5 (a) ).

Basic contrastive loss is similar to GCL #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR that contrasts the same data from different views.

Two used contrastive loss notions are used commonly: node-level contrastive loss #OTHEREFR , #OTHEREFR , #OTHEREFR and clusterlevel contrastive loss #OTHEREFR .

The former aims to distribute nodes uniformly in the hyperplane, while the latter focuses on clustering nodes with similar categories and separating nodes with different categories.

Clearly, the latter aligns with the objective of classification, while the former does not.

Thus, we include cluster-level contrastive loss in the basic contrastive loss but exclude node-level contrastive loss.

Conversely, the cluster membership probability distributions of different nodes learned from different views are treated as negative samples (e.g., cluster Fig.

5 (a) ), to differentiate between node clusters rather than nodes.

Ideally, each node cluster should correspond to nodes of the same class.

To achieve this, we project node embeddings into a space with dimensionality equal to the number of data classes (i.e., the target number of clusters).

In particular, the projection is carried out using a two-layer MLP, i.e., the projection head in Fig. 5 (a) .

Chgnn: A Semi-Supervised Contrastive Hypergraph Learning Network

Abstract: Hypergraphs can model higher-order relationships among data objects that are found in applications such as social networks and bioinformatics. However, recent studies on hypergraph learning that extend graph convolutional networks to hypergraphs cannot learn effectively from features of unlabeled data. To such learning, we propose a contrastive hypergraph neural network, CHGNN, that exploits self-supervised contrastive learning techniques to learn from labeled and unlabeled data. First, CHGNN includes an adaptive hypergraph view generator that adopts an auto-augmentation strategy and learns a perturbed probability distribution of minimal sufficient views. Second, CHGNN encompasses an improved hypergraph encoder that considers hyperedge homogeneity to fuse information effectively. Third, CHGNN is equipped with a joint loss function that combines a similarity loss for the view generator, a node classification loss, and a hyperedge homogeneity loss to inject supervision signals. It also includes basic and cross-validation contrastive losses, associated with an enhanced contrastive loss training process. Experimental results on nine real datasets offer insight into the effectiveness of CHGNN, showing that it outperforms 13 competitors in terms of classification accuracy consistently.

Representation Learning With Contrastive Predictive Coding

Abstract: While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.


When we write computations, we will work directly with the pre-and post-condition, and not with the predicate transformer, but these two approaches are equivalent #REFR .

For simplicity, we define operations for file management and for socket communication, but these account only for one possible instantiation of the monad.

Our approach can be extended to other IO operations as needed.

Moreover, the free monad can be used to implement other effects such as exceptions, state and non-determinism by extending the signature of effects #OTHEREFR , and it is future work to integrate these effects in our account.

The MIO monadic effect uses a specification monad that captures the behavior of a computation as a predicate transformer, i.e., a function that given a post-condition, it returns a pre-condition that is strong enough to guarantee the post-condition after execution of the computation.

As explained in the introduction, in our setting a pre-condition is a property of the current trace, and a post-condition is a property of the result and the new trace.

Finally, we need to define a monad morphism from the computation monad to the specification monad.

This is done using a translation from computations written in the free monad to a predicate transformer written in the specification monad, following a recipe widely employed in F ★ .

Up to this point, we focused on defining a Dijkstra monad that allows statically verifying IO programs via specifications, but we have not mentioned any mechanism to support dynamic checks.

To enable dynamic verification, we include a new silent operation called get_trace that is meant to return the trace computed until the point it is called.

This operation allows to write computations that reflect on the past events performed, and which allows to write dynamic checks based on that.

Securely Compiling Verified F ★ Programs With Io

Abstract: We propose a secure compilation chain for statically verified partial programs with input-output (IO). The source language is an F ★ subset in which a verified IO-performing program interacts with its IO-performing context via a higher-order interface that includes refinement types as well as pre-and post-conditions about past IO events. The target language is a smaller F ★ subset in which the compiled program is linked with an adversarial context via an interface without refinement types or pre-and post-conditions. To bridge this interface gap and make compilation and linking secure we propose a novel combination of higher-order contracts and reference monitoring for recording and controlling IO operations. During compilation we use contracts to convert the logical assumptions the program makes about the context into dynamic checks on each context-program boundary crossing. These boundary checks can depend on information about past IO events stored in the monitor's state, yet these checks cannot stop the adversarial target context before it performs dangerous IO operations. So, additionally, our linking forces the context to perform all IO via a secure IO library that uses reference monitoring to dynamically enforce an access control policy before each IO operation. We propose a novel way to model in F ★ that the context cannot directly access the IO operations and the monitor's internal state, based on F ★ 's recent support for flag-based effect polymorphism. We prove in F ★ that enforcing the access control policy on the context in combination with static verification of the program soundly enforces a global trace property. Moreover, we prove in F ★ that our secure compilation chain satisfies by construction Robust Relational Hyperproperty Preservation, a very strong secure compilation criterion. Finally, we illustrate our secure compilation chain at work on a simple web server example.

Dijkstra Monads For All

Abstract: This paper proposes a general semantic framework for verifying programs with arbitrary monadic side-effects using Dijkstra monads, which we define as monad-like structures indexed by a specification monad. We prove that any monad morphism between a computational monad and a specification monad gives rise to a Dijkstra monad, which provides great flexibility for obtaining Dijkstra monads tailored to the verification task at hand. We moreover show that a large variety of specification monads can be obtained by applying monad transformers to various base specification monads, including predicate transformers and Hoare-style pre- and postconditions. For defining correct monad transformers, we propose a language inspired by Moggi's monadic metalanguage that is parameterized by a dependent type theory. We also develop a notion of algebraic operations for Dijkstra monads, and start to investigate two ways of also accommodating effect handlers. We implement our framework in both Coq and F*, and illustrate that it supports a wide variety of verification styles for effects such as exceptions, nondeterminism, state, input-output, and general recursion.


The reason are twofold: 1) whereas the propagation of image classification models is unidirectional, intermediate features calculated by detector's backbone model may be revisited long after its first usage due to the existence of FPN #REFR , region proposal networks (RPN), and ROI heads.

On object detection, we do not match features by feed the transformed student (teacher) backbone features to the later part of the teacher (student) backbone model.

Therefore, directly matching intermediate features calculated by detectors' backbone models w.r.t.

function will make the subsequent propagation process complicated, and could involve great extra overhead; 2) In existing works, the common practice for conducting feature distillation on detectors is to match the features output by FPN #OTHEREFR , rather than by the backbone.

When considering how to functionally align FPN features, the situation becomes clearer and easier.

Therefore, the detection version of FCFD is designed to match FPN features.

Denote the list of teacher features output by FPN as F t = { F k t } K k=1 , and the list of student features output by FPN as

Abstract: Feature distillation makes the student mimic the intermediate features of the teacher. Nearly all existing feature-distillation methods use L2 distance or its slight variants as the distance metric between teacher and student features. However, while L2 distance is isotropic w.r.t. all dimensions, the neural network's operation on different dimensions is usually anisotropic, i.e., perturbations with the same 2-norm but in different dimensions of intermediate features lead to changes in the final output with largely different magnitude. Considering this, we argue that the similarity between teacher and student features should not be measured merely based on their appearance (i.e., L2 distance), but should, more importantly, be measured by their difference in function, namely how later layers of the network will read, decode, and process them. Therefore, we propose Function-Consistent Feature Distillation (FCFD), which explicitly optimizes the functional similarity between teacher and student features. The core idea of FCFD is to make teacher and student features not only numerically similar, but more importantly produce similar outputs when fed to the later part of the same network. With FCFD, the student mimics the teacher more faithfully and learns more from the teacher. Extensive experiments on image classification and object detection demonstrate the superiority of FCFD to existing methods. Furthermore, we can combine FCFD with many existing methods to obtain even higher accuracy. Our codes are available at https://github.com/LiuDongyang6/FCFD.

Feature Pyramid Networks For Object Detection

Abstract: Feature pyramids are a basic component in recognition systems for detecting objects at different scales. But pyramid representations have been avoided in recent object detectors that are based on deep convolutional networks, partially because they are slow to compute and memory intensive. In this paper, we exploit the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed for building high-level semantic feature maps at all scales. This architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using a basic Faster R-CNN system, our method achieves state-of-the-art single-model results on the COCO detection benchmark without bells and whistles, surpassing all existing single-model entries including those from the COCO 2016 challenge winners. In addition, our method can run at 5 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. Code will be made publicly available.

unidirectional, intermediate features
region proposal networks


This objective has since been generalized for the setting of entropy-regularized RL [Ziebart, 2010 #REFR , which augments the standard RL objective in Eq.

In this work, we analyze the case of finite, discrete state and action spaces, with the Markov Decision Process (MDP) model #OTHEREFR .

The MDP is represented as a tuple T = S, A, p, µ, r, γ where S is the set of available states; A is the set of possible actions; p : S × A → S is a transition function describing the system dynamics; µ is the initial state distribution; r : S × A → R is a (bounded) reward function which associates a reward (or cost) with each state-action pair; and γ ∈ (0, 1] is a discount factor which discounts future rewards and guarantees the convergence of total reward for infinitely long trajectories (T → ∞).

In "standard" (un-regularized) RL, the agent maximizes an objective function which is the expected future reward:

(1) by appending an entropic regularization term for the policy:

where π 0 (a|s) is the fixed prior policy.

The additional entropic control cost discourages the agent from choosing policies that deviate too much from the prior policy.

Importantly, entropy-regularized MDPs lead to stochastic optimal policies that are provably robust to perturbations of rewards and dynamics #OTHEREFR ; making them a more suitable choice for real-world problems.

Bounding The Optimal Value Function In Compositional Reinforcement Learning

Abstract: In the field of reinforcement learning (RL), agents are often tasked with solving a variety of problems differing only in their reward functions. In order to quickly obtain solutions to unseen problems with new reward functions, a popular approach involves functional composition of previously solved tasks. However, previous work using such functional composition has primarily focused on specific instances of composition functions whose limiting assumptions allow for exact zero-shot composition. Our work unifies these examples and provides a more general framework for compositionality in both standard and entropy-regularized RL. We find that, for a broad class of functions, the optimal solution for the composite task of interest can be related to the known primitive task solutions. Specifically, we present double-sided inequalities relating the optimal composite value function to the value functions for the primitive tasks. We also show that the regret of using a zero-shot policy can be bounded for this class of functions. The derived bounds can be used to develop clipping approaches for reducing uncertainty during training, allowing agents to quickly adapt to new tasks.

Reinforcement Learning And Control As Probabilistic Inference: Tutorial And Review

Abstract: The framework of reinforcement learning or optimal control provides a mathematical formalization of intelligent decision making that is powerful and broadly applicable. While the general form of the reinforcement learning problem enables effective reasoning about uncertainty, the connection between reinforcement learning and inference in probabilistic models is not immediately obvious. However, such a connection has considerable value when it comes to algorithm design: formalizing a problem as probabilistic inference in principle allows us to bring to bear a wide array of approximate inference tools, extend the model in flexible and powerful ways, and reason about compositionality and partial observability. In this article, we will discuss how a generalization of the reinforcement learning or optimal control problem, which is sometimes termed maximum entropy reinforcement learning, is equivalent to exact probabilistic inference in the case of deterministic dynamics, and variational inference in the case of stochastic dynamics. We will present a detailed derivation of this framework, overview prior work that has drawn on this and related ideas to propose new reinforcement learning and control algorithms, and describe perspectives on future research.


Interestingly (and ironically, in a way), the results of this paper point to a previously unseen relationship between Dice loss and Boundary loss #REFR .

We have shown, both analytically and empirically, that it is possible to mimic the dice loss with a simple negative of the ground truth.

This indicates that supervision comes mostly from the gradient sign, and much less than its magnitude.

This interesting finding should help to interpret and diagnose results when tuning losses for segmentation networks, and guide researchers when evaluating or designing novel segmentation losses.

Those initial findings also point to many future and exciting new works.

For instance, the simpler mime might be more straightforward to adapt for imbalanced tasks #OTHEREFR .

Another area to investigate is the behavior with respect to sub-patching.

While computable, the definition of Dice loss makes less semantic sense when applied to only a sub-patch of an image-while potentially causing divide by 0 errors on empty patches-, but this is not the case with a mime loss.

On The Dice Loss Gradient And The Ways To Mimic It

Abstract: In the past few years, in the context of fully-supervised semantic segmentation, several losses-such as cross-entropy and dicehave emerged as de facto standards to supervise neural networks. The Dice loss is an interesting case, as it comes from the relaxation of the popular Dice coefficient; one of the main evaluation metric in medical imaging applications. In this paper, we first study theoretically the gradient of the dice loss, showing that concretely it is a weighted negative of the ground truth, with a very small dynamic range. This enables us, in the second part of this paper, to mimic the supervision of the dice loss, through a simple element-wise multiplication of the network output with a negative of the ground truth. This rather surprising result sheds light on the practical supervision performed by the dice loss during gradient descent. This can help the practitioner to understand and interpret results while guiding researchers when designing new losses.

Boundary Loss For Highly Unbalanced Segmentation

Abstract: Widely used loss functions for convolutional neural network (CNN) segmentation, e.g., Dice or cross-entropy, are based on integrals (summations) over the segmentation regions. Unfortunately, it is quite common in medical image analysis to have highly unbalanced segmentations, where standard losses contain regional terms with values that differ considerably --typically of several orders of magnitude-- across segmentation classes, which may affect training performance and stability. The purpose of this study is to build a boundary loss, which takes the form of a distance metric on the space of contours, not regions. We argue that a boundary loss can mitigate the difficulties of regional losses in the context of highly unbalanced segmentation problems because it uses integrals over the boundary between regions instead of unbalanced integrals over regions. Furthermore, a boundary loss provides information that is complementary to regional losses. Unfortunately, it is not straightforward to represent the boundary points corresponding to the regional softmax outputs of a CNN. Our boundary loss is inspired by discrete (graph-based) optimization techniques for computing gradient flows of curve evolution. Following an integral approach for computing boundary variations, we express a non-symmetric L2 distance on the space of shapes as a regional integral, which avoids completely local differential computations involving contour points. Our boundary loss is the sum of linear functions of the regional softmax probability outputs of the network. Therefore, it can easily be combined with standard regional losses and implemented with any existing deep network architecture for N-D segmentation. Our boundary loss has been validated on two benchmark datasets corresponding to difficult, highly unbalanced segmentation problems: the ischemic stroke lesion (ISLES) and white matter hyperintensities (WMH).


In order to achieve higher-quality textual action descriptions and explanations, we modified the language generator in #REFR with the aim of generating better sentence grammar.

In the training, vehicle acceleration and vehicle course values are fed to the model along with corresponding ground truth natural language annotations.

Structurally, the pipeline consists of a CNNbased visual feature encoder that extracts visual information which represent high-level object descriptions.

Afterwards, an LSTM-based vehicle controller with implemented spatial attention predicts vehicle acceleration and change of course for a series of image frames.

In the end, an LSTM-based language generator uses the controller's spatial attention together with temporal attention over multiple video frames.

The textual action descriptions and explanations are generated by outputting per-word softmax probabilities.

Further, we integrated part of speech prediction in the generator and included special token penalties in the loss function.

Textual Explanations For Automated Commentary Driving

Abstract: The vehicle stops because the traffic light turns red" Neural Network (Black Box) Fig. 1. On a high level, the approach takes a series of image frames as input into an opaque neural network and outputs a natural language explanation.

Textual Explanations For Self-Driving Vehicles

Abstract: Deep neural perception and control networks have become key components of self-driving vehicles. User acceptance is likely to benefit from easy-to-interpret textual explanations which allow end-users to understand what triggered a particular behavior. Explanations may be triggered by the neural controller, namely introspective explanations, or informed by the neural controller's output, namely rationalizations. We propose a new approach to introspective explanations which consists of two parts. First, we use a visual (spatial) attention model to train a convolutional network end-to-end from images to the vehicle control commands, i.e., acceleration and change of course. The controller's attention identifies image regions that potentially influence the network's output. Second, we use an attention-based video-to-text model to produce textual explanations of model actions. The attention maps of controller and explanation model are aligned so that explanations are grounded in the parts of the scene that mattered to the controller. We explore two approaches to attention alignment, strong- and weak-alignment. Finally, we explore a version of our model that generates rationalizations, and compare with introspective explanations on the same video segments. We evaluate these models on a novel driving dataset with ground-truth human explanations, the Berkeley DeepDrive eXplanation (BDD-X) dataset. Code is available at https://github.com/JinkyuKimUCB/explainable-deep-driving.

higher-quality textual action
explanations


Implementation Details 4.1. Mesh Fitting

On the contrary, results of iterative non-linear optimization #REFR better correspond 2D RGB images but are less plausible or unstable in 3D, since the fitting process relies on 2D image keypoints, which are poorly constrained along the depth dimension.

For both our system and the standard DNR model to perform well, precise and time-consistent SMPL-X mesh fitting to the training video frames (and at test time) is important.

As for RGB photos, the convolutional regressionbased techniques #OTHEREFR produce somewhat accurate but inconsistent results that frequently mismatch the silhouettes.

Regressionbased and optimization-based methods are unable to handle complex cases such as poses with self contact.

Below we describe existing enhancements to SMPL-X optimizationbased fitting that increase the stability of DNR-based avatar systems such as ours.

We later validate the individual impact of these adjustments through ablation study.

Shared shape A person's identity does not change between frames of a training video, thus the shape-specific parameters of SMPL-X model can be shared.

Since those shape parameters are also suitable for inference, the overhead of SMPL-X inference on mobile can be reduced.

Morf: Mobile Realistic Fullbody Avatars From A Monocular Video

Abstract: We present a new approach for learning Mobile Realistic Fullbody (MoRF) avatars. MoRF avatars can be rendered in real-time on mobile phones, have high realism, and can be learned from monocular videos. As in previous works, we use a combination of neural textures and the mesh-based body geometry modeling SMPL-X. We improve on prior work, by learning per-frame warping fields in the neural texture space, allowing to better align the training signal between different frames. We also apply existing SMPL-X fitting procedure refinements for videos to improve overall avatar quality. In the comparisons to other monocular video-based avatar systems, MoRF avatars achieve higher image sharpness and temporal consistency. Participants of our user study also preferred avatars generated by MoRF.

Keep It Smpl: Automatic Estimation Of 3D Human Pose And Shape From A Single Image

Abstract: We describe the first method to automatically estimate the 3D pose of the human body as well as its 3D shape from a single unconstrained image. We estimate a full 3D mesh and show that 2D joints alone carry a surprising amount of information about body shape. The problem is challenging because of the complexity of the human body, articulation, occlusion, clothing, lighting, and the inherent ambiguity in inferring 3D from 2D. To solve this, we first use a recently published CNN-based method, DeepCut, to predict (bottom-up) the 2D body joint locations. We then fit (top-down) a recently published statistical body shape model, called SMPL, to the 2D joints. We do so by minimizing an objective function that penalizes the error between the projected 3D model joints and detected 2D joints. Because SMPL captures correlations in human shape across the population, we are able to robustly fit it to very little data. We further leverage the 3D model to prevent solutions that cause interpenetration. We evaluate our method, SMPLify, on the Leeds Sports, HumanEva, and Human3.6M datasets, showing superior pose accuracy with respect to the state of the art.


Soon after the technology was deployed, it was found that Apple's implementation uses ε = 4 when collecting emoji usage data and ε = 8 when collecting web domain data #REFR .

One of these mechanisms is Count Mean Sketch (CMS).

CMS is used on iOS and Mac OS devices to report both emojis used and websites visited to Apple.

Featured in Apple's keynote, local differential privacy allows the company to "help discover the usage patterns of a large number of users without compromising individual privacy" #OTHEREFR .

This implementation, and in particular Apple's choice of the parameter ε, has come under criticism from privacy researchers.

It is generally believed that ε -which controls the privacy loss incurred by the user -should typically not exceed ln(3) (≈ 1.10) #OTHEREFR .

Apple's choice to only consider the privacy loss per submission -once a day for both the web domain and emoji data -instead of a total privacy loss ε tot (after which objects would no longer be collected from the user #OTHEREFR ) has similarly raised concerns on theoretical ground.

While Apple states that they remove user identifiers and IP addresses after the obfuscated objects are received by their server #OTHEREFR , this is a measure that relies on trust and hence conflicts with local differential privacy's purpose of protecting against an untrusted curator #OTHEREFR .

It is indeed well-known that the mathematical guarantees offered by local differential privacy degrade as multiple objects are collected from the same user, something that can be quantified with an upper bound using the Composition Theorem #OTHEREFR (ε tot ≤ ε 1 + . . . + ε n ).

Regardless of how revealing the user's original data may be, a low ε tot would guarantee that the obfuscated data will never leak much information.

However, ε tot is a worst-case theoretical measure: it is unclear the extent to which collecting multiple objects and using a large ε for each object open the door to attacks in practice. Pool inference attack.

Pool Inference Attacks On Local Differential Privacy: Quantifying The Privacy Guarantees Of Apple'S Count Mean Sketch In Practice

Abstract: Figure 1: Example of pools defined on a universe Ω consisting of emojis, when the adversary is interested in determining the skin tone that is most often selected by the user. In this case, Usr's preferred pool is the one containing medium-light skin tone emojis.

Privacy Loss In Apple'S Implementation Of Differential Privacy On Macos 10.12

Abstract: In June 2016, Apple announced that it will deploy differential privacy for some user data collection in order to ensure privacy of user data, even from Apple. The details of Apple's approach remained sparse. Although several patents have since appeared hinting at the algorithms that may be used to achieve differential privacy, they did not include a precise explanation of the approach taken to privacy parameter choice. Such choice and the overall approach to privacy budget use and management are key questions for understanding the privacy protections provided by any deployment of differential privacy. ::: In this work, through a combination of experiments, static and dynamic code analysis of macOS Sierra (Version 10.12) implementation, we shed light on the choices Apple made for privacy budget management. We discover and describe Apple's set-up for differentially private data processing, including the overall data pipeline, the parameters used for differentially private perturbation of each piece of data, and the frequency with which such data is sent to Apple's servers. ::: We find that although Apple's deployment ensures that the (differential) privacy loss per each datum submitted to its servers is $1$ or $2$, the overall privacy loss permitted by the system is significantly higher, as high as $16$ per day for the four initially announced applications of Emojis, New words, Deeplinks and Lookup Hints. Furthermore, Apple renews the privacy budget available every day, which leads to a possible privacy loss of 16 times the number of days since user opt-in to differentially private data collection for those four applications. ::: We advocate that in order to claim the full benefits of differentially private data collection, Apple must give full transparency of its implementation, enable user choice in areas related to privacy loss, and set meaningful defaults on the privacy loss permitted.


Hence, even if a very small β is selected -so that (12) holds with practical certainty -the number of scenarios to be sampled remains manageable, see also #REFR .

Conversely, if the safety constraints (10b) are overly restrictive, the scenario problem (11) may become unfeasible; if this were the case, however, the original problem (9) would also certainly be unfeasible, and one would need to consider broader classes of policies, or to relax the safety requirements, e.g., by introducing slack variables in #OTHEREFR . θ , it holds that:

P θ max w 2 ≤1 R(Φ u , Ψ u (θ), w, θ) ≤R N ,

and (x, u) ∈ S(θ) , ∀w ∈ W(θ) ≥ 1 − , #OTHEREFR that is, given a parameter realization θ ∼ P θ , the probability that the optimal policy Φ u (Ψ u (θ), D) computed solving (11) both incurs regret of at mostR N (Ψ u (θ), D) and complies with the safety constraints is lower bounded by 1 − .

Theorem 1 presents an explicit sample complexity bound that, given a priori specified confidence and violation levels, ensures that our safety and regret guarantees extend to all but at most a fraction of unseen dynamics θ ∈ Θ with arbitrarily high probability 1 − β.

As well-known in the literature on scenario optimization, the minimum number of scenarios N ( , β) required to fulfill the conditions of Theorem 1 grows at most logarithmically with β −1 .

Further, we note that the condition on the number N of uncertainty samples given in Theorem 1 is tight for fully-supported problems #OTHEREFR ; a simpler, albeit not tight, sufficient condition on N is given by #OTHEREFR :

We remark that following a scenario approach allows us to explicitly compute the clairvoyant optimal policy ψ(w, θ) by replacing the uncertain system dynamics with their sampled counterparts.

Regret bounds relative to the instance-wise optimal benchmark ψ(w, θ) are attractive, as they yield upper bounds on the closed-loop cost that adapt to the realized dynamics θ and perturbation w.

To illustrate this point more thoroughly, let us consider an alternative design based on a classical worst-case H ∞ objective:

Regret Optimal Control For Uncertain Stochastic Systems

Abstract: We consider control of uncertain linear timevarying stochastic systems from the perspective of regret minimization. Specifically, we focus on the problem of designing a feedback controller that minimizes the loss relative to a clairvoyant optimal policy that has foreknowledge of the system dynamics and the exogenous disturbances. In this competitive framework, establishing robustness guarantees proves challenging as, differently from the case where the model is known, the benchmark policy is not only inapplicable, but also impossible to compute without knowledge of the system parameters. To overcome this issue, we embrace a scenario optimization approach, and we propose minimizing regret robustly over a finite set of randomly sampled system parameters. We prove that this policy optimization problem can be efficiently solved through semidefinite programming, and that the corresponding solution retains strong probabilistic out-of-sample regret guarantees in face of the uncertain dynamics. Our method naturally extends to include satisfaction of safety constraints with high probability. We validate our theoretical results and showcase the potential of our approach by means of numerical simulations.

Robust Model Predictive Control Via Scenario Optimization

Abstract: This paper discusses a novel probabilistic approach for the design of robust model predictive control (MPC) laws for discrete-time linear systems affected by parametric uncertainty and additive disturbances. The proposed technique is based on the iterated solution, at each step, of a finite-horizon optimal control problem (FHOCP) that takes into account a suitable number of randomly extracted scenarios of uncertainty and disturbances, followed by a specific command selection rule implemented in a receding horizon fashion. The scenario FHOCP is always convex, also when the uncertain parameters and disturbance belong to nonconvex sets, and irrespective of how the model uncertainty influences the system's matrices. Moreover, the computational complexity of the proposed approach does not depend on the uncertainty/disturbance dimensions, and scales quadratically with the control horizon. The main result in this work is related to the analysis of the closed loop system under receding-horizon implementation of the scenario FHOCP, and essentially states that the devised control law guarantees constraint satisfaction at each step with some a priori assigned probability p, while the system's state reaches the target set either asymptotically, or in finite time with probability at least p. The proposed method may be a valid alternative when other existing techniques, either deterministic or stochastic, are not directly usable due to excessive conservatism or to numerical intractability caused by lack of convexity of the robust or chance-constrained optimization problem.


Vision-Language Pretraining For Aesthetics

For aesthetic pretraining, we use the AVA-Captions dataset #REFR which is currently the largest available dataset for aesthetic comments.

Many of these methods are trained on large proprietary datasets #OTHEREFR with image-text pairs crawled from the web.

While this general pretraining strategy has proven useful for tasks such as image classification and retrieval, it is limited in its ability to represent aestheticrelated information due to the under-representation of such information on the web.

Consequently, the aesthetic information gets diluted in the vast amount of pretraining data.

To address this limitation, we propose a two-stage pretraining approach that involves initializing the model with a generally pretrained image-text model and then further pretraining it on aesthetic image-comment pairs.

For general pretraining, we use a 650M filtered subset of the openly available LAION-5B-English #OTHEREFR dataset.

Each image in AVA-Captions is associated with one or more user comments that provide informative insights into different aesthetic aspects of the image.

We randomly sample one comment for each image to construct imagecomment pairs during training.

In contrast to traditional supervised learning with predefined labels or categories, vision-language pretraining enables learning of open-set aesthetic concepts through noisy image-comment pairs.

This results in visual and textual representations that encompass a wider range of aesthetic concepts, enhancing transferability to downstream tasks.

Vila: Learning Image Aesthetics From User Comments With Vision-Language Pretraining

Abstract: Assessing the aesthetics of an image is challenging, as it is influenced by multiple factors including composition, color, style, and high-level semantics. Existing image aesthetic assessment (IAA) methods primarily rely on humanlabeled rating scores, which oversimplify the visual aesthetic information that humans perceive. Conversely, user comments offer more comprehensive information and are a more natural way to express human opinions and preferences regarding image aesthetics. In light of this, we propose learning image aesthetics from user comments, and exploring vision-language pretraining methods to learn multimodal aesthetic representations. Specifically, we pretrain an image-text encoder-decoder model with image-comment pairs, using contrastive and generative objectives to learn rich and generic aesthetic semantics without human labels. To efficiently adapt the pretrained model for downstream IAA tasks, we further propose a lightweight rank-based adapter that employs text as an anchor to learn the aesthetic ranking concept. Our results show that our pretrained aesthetic vision-language model outperforms prior works on image aesthetic captioning over the AVA-Captions dataset, and it has powerful zero-shot capability for aesthetic tasks such as zero-shot style classification and zero-shot IAA, surpassing many supervised baselines. With only minimal finetuning parameters using the proposed adapter module, our model achieves state-of-the-art IAA performance over the AVA dataset.

Aesthetic Image Captioning From Weakly-Labelled Photographs

Abstract: Aesthetic image captioning (AIC) refers to the multi-modal task of generating critical textual feedbacks for photographs. While in natural image captioning (NIC), deep models are trained in an end-to-end manner using large curated datasets such as MS-COCO, no such large-scale, clean dataset exists for AIC. Towards this goal, we propose an automatic cleaning strategy to create a benchmarking AIC dataset, by exploiting the images and noisy comments easily available from photography websites. We propose a probabilistic caption-filtering method for cleaning the noisy web-data, and compile a large-scale, clean dataset "AVA-Captions", (230, 000 images with 5 captions per image). Additionally, by exploiting the latent associations between aesthetic attributes, we propose a strategy for training the convolutional neural network (CNN) based visual feature extractor, the first component of the AIC framework. The strategy is weakly supervised and can be effectively used to learn rich aesthetic representations, without requiring expensive ground-truth annotations. We finally show-case a thorough analysis of the proposed contributions using automatic metrics and subjective evaluations.