The generation module is inspired by NeRF #REFR that predicts the density of the position but with the input image feature condition.

We made these simulated PX using a ray-based method and sample rays based on the principles of PX.

Those rays are rendered as a synthesized image pixel using the Beer-Lambert law with CBCT data.

Still, there exists a domain gap between real-world PX and simulated ones.

We utilized CycleGAN-based translation module that supports unpaired image-to-image translation, as obtaining paired PX and CBCT datasets is challenging.

We propose a new loss function that enhances the existing CycleGAN model to generate a more plausible synthesized image from the real-world image.

Lastly, we add an encoder-decoder-based refinement module to enhance the 3D reconstruction quality.

The main contributions of our paper are as follows: #OTHEREFR We introduce a novel architecture that can process real-world PX to the 3D oral structure without any prior data such as a dental arch, #OTHEREFR We propose a new synthesizing method that eliminates the need for matching CBCT and PX datasets during training, and a loss function that reduces the gap between the simulated image and real-world image, #OTHEREFR Results show that our model can generate high-quality oral structure and much more robust to the real-world image than other state-of-the-art models.

Nebla: Neural Beer-Lambert For 3D Reconstruction Of Oral Structures From Panoramic Radiographs

Abstract: Panoramic radiography (panoramic X-ray, PX) is a widely used imaging modality for dental examination. However, its applicability is limited as compared to 3D Conebeam computed tomography (CBCT), because PX only provides 2D flattened images of the oral structure. In this paper, we propose a new framework which estimates 3D oral structure from real-world PX images. Since there are not many matching PX and CBCT data, we used simulated PX from CBCT for training, however, we used real-world panoramic radiographs at the inference time. We propose a new ray-sampling method to make simulated panoramic radiographs inspired by the principle of panoramic radiography along with the rendering function derived from the Beer-Lambert law. Our model consists of three parts: translation module, generation module, and refinement module. The translation module changes the real-world panoramic radiograph to the simulated training image style. The generation module makes the 3D structure from the input image without any prior information such as a dental arch. Our ray-based generation approach makes it possible to reverse the process of generating PX from oral structure in order to reconstruct CBCT data. Lastly, the refinement module enhances the quality of the 3D output. Results show that our approach works better for simulated and real-world images compared to other state-of-the-art methods.

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.


For the experiments in Section 3, we utilize a widelyused model architecture in STR, denoted as TRBA #REFR .

Note that we use the same model architecture for Section 3 experiments for a fair comparison.

According to the previous work #OTHEREFR , the four stages derived from the STR models are as follows:

The thin-plate spline (TPS) transformation, a variant of a spatial transformation network #OTHEREFR , normalizes the perspective or curved text image into a horizontal text image.

Improving Scene Text Recognition For Character-Level Long-Tailed Distribution

Abstract: Figure 1: (a) We visualize the character-level distributions of WikiSynth of Korean (Kr) and Chinese (Cn). We categorize the characters according to the number of training samples: many, medium, and few. We also show misclassified images of tail characters predicted wrongly as head characters. (b) Our approach outperforms the baseline model when evaluated with character-level (char) F1 score, a newly proposed evaluation metric, which measures the performance at the character level. The higher score, the better. This result shows that our method enhances the performance on few characters significantly.

What Is Wrong With Scene Text Recognition Model Comparisons? Dataset And Model Analysis

Abstract: Many new proposals for scene text recognition (STR) models have been introduced in recent years. While each claim to have pushed the boundary of the technology, a holistic and fair comparison has been largely missing in the field due to the inconsistent choices of training and evaluation datasets. This paper addresses this difficulty with three major contributions. First, we examine the inconsistencies of training and evaluation datasets, and the performance gap results from inconsistencies. Second, we introduce a unified four-stage STR framework that most existing STR models fit into. Using this framework allows for the extensive evaluation of previously proposed STR modules and the discovery of previously unexplored module combinations. Third, we analyze the module-wise contributions to performance in terms of accuracy, speed, and memory demand, under one consistent set of training and evaluation datasets. Such analyses clean up the hindrance on the current comparisons to understand the performance gain of the existing modules.


As in #REFR , this can be constructed using the locally cartesian closed structure of C q .

Thus, the assumption of pre-Π-structure tells us that L r ( p A ) is type-exponentiable, and hence the pushforward of B along it is represented by a type.

What remains is to construct an appropriate local universe to make this strictly stable.

Let V Π(A,B) be the universal object equipped with maps

π A ∶ V Π(A,B) → C ω (V A ) π B ∶ π * A (C ω (V A E A )) → V B .

Now because we have pre-Π-structure, the map Π(A,B) ), i.e. we have a distributivity pullback

π * A (C ω (p E A )) is type- exponentiable.

Thus, the pushforward of E B [π B ] ∈ Ty q (π * A (C ω (V A E A ))) along it is represented by a type E Π(A,B) ∈ Ty q (V

• π * A (C ω (V A E A )) E B [π B ] π * A (C ω (V A E A )) V Π(A,B) E Π(A,B) V Π(A,B)

Now the bottom map in (5.16) together with ⌜B⌝ Π(A,B) .

Semantics Of Multimodal Adjoint Type Theory

Abstract: We show that contrary to appearances, Multimodal Type Theory (MTT) over a 2-category M can be interpreted in any M-shaped diagram of categories having, and functors preserving, M-sized limits, without the need for extra left adjoints. This is achieved by a construction called "co-dextrification" that co-freely adds left adjoints to any such diagram, which can then be used to interpret the "context lock" functors of MTT. Furthermore, if any of the functors in the diagram have right adjoints, these can also be internalized in type theory as negative modalities in the style of FitchTT. We introduce the name Multimodal Adjoint Type Theory (MATT) for the resulting combined general modal type theory. In particular, we can interpret MATT in any finite diagram of toposes and geometric morphisms, with positive modalities for inverse image functors and negative modalities for direct image functors.

The Local Universes Model: An Overlooked Coherence Construction For Dependent Type Theories

Abstract: We present a new coherence theorem for comprehension categories, providing strict models of dependent type theory with all standard constructors, including dependent products, dependent sums, identity types, and other inductive types. Precisely, we take as input a"weak model": a comprehension category, equipped with structure corresponding to the desired logical constructions. We assume throughout that the base category is close to locally Cartesian closed: specifically, that products and certain exponentials exist. Beyond this, we require only that the logical structure should be *weakly stable* --- a pure existence statement, not involving any specific choice of structure, weaker than standard categorical Beck--Chevalley conditions, and holding in the now standard homotopy-theoretic models of type theory. Given such a comprehension category, we construct an equivalent split one, whose logical structure is strictly stable under reindexing. This yields an interpretation of type theory with the chosen constructors. The model is adapted from Voevodsky's use of universes for coherence, and at the level of fibrations is a classical construction of Giraud. It may be viewed in terms of local universes or delayed substitutions.


DenseNets #REFR : Introduced in 2018, the DenseNet architecture contributes to techniques aimed at information preservation in deep neural networks.

Coming at the back of successes (and challenges) with preceding architectures, notably ResNets and InceptionNets, DenseNets take advantage of feature re-use as a way to preserve information flow.

What this means is, as opposed to combining features of each layer with those of preceding layers via addition (as is the case in ResNets), DenseNets use concatenation technique, such that the output feature Figure 4 : A residual connection between a layer and successive layers map of each layer is concatenated with those of preceding layers, the output of which is sent to subsequent layers.

Compared to ResNets, the number of parameters is considerably smaller and it has the added advantage of improved flow of information and gradients through the network, thus making it easier to train.

Compared to InceptionNets which also concatenate feature maps, making the network wider, the DenseNets are simpler and more efficient.

For this task, we train DenseNet-121 architecture on our data.

Simulating Malaria Detection In Laboratories Using Deep Learning

Abstract: Malaria is usually diagnosed by a microbiologist by examining a small sample of blood smear. Reducing mortality from malaria infection is possible if it is diagnosed early and followed with appropriate treatment. While the WHO has set audacious goals of reducing malaria incidence and mortality rates by 90% in 2030 and eliminating malaria in 35 countries by that time [1], it still remains a difficult challenge. Computer-assisted diagnostics are on the rise these days as they can be used effectively as a primary test in the absence of or providing assistance to a physician or pathologist. The purpose of this paper is to describe an approach to detecting, localizing and counting parasitic cells in blood sample images towards easing the burden on healthcare workers.

Densely Connected Convolutional Networks

Abstract: Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections - one between each layer and its subsequent layer - our network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). DenseNets obtain significant improvements over the state-of-the-art on most of them, whilst requiring less computation to achieve high performance. Code and pre-trained models are available at this https URL .

deep neural networks
DenseNet architecture


A bit later, I proved that in the heptagrid, the tessellation {7, 3} of the hyperbolic plane, there is a weakly universal cellular automaton with three states which is rotation invariant and which is truly planar, #REFR .

The first paper about a universal cellular automaton in the pentagrid, the tessellation {5, 4} of the hyperbolic plane, was #OTHEREFR .

This cellular automaton was also rotation invariant, at each step of the computation, the set of non quiescent states had infinitely many cycles: we shall say that it is a truly planar cellular automaton. That automaton had 22 states.

That result was improved by a cellular automaton with 9 states in #OTHEREFR .

Recently, it was improved with 5 states, see #OTHEREFR .

Later, I improved the result down to two states but the rules are no more rotation invariant, see #OTHEREFR .

Paper #OTHEREFR constructs three cellular automata which are strongly universal and rotation invariant: one in the pentagrid, one in the heptagrid, one in the tessellation {5, 3, 4} of the hyperbolic 3D-space.

By strongly universal we mean that the initial configuration is finite, i.e. it lies within a large enough circle.

In the present paper, we borrow ideas from #OTHEREFR and from the previous paper, #OTHEREFR .

To #OTHEREFR , we borrow the idea of implementing a two register structure which is finite at each time of the computation.

A Strongly Universal Cellular Automaton On The Heptagrid With Seven States, New Proof

Abstract: In this paper, we prove that there is a strongly universal cellular automaton on the heptagrid with seven states which is rotation invariant. This improves a previous paper of the author with the same number of states. Here, the structures are simpler and the number of rules is much less.

A Weakly Universal Cellular Automaton On The Heptagrid With Three States

Abstract: In this paper, we prove that there is a weakly universal cellular automaton on the pentagrid with three states which is rotation invariant and which uses \`a la Moore neighbourhood. Moreover, at each step of the computation, the set of non quiescent states has always infinitely many cycles.


The qualitative results for TEGLO trained on FFHQ #REFR data for single-view 3D reconstruc- tion on samples from CelebA-HQ are in Fig.(9) .

In Fig.(10) , we show qualitative results including the texture image (t O ) for complex appearance and geometry such as multi-view consistent eyeglasses, 3D make-up and hair.

Compared with Fig.(24) in #OTHEREFR , we show improved multi-view consistent results for eyeglasses in row-1, and 3D make-up in row-2.

Compared with Fig.(25) in #OTHEREFR , we show multi-view consistent representations for beard that the baseline method #OTHEREFR was unable to model.

It is the task of representing an in-the-wild or out-of-distribution image using a trained network.

Previous work such as AUVNet #OTHEREFR require additional training of a ResNet-18 #OTHEREFR for the image encoder and IM-Net #OTHEREFR for the shape decoder followed by ray marching to obtain the mesh to represent the image while methods such as EG3D #OTHEREFR require PTI (Pivotal Tuning Inversion #OTHEREFR ) fine-tuning to represent the image.

For single-view textured 3D representation in TEGLO, we simply invert the image into the latent and do not require any fine-tuning.

Reconstructing single-view images at arbitrary resolutions while preserving 3D consistency is very desirable for many applications.

However, EG3D #OTHEREFR has a limitation in performing this task because its generator is conditioned on the camera intrinsic and extrinsic parameters, leading to a "baked-in" training image resolution.

As TEGLO does not condition on the camera, it enables single-view 3D reconstruction and novel view synthesis at arbitrary resolutions without requiring re-training for different resolutions.

Teglo: High Fidelity Canonical Texture Mapping From Single-View Images

Abstract: Recent work in Neural Fields (NFs) learn 3D representations from class-specific single view image collections. However, they are unable to reconstruct the input data preserving high-frequency details. Further, these methods do not disentangle appearance from geometry and hence are not suitable for tasks such as texture transfer and editing. In this work, we propose TEGLO (Textured EG3D-GLO) for learning 3D representations from single view in-the-wild image collections for a given class of objects. We accomplish this by training a conditional Neural Radiance Field (NeRF) without any explicit 3D supervision. We equip our method with editing capabilities by creating a dense correspondence mapping to a 2D canonical space. We demonstrate that such mapping enables texture transfer and texture editing without requiring meshes with shared topology. Our key insight is that by mapping the input image pixels onto the texture space we can achieve near perfect reconstruction (≥ 74 dB PSNR at 1024 2 resolution). Our formulation allows for high quality 3D consistent novel view synthesis with high-frequency details at megapixel image resolution.

A Style-Based Generator Architecture For Generative Adversarial Networks

Abstract: We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.


In contrast to prior work #REFR that considers only uniformity with respect to the data-set, we extend the generalization error upper bounds for a general class of gradient-based algorithms (A(η t , T )).

A key part of our analysis is the derivation of upper bounds by considering on-average algorithmic stability, namely 1

Through on-average stability we show upper bounds for the generalization error by using a unified analysis among all algorithms A R within A(η t , T ).

Specifically, we develop a uniform stability analysis for all data-sets S and selection rules R.

In line with standard prior work, for convex and nonconvex losses we assume that the loss is uniformly Lipschitz.

For strongly-convex losses, the Lipschitz assumption does not hold and we instead consider a relaxed path-boundedness assumption on the loss gradients, introduced later to avoid clutter (see Section 5).

Select Without Fear: Almost All Mini-Batch Schedules Generalize Optimally

Abstract: We establish matching upper and lower generalization error bounds for mini-batch Gradient Descent (GD) training with either deterministic or stochastic, data-independent, but otherwise arbitrary batch selection rules. We consider smooth Lipschitz-convex/nonconvex/stronglyconvex loss functions, and show that classical upper bounds for Stochastic GD (SGD) also hold verbatim for such arbitrary nonadaptive batch schedules, including all deterministic ones. Further, for convex and strongly-convex losses we prove matching lower bounds directly on the generalization error uniform over the aforementioned class of batch schedules, showing that all such batch schedules generalize optimally. Lastly, for smooth (non-Lipschitz) nonconvex losses, we show that full-batch (deterministic) GD is essentially optimal, among all possible batch schedules within the considered class, including all stochastic ones.

Train Faster, Generalize Better: Stability Of Stochastic Gradient Descent

Abstract: We show that parametric models trained by a stochastic gradient method (SGM) with few iterations have vanishing generalization error. We prove our results by arguing that SGM is algorithmically stable in the sense of Bousquet and Elisseeff. Our analysis only employs elementary tools from convex and continuous optimization. We derive stability bounds for both convex and non-convex optimization under standard Lipschitz and smoothness assumptions. ::: Applying our results to the convex case, we provide new insights for why multiple epochs of stochastic gradient methods generalize well in practice. In the non-convex case, we give a new interpretation of common practices in neural networks, and formally show that popular techniques for training large deep models are indeed stability-promoting. Our findings conceptually underscore the importance of reducing training time beyond its obvious benefit.


Table 3 demonstrates that (1) without stage 1 (epoch=0), our method performs even worse than TF-VAEGAN #REFR , and (2) including stage 1 in training has a positive effect, and the more epochs we pretrain at stage 1, the better results we finally achieve.

To further verify our method, we conduct t-SNE #OTHEREFR visualization for synthesized unseen features of our method and TF-VAEGAN #OTHEREFR on CUB.

As shown in Figure 2 , compared to TF-VAEGAN #OTHEREFR , the synthesized features of our method are more compact and more distinct, especially in the top-left corner of the figure. Importance of stage 1.

We take experiments to verify the importance of pretraining at stage 1.

Table 3 shows the conventional ZSL results of our method of CUB dataset with different training epochs in stage 1.

Exploiting Semantic Attributes For Transductive Zero-Shot Learning

Abstract: Zero-shot learning (ZSL) aims to recognize unseen classes by generalizing the relation between visual features and semantic attributes learned from the seen classes. A recent paradigm called transductive zero-shot learning further leverages unlabeled unseen data during training and has obtained impressive results. These methods always synthesize unseen features from attributes through a generative adversarial network to mitigate the bias towards seen classes. However, they neglect the semantic information in the unlabeled unseen data and thus fail to generate high-fidelity attribute-consistent unseen features. To address this issue, we present a novel transductive ZSL method that produces semantic attributes of the unseen data and imposes them on the generative process. In particular, we first train an attribute decoder that learns the mapping from visual features to semantic attributes. Then, from the attribute decoder, we obtain pseudo-attributes of unlabeled data and integrate them into the generative model, which helps capture the detailed differences within unseen classes so as to synthesize more discriminative features. Experiments on five standard benchmarks show that our method yields state-of-the-art results for zero-shot learning.

Latent Embedding Feedback And Discriminative Features For Zero-Shot Classification

Abstract: Zero-shot learning strives to classify unseen categories for which no data is available during training. In the generalized variant, the test samples can further belong to seen or unseen categories. The state-of-the-art relies on Generative Adversarial Networks that synthesize unseen class features by leveraging class-specific semantic embeddings. During training, they generate semantically consistent features, but discard this constraint during feature synthesis and classification. We propose to enforce semantic consistency at all stages of (generalized) zero-shot learning: training, feature synthesis and classification. We further introduce a feedback loop, from a semantic embedding decoder, that iteratively refines the generated features during both the training and feature synthesis stages. The synthesized features together with their corresponding latent embeddings from the decoder are transformed into discriminative features and utilized during classification to reduce ambiguities among categories. Experiments on (generalized) zero-shot learning for object and action classification reveal the benefit of semantic consistency and iterative feedback for GAN-based networks, outperforming existing methods on six zero-shot learning benchmarks.


Matching the pathology of new patients with already diagnosed and curated cases offers pathologists a new approach to improve diagnostic accuracy through visual inspection of similar cases and a computational majority vote for consensus building #REFR .

The emergence of digital pathology (DP) has opened new horizons for histopathology #OTHEREFR .

Machine Learning (ML) algorithms are able to operate on digitized slides to assist pathologists with different tasks.

Whereas ML-involving classification and segmentation methods have obvious benefits for image analysis, image search represents a fundamental shift in computational pathology #OTHEREFR .

Pathologists examine tissue slides under a microscope on a regular basis and write diagnostic and prognostic reports based on their visual inspections.

The use of immunologic research methodologies in histopathology has resulted in a significant improvement in neoplasm microscopic diagnosis #OTHEREFR .

Immunohistochemistry (IHC) has become a formidable tool at the pathologist's disposal, despite the fact that histological analysis of hematoxylin and eosin (H&E) stained tissue sections remain at the core of the discipline #OTHEREFR - #OTHEREFR .

IHC is a technique for detecting specific antigens (proteins) in tissue slices using labeled antibodies that bind with the antigen #OTHEREFR .

The purpose of staining is to draw attention to the area of interest while also providing contrast against the 'background'.

Immunohistochemistry Biomarkers-Guided Image Search For Histopathology

Abstract: Medical practitioners use a number of diagnostic tests to make a reliable diagnosis. Traditionally, Haematoxylin and Eosin (H&E) stained glass slides have been used for cancer diagnosis and tumor detection. However, recently a variety of immunohistochemistry (IHC) stained slides can be requested by pathologists to examine and confirm diagnoses for determining the subtype of a tumor when this is difficult using H&E slides only. Deep learning (DL) has received a lot of interest recently for image search engines to extract features from tissue regions, which may or may not be the target region for diagnosis. This approach generally fails to capture high-level patterns corresponding to the malignant or abnormal content of histopathology images. In this work, we are proposing a targeted image search approach, inspired by the pathologists' workflow, which may use information from multiple IHC biomarker images when available. These IHC images could be aligned, filtered, and merged together to generate a composite biomarker image (CBI) that could eventually be used to generate an attention map to guide the search engine for localized search. In our experiments, we observed that an IHC-guided image search engine can retrieve relevant data more accurately than a conventional (i.e., H&E-only) search engine without IHC guidance. Moreover, such engines are also able to accurately conclude the subtypes through majority votes.

Yottixel -- An Image Search Engine For Large Archives Of Histopathology Whole Slide Images

Abstract: With the emergence of digital pathology, searching for similar images in large archives has gained considerable attention. Image retrieval can provide pathologists with unprecedented access to the evidence embodied in already diagnosed and treated cases from the past. This paper proposes a search engine specialized for digital pathology, called Yottixel, a portmanteau for "one yotta pixel," alluding to the big-data nature of histopathology images. The most impressive characteristic of Yottixel is its ability to represent whole slide images (WSIs) in a compact manner. Yottixel can perform millions of searches in real-time with a high search accuracy and low storage profile. Yottixel uses an intelligent indexing algorithm capable of representing WSIs with a mosaic of patches by converting them into a small number of methodically extracted barcodes, called "Bunch of Barcodes" (BoB), the most prominent performance enabler of Yottixel. The performance of the prototype platform is qualitatively tested using 300 WSIs from the University of Pittsburgh Medical Center (UPMC) and 2,020 WSIs from The Cancer Genome Atlas Program (TCGA) provided by the National Cancer Institute. Both datasets amount to more than 4,000,000 patches of 1000x1000 pixels. We report three sets of experiments that show that Yottixel can accurately retrieve organs and malignancies, and its semantic ordering shows good agreement with the subjective evaluation of human observers.


We again use pre-trained roBERTa #REFR for text embedding which gives an embedding of size 768 for 'determinant + subject', 'verb', 'auxiliary verb', 'determinant + object', and the whole caption.

We use the same train, validation, and test split as the existing methods. Metrics.

For evaluation with the existing methods, we use the following set of video captioning metrics: BLEU@4 #OTHEREFR , METEOR #OTHEREFR , ROUGE-L #OTHEREFR and CIDEr #OTHEREFR .

CIDEr #OTHEREFR is studied to be robust in the condition where the semantic meaning of the caption remains intact #OTHEREFR .

Also, we are the first to use GPT-2 #OTHEREFR pre-trained model for measuring the Grammatical correctness Score (GS) of the captions generated by our model in comparison to state-of-the-art which we implement for the GS metric, demonstrating improved performance (see Section 4.2). Implementation.

For text, we have used spaCy 2 with roBERTa #OTHEREFR , a version of BERT #OTHEREFR , to extract POS components along with nouns from the groundtruth captions.

Following the existing methods, we use InceptionRes-NetV2 #OTHEREFR to extract the spatial features and C3D #OTHEREFR to extract the temporal features.

These features are projected to 512 sizes before being input into the network.

We train for epochs 25 and use a learning rate of 0.00015, batch size 16, ADAM optimizer #OTHEREFR , 16 samples per video as well as a hidden state size of 512 for the Caption block. Our model has 76M parameters and 0.045s inference time.

Apart from that, we use Yolov7 #OTHEREFR for extracting object features for the noun anchor.

The whole implementation is performed using one NVIDIA GeForce RTX 3090 and PyTorch.

Sem-Pos: Grammatically And Semantically Correct Video Captioning

Abstract: Generating grammatically and semantically correct captions in video captioning is a challenging task. The captions generated from the existing methods are either word-byword that do not align with grammatical structure or miss key information from the input videos. To address these issues, we introduce a novel globallocal fusion network, with a Global-Local Fusion Block (GLFB) that encodes and fuses features from different parts of speech (POS) components with visual-spatial features. We use novel combinations of different POS components-'determinant + subject', 'auxiliary verb', 'verb', and 'determinant + object' for supervision of the POS blocks-Det + Subject, Aux Verb, Verb, and Det + Object respectively. The novel global-local fusion network together with POS blocks helps align the visual features with language description to generate grammatically and semantically correct captions. Extensive qualitative and quantitative experiments on benchmark MSVD and MSRVTT datasets demonstrate that the proposed approach generates more grammatically and semantically correct captions compared to the existing methods, achieving the new state-of-the-art. Ablations on the POS blocks and the GLFB demonstrate the impact of the contributions on the proposed method.

Roberta: A Robustly Optimized Bert Pretraining Approach

Abstract: Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.


The chosen activation function is the exponential linear unit (ELU) #REFR for all layers except the last, where the activation is linear.

During training, the distributions are sampled to generate inputs to the decoder network, while only the µ values are used to encode the time series used later by the predictor.

The decoder model is designed as an almost symmetric network to the encoder.

Latent-space samples are fed into a fully-connected layer, and its output is reshaped to the same shape as the last convolutional layer in the encoder.

Six transposed convolution layers are then used to increase the spatial dimension with decreasing filters.

A final transposed convolution layer with two filters produces the two output channels.

Β-Variational Autoencoders And Transformers For Reduced-Order Modelling Of Fluid Flows

Abstract: Variational autoencoder (VAE) architectures have the potential to develop reduced-order models (ROMs) for chaotic fluid flows. We propose a method for learning compact and near-orthogonal ROMs using a combination of a β-VAE and a transformer, tested on numerical data from a twodimensional viscous flow in both periodic and chaotic regimes. The β-VAE is trained to learn a compact latent representation of the flow velocity, and the transformer is trained to predict the temporal dynamics in latent space. Using the β-VAE to learn disentangled representations in latentspace, we obtain a more interpretable flow model with features that resemble those observed in the proper orthogonal decomposition, but with a more efficient representation. Using Poincaré maps, the results show that our method can capture the underlying dynamics of the flow outperforming other prediction models. The proposed method has potential applications in other fields such as weather forecasting, structural dynamics or biomedical engineering.

Fast And Accurate Deep Network Learning By Exponential Linear Units (Elus)

Abstract: We introduce the"exponential linear unit"(ELU) which speeds up learning in deep neural networks and leads to higher classification accuracies. Like rectified linear units (ReLUs), leaky ReLUs (LReLUs) and parametrized ReLUs (PReLUs), ELUs alleviate the vanishing gradient problem via the identity for positive values. However, ELUs have improved learning characteristics compared to the units with other activation functions. In contrast to ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero like batch normalization but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit natural gradient because of a reduced bias shift effect. While LReLUs and PReLUs have negative values, too, they do not ensure a noise-robust deactivation state. ELUs saturate to a negative value with smaller inputs and thereby decrease the forward propagated variation and information. Therefore, ELUs code the degree of presence of particular phenomena in the input, while they do not quantitatively model the degree of their absence. In experiments, ELUs lead not only to faster learning, but also to significantly better generalization performance than ReLUs and LReLUs on networks with more than 5 layers. On CIFAR-100 ELUs networks significantly outperform ReLU networks with batch normalization while batch normalization does not improve ELU networks. ELU networks are among the top 10 reported CIFAR-10 results and yield the best published result on CIFAR-100, without resorting to multi-view evaluation or model averaging. On ImageNet, ELU networks considerably speed up learning compared to a ReLU network with the same architecture, obtaining less than 10% classification error for a single crop, single model network.

chosen activation function
exponential linear unit


To this end, we don't need to include the case where m = 0 in the boundary condition in #REFR .

It is worth pointing out that when an honest miner mines a block and then the length of the authentic branch grows to L, the length of the counterfeit branch cannot be the same as that of the authentic branch.

This is because if so, then before the honest miner mines a block, the counterfeit branch has already grown to L blocks, and hence, the counterfeit branch has already been longer than the authentic branch which implies that we have already reached the boundary condition in (2) before the honest miner mines a block.

By employing (1), #OTHEREFR and 3, the pursuit of the closed-form expression for È L (m, n) can be cast as a two-sided boundary hitting problem for a two-dimensional random walk with two possible moving directions, which is illustrated in Fig. 4 .

To be specific, let s T [m, n] T + t=T t=1 δ t denote the position of the random walk in two-dimensional space after the T -th step (i.e., after T blocks have been mined in the blockchain), where for any t, δ t is a random vector δ t = δ #OTHEREFR [−1, 0]

Vulnerability Of Finitely-Long Blockchains In Securing Data

Abstract: Recently, blockchain has been applied in various fields to secure data exchanges and storage in decentralized systems. In a blockchain application where the task of the application which makes use of the data stored in a blockchain has to be accomplished by a time instant, the employed blockchain is essentially finitely-long. In this paper, we consider a general finitely-long blockchain model which is generalized from most existing works on finitely-long blockchain applications, and take the first step towards characterizing the vulnerability of finitely-long blockchains in securing data against double-spending attacks. For the first time, we develop a general closed-form expression for the probability of success in launching a double-spending attack on a finitely-long blockchain. This probability essentially characterizes the vulnerability of finitely-long blockchains. Then, we prove that the probability of success in launching a doublespending attack on a finitely-long blockchain is no greater than that on an infinitely-long blockchain, which implies that finitelylong blockchains are less vulnerable to double-spending attacks than infinitely-long blockchains. Moreover, we show that unlike infinitely-long blockchains which can be surely paralyzed by a 51% attack, finitely-long blockchains are more resistant to 51% attacks. Index Terms-Finitely-long blockchain, double-spending attack, proof-of-work, 51% attack.

Secure Distributed Dynamic State Estimation In Wide-Area Smart Grids

Abstract: Smart grid is a large complex network with a myriad of vulnerabilities, usually operated in adversarial settings and regulated based on estimated system states. In this study, we propose a novel highly secure distributed dynamic state estimation mechanism for wide-area (multi-area) smart grids, composed of geographically separated subregions, each supervised by a local control center. We first propose a distributed state estimator assuming regular system operation that achieves near-optimal performance based on the local Kalman filters and with the exchange of necessary information between local centers. To enhance the security, we further propose to 1) protect the network database and the network communication channels against attacks and data manipulations via a blockchain (BC)-based system design, where the BC operates on the peer-to-peer network of local centers, 2) locally detect the measurement anomalies in real-time to eliminate their effects on the state estimation process, and 3) detect misbehaving (hacked/faulty) local centers in real-time via a distributed trust management scheme over the network. We provide theoretical guarantees regarding the false alarm rates of the proposed detection schemes, where the false alarms can be easily controlled. Numerical studies illustrate that the proposed mechanism offers reliable state estimation under regular system operation, timely and accurate detection of anomalies, and good state recovery performance in case of anomalies.


Indeed, Harase #REFR obtained Tausworthe generators over F 2 with t-value two or three for s = 3, but they were not optimal with respect to the t-value.

#OTHEREFR ), have the t-value zero for s = 3 only if the period length is exactly three.

Their proof was specialized for the case F 2 ; for example, they used the property

L m ∩ U m = {I} in [16, Proof of Theorem 1],

where L m and U m denote a set of non-singular m × m lower-triangular and upper-triangular matrices, respectively.

This is false in the fields F b except for F 2 .

Thus, we conducted a search over F b , whose restrictions are looser than those over F 2 .

A Generalization Of Short-Period Tausworthe Generators And Its Application To Markov Chain Quasi-Monte Carlo

Abstract: A one-dimensional sequence u 0 , u 1 , u 2 ,. .. ∈ [0, 1) is said to be completely uniformly distributed (CUD) if overlapping s-blocks (u i , u i+1 ,. .. , u i+s−1), i = 0, 1, 2,. . ., are uniformly distributed for every dimension s ≥ 1. This concept naturally arises in Markov chain quasi-Monte Carlo (QMC). However, the definition of CUD sequences is not constructive, and thus there remains the problem of how to implement the Markov chain QMC algorithm in practice. Harase (2021) focused on the t-value, which is a measure of uniformity widely used in the study of QMC, and implemented short-period Tausworthe generators (i.e., linear feedback shift register generators) over the twoelement field F 2 that approximate CUD sequences by running for the entire period. In this paper, we generalize a search algorithm over F 2 to that over arbitrary finite fields F b with b elements and conduct a search for Tausworthe generators over F b with t-values zero (i.e., optimal) for dimension s = 3 and small for s ≥ 4, especially in the case where b = 3, 4, and 5. We provide a parameter table of Tausworthe generators over F 4 , and report a comparison between our new generators over F 4 and existing generators over F 2 in numerical examples using Markov chain QMC.

A Table Of Short-Period Tausworthe Generators For Markov Chain Quasi-Monte Carlo

Abstract: We consider the problem of estimating expectations by using Markov chain Monte Carlo methods and improving the accuracy by replacing IID uniform random points with quasi-Monte Carlo (QMC) points. Recently, it has been shown that Markov chain QMC remains consistent when the driving sequences are completely uniformly distributed (CUD). However, the definition of CUD sequences is not constructive, so an implementation method using short-period Tausworthe generators (i.e., linear feedback shift register generators over the two-element field) that approximate CUD sequences has been proposed. In this paper, we conduct an exhaustive search of short-period Tausworthe generators for Markov chain QMC in terms of the $t$-value, which is a criterion of uniformity widely used in the study of QMC methods. We provide a parameter table of Tausworthe generators and show the effectiveness in a numerical example using Gibbs sampling.


In the literature, there exist many further evaluation scores, particularly for assessing probability calibration #REFR .

Moreover, NLL is a proper scoring rule #OTHEREFR ) such that the best score corresponds to a perfect prediction.

Brier score (BS, ↓), proposed by Brier (1950), is another proper scoring rule, which measures the squared error between predicted probability vectors and one-hot encoded target vectors:

As a comprehensive evaluation of probabilities is beyond this article's scope, we focus on proper scoring rules inducing calibration measures.

Accordingly, we have omitted other evaluation scores, such as the expected calibration error (Naeini et al., 2015) being a non-proper scoring rule.

Multi-annotator supervised learning techniques: By default, we train MaDL via the weighted loss function in Eq.

25 using the hyperparameter values from Section 4 and the most general architecture depicted by Fig. 3 .

In addition to ablations as part of analyzing the three RQs, we present a detailed ablation study on the hyperparameters of MaDL in Appendix A.

Multi-Annotator Deep Learning: A Probabilistic Framework For Classification

Abstract: Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowd workers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL). A ground truth and an annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers probabilistic estimates of annotators' performances. A modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' densities within a latent space as proxies of their potentially correlated annotations. Together with a weighted loss function, we improve the learning from correlated annotation patterns. In a comprehensive evaluation, we examine three research questions about multi-annotator supervised learning. Our findings indicate MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators.

Can You Trust Your Model'S Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

Abstract: Modern machine learning methods including deep learning have achieved great success in predictive accuracy for supervised learning tasks, but may still fall short in giving useful estimates of their predictive uncertainty. Quantifying uncertainty is especially critical in real-world settings, which often involve input distributions that are shifted from the training distribution due to a variety of factors including sample bias and non-stationarity. In such settings, well calibrated uncertainty estimates convey information about when a model's output should (or should not) be trusted. Many probabilistic deep learning methods, including Bayesian-and non-Bayesian methods, have been proposed in the literature for quantifying predictive uncertainty, but to our knowledge there has not previously been a rigorous large-scale empirical comparison of these methods under dataset shift. We present a large-scale benchmark of existing state-of-the-art methods on classification problems and investigate the effect of dataset shift on accuracy and calibration. We find that traditional post-hoc calibration does indeed fall short, as do several other previous methods. However, some methods that marginalize over models give surprisingly strong results across a broad spectrum of tasks.


The framework also employs state-of-the-art transformer neural networks #REFR augmented by manoeuvre-specific heads to predict multiple trajectories conditioned on the predicted manoeuvre vectors.

To address the aforementioned limitations, we propose a novel Multimodal Manoeuvre and Trajectory Prediction (MMnTP) framework.

Firstly, we propose a bespoke formulation of manoeuvre prediction based on a vector representation of manoeuvres.

This representation includes a sequence of manoeuvre types and transition times between those during the prediction window.

To increase the plausibility of the predictions, constraints are introduced on the manoeuvre types and the number of allowed manoeuvre changes within the prediction horizon.

We then propose a multimodal discriminative manoeuvre prediction model using this new formulation.

To train the model, a novel multimodal manoeuvre prediction loss function and a mode selection method are proposed based on the predicted types and timings of manoeuvres.

The proposed framework is evaluated in highway driving scenarios using two public trajectory datasets, namely NGSIM #OTHEREFR and highD #OTHEREFR . Our contributions can be summarised as follows:

• A bespoke formulation of manoeuvre prediction, which allows estimating a sequence of manoeuvre types and transition times between them.

• A novel transformer-based model to predict multimodal manoeuvres and their corresponding trajectories.

• A tailored multimodal training method using a new multimodal manoeuvre loss function and mode selection method.

Multimodal Manoeuvre And Trajectory Prediction For Autonomous Vehicles Using Transformer Networks

Abstract: Predicting the behaviour (i.e. manoeuvre/trajectory) of other road users, including vehicles, is critical for the safe and efficient operation of autonomous vehicles (AVs), a.k.a. automated driving systems (ADSs). Due to the uncertain future behaviour of vehicles, multiple future behaviour modes are often plausible for a vehicle in a given driving scene. Therefore, multimodal prediction can provide richer information than single-mode prediction enabling AVs to perform a better risk assessment. To this end, we propose a novel multimodal prediction framework that can predict multiple plausible behaviour modes and their likelihoods. The proposed framework includes a bespoke problem formulation for manoeuvre prediction, a novel transformer-based prediction model, and a tailored training method for multimodal manoeuvre and trajectory prediction. The performance of the framework is evaluated using two public benchmark highway driving datasets, namely NGSIM and highD. The results show that the proposed framework outperforms the state-of-theart multimodal methods in the literature in terms of prediction error and is capable of predicting plausible manoeuvre and trajectory modes.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


In contrast, we show that combining one specialized expert with the generic knowledge base model with simple ensemble methods such as averaging or linear stacking #REFR is generally more efficient than ensembling multiple specialized experts.

A follow-up line of thought extracts such information from a pretrained classifier #OTHEREFR , or even learns the optimal taxonomy jointly with the image representations #OTHEREFR .

Such models have been shown to improve the efficiency/accuracy trade-off in classification tasks.

However, this class-based routing is a limiting assumption, and per-sample routing has been shown to outperform hierarchical classification models when correctly parametrized #OTHEREFR .

Finally, MoE can be seen as an ensembling technique whose weights are learned by the gate.

While it is common to assume each sample is routed to a unique expert to maximize efficiency, some works #OTHEREFR have considered combining several experts to boost accuracy.

Revisiting Single-Gated Mixtures Of Experts

Abstract: Mixture of Experts (MoE) are rising in popularity as a means to train extremely large-scale models, yet allowing for a reasonable computational cost at inference time. Recent state-of-the-art approaches usually assume a large number of experts, and require training all experts jointly, which often lead to training instabilities such as the router collapsing. In contrast, in this work, we propose to revisit simple single-gate MoE, which allows for more practical training. Key to our work are (i) a base model branch acting both as an early-exit and an ensembling regularization scheme, (ii) a simple and efficient asynchronous training pipeline without router collapse issues, and finally (iii) an automatic per-sample clustering-based initialization. We show experimentally that the proposed model obtains efficiency-to-accuracy trade-offs comparable with other more complex MoE, and outperforms non-mixture baselines. This showcases the merits of even a simple single-gate MoE, and motivates further exploration in this area.

Feature-Weighted Linear Stacking

Abstract: Ensemble methods, such as stacking, are designed to boost predictive accuracy by blending the predictions of multiple machine learning models. Recent work has shown that the use of meta-features, additional inputs describing each example in a dataset, can boost the performance of ensemble methods, but the greatest reported gains have come from nonlinear procedures requiring significant tuning and training time. Here, we present a linear technique, Feature-Weighted Linear Stacking (FWLS), that incorporates meta-features for improved accuracy while retaining the well-known virtues of linear regression regarding speed, stability, and interpretability. FWLS combines model predictions linearly using coefficients that are themselves linear functions of meta-features. This technique was a key facet of the solution of the second place team in the recently concluded Netflix Prize competition. Significant increases in accuracy over standard linear stacking are demonstrated on the Netflix Prize collaborative filtering dataset.

multiple specialized experts
simple ensemble methods


Algorithm 1 Computing Task-Specific Parameter Sensitivities

Note that although our criterion draws inspiration from pruning work #REFR , it is distinct from it.

Since is same for all parameters, we can eliminate it when comparing the sensitivity with the other parameters and finally get

Therefore, the sensitivity of a parameter can be efficiently measured by its potential to reduce the loss on the target domain.

#OTHEREFR measures the parameter importance by the squared change in loss when removing them, i.e., (E(D t , w) − E(D t , w | w n = 0)) 2 and finally derives the parameter importance by (g n w n ) 2 , which is different from our formulations in Eqs. #OTHEREFR and #OTHEREFR .

In practice, we accumulate S from a total number of C training samples ahead of fine-tuning to generate accurate sensitivity as shown in Algorithm 1, where C is a predefined hyper-parameter.

In Section 4.3, we show that employing only 400 training samples is sufficient for getting reasonable parameter sensitivity, which requires only 5.5 seconds with a single GPU for any VTAB-1k dataset with ViT-B/16 backbone #OTHEREFR .

Sensitivity-Aware Visual Parameter-Efficient Tuning

Abstract: Visual Parameter-Efficient Tuning (VPET) has become a powerful alternative for full fine-tuning so as to adapt pre-trained vision models to downstream tasks, which only tunes a small number of parameters while freezing the vast majority ones to ease storage burden and optimization difficulty. However, existing VPET methods introduce trainable parameters to the same positions across different tasks depending solely on human heuristics and neglect the domain gaps. To this end, we study where to introduce and how to allocate trainable parameters by proposing a novel Sensitivity-aware visual Parameter-efficient Tuning (SPT) scheme, which adaptively allocates trainable parameters to task-specific important positions given a desired tunable parameter budget. Specifically, our SPT first quickly identifies the sensitive parameters that require tuning for a given task in a data-dependent way. Next, our SPT further boosts the representational capability for the weight matrices whose number of sensitive parameters exceeds a pre-defined threshold by utilizing any of the existing structured tuning methods, e.g., LoRA [27] or Adapter [26], to replace directly tuning the selected sensitive parameters (unstructured tuning) under the budget. Extensive experiments on a wide range of downstream recognition tasks show that our SPT is complementary to the existing VPET methods and largely boosts their performance, e.g., SPT improves Adapter with supervised pre-trained ViT-B/16 backbone by 4.2% and 1.4% mean Top-1 accuracy, reaching SOTA performance on FGVC and VTAB-1k benchmarks, respectively. Source code is at https://github.com/ ziplab/SPT.

Importance Estimation For Neural Network Pruning

Abstract: Structural pruning of neural network parameters reduces computational, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the first and second-order Taylor expansions to approximate a filter's contribution. Both methods scale consistently across any network layer without requiring per-layer sensitivity analysis and can be applied to any kind of layer, including skip connections. For modern networks trained on ImageNet, we measured experimentally a high (>93%) correlation between the contribution computed by our methods and a reliable estimate of the true importance. Pruning with the proposed methods led to an improvement over state-of-the-art in terms of accuracy, FLOPs, and parameter reduction. On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0.02% in the top-1 accuracy on ImageNet.


Instances Of The Redistribution Mechanism Framework

The largest known set of diffusion auction mechanisms with the above properties is Critical Diffusion Mechanism (CDM) #REFR .

In our network-based redistribution mechanism framework, if we require the output mechanism to be IC and IR, then the input diffusion auction mechanism should also be IC, IR and non-deficit.

Especially, the first diffusion auction mechanism, Incentive Diffusion Mechanism (IDM) #OTHEREFR is also a member in CDM, which has the highest efficiency.

In this section, we input IDM and another mechanism in CDM called Threshold Neighbourhood Mechanism (TNM) [12] into our framework to see the outcomes.

For convenience, we briefly introduce the idea of the IDM and TNM with our notations.

Both IDM and TNM first find the agent with the highest valuation and their critical ancestors.

Then the mechanisms check these agents from the sponsor to the agent with the highest valuation.

A Redistribution Framework For Diffusion Auctions

Abstract: Redistribution mechanism design aims to redistribute the revenue collected by a truthful auction back to its participants without affecting the truthfulness. We study redistribution mechanisms for diffusion auctions, which is a new trend in mechanism design [19]. The key property of a diffusion auction is that the existing participants are incentivized to invite new participants to join the auctions. Hence, when we design redistributions, we also need to maintain this incentive. Existing redistribution mechanisms in the traditional setting are targeted at modifying the payment design of a truthful mechanism, such as the Vickrey auction. In this paper, we do not focus on one specific mechanism. Instead, we propose a general framework to redistribute the revenue back for all truthful diffusion auctions for selling a single item. The framework treats the original truthful diffusion auction as a black box, and it does not affect its truthfulness. The framework can also distribute back almost all the revenue.

Diffusion And Auction On Graphs

Abstract: Auction is the common paradigm for resource allocation which is a fundamental problem in human society. Existing research indicates that the two primary objectives, the seller's revenue and the allocation efficiency, are generally conflicting in auction design. For the first time, we expand the domain of the classic auction to a social graph and formally identify a new class of auction mechanisms on graphs. All mechanisms in this class are incentive-compatible and also promote all buyers to diffuse the auction information to others, whereby both the seller's revenue and the allocation efficiency are significantly improved comparing with the Vickrey auction. It is found that the recently proposed information diffusion mechanism is an extreme case with the lowest revenue in this new class. Our work could potentially inspire a new perspective for the efficient and optimal auction design and could be applied into the prevalent online social and economic networks. © 2019 International Joint Conferences on Artificial Intelligence. All rights reserved.


We only compare with w2v-U since it has achieved significantly better results than former GAN or HMM based unsupervised methods #REFR .

Our model performs better than w2v-U-L under three configurations: with or without ST and 4-gram or transformer language models.

The absolute improvements ranges from 0.2% to 1.7%, and the relative improvements ranges from 5.1% to 12.8%.

These results reflect that injecting instance noises of various intensities and appending diffusion timestep-dependent discriminators during adversarial training are effective.

Table 2 reports phoneme error rates (PER) under TIMIT's matched and unmatched training data setups.

The same 4gram LM is used for these four models.

Under both matched and unmatched setups, our diffusion-GAN enhanced w2v-U models outperforms w2v-U, with absolution improvements from 0.3% to 1.4% and relative improvements from 2.7% to 7.9%.

Enhancing Unsupervised Speech Recognition With Diffusion Gans

Abstract: We enhance the vanilla adversarial training method for unsupervised Automatic Speech Recognition (ASR) by a diffusion-GAN. Our model (1) injects instance noises of various intensities to the generator's output and unlabeled reference text which are sampled from pretrained phoneme language models with a length constraint, (2) asks diffusion timestep-dependent discriminators to separate them, and (3) back-propagates the gradients to update the generator. Word/phoneme error rate comparisons with wav2vec-U under Librispeech (3.1% for test-clean and 5.6% for test-other), TIMIT and MLS datasets, show that our enhancement strategies work effectively.

Completely Unsupervised Phoneme Recognition By A Generative Adversarial Network Harmonized With Iteratively Refined Hidden Markov Models

Abstract: Producing a large annotated speech corpus for training ASR systems remains difficult for more than 95% of languages all over the world which are low-resourced, but collecting a relatively big unlabeled data set for such languages is more achievable. This is why some initial effort have been reported on completely unsupervised speech recognition learned from unlabeled data only, although with relatively high error rates. In this paper, we develop a Generative Adversarial Network (GAN) to achieve this purpose, in which a Generator and a Discriminator learn from each other iteratively to improve the performance. We further use a set of Hidden Markov Models (HMMs) iteratively refined from the machine generated labels to work in harmony with the GAN. The initial experiments on TIMIT data set achieve an phone error rate of 33.1%, which is 8.5% lower than the previous state-of-the-art.


We trained all agents using Tensorflow 2.3 #REFR and the OpenAI Baselines PPO2 implementation [17] with a distributed sampler.

We provide important details about our training setup here.

We train all three agents with PPO #OTHEREFR , a popular deep reinforcement learning algorithm.

We choose PPO for training our agents for a few reasons.

This algorithm is commonly used because it is found to be empirically robust and effective in a wide range of tasks #OTHEREFR .

We train each of the three agent architectures for 15 hours, the equivalent of 10 million training timesteps, on at least 3 different random seeds.

For a full list of training hyperparameters used in all agent versions, please refer to Table 6 .

We found this set to perform best on preliminary experiments.

To effectively train agents in a complex video game setting, we use a distributed approach leveraging an in-house sample collection framework and Azure cloud resources.

Training samples are collected from a scaleset of 20 low priority GPU virtual machines (Azure NV6), each running 3 video game instances.

The samples are then sent to one training head node, a CPU-only Azure E32s memory-optimized virtual machine.

Navigates Like Me: Understanding How People Evaluate Human-Like Ai In Video Games

Abstract: We aim to understand how people assess human likeness in navigation produced by people and artificially intelligent (AI) agents in a video game. To this end, we propose a novel AI agent with the goal of generating more human-like behavior. We collect hundreds of crowd-sourced assessments comparing the human-likeness of navigation behavior generated by our agent and baseline AI agents with human-generated behavior. Our proposed agent passes a Turing Test, while the baseline agents do not. By passing a Turing Test, we mean that human judges could not quantitatively distinguish between videos of a person and an AI agent navigating. To understand what people believe constitutes human-like navigation, we extensively analyze the justifications of these assessments. This work provides insights into the characteristics that people consider human-like in the context of goal-directed video game navigation, which is a key step for further improving human interactions with AI agents. CCS CONCEPTS • Applied computing → Computer games; • Computing methodologies → Reinforcement learning; • Human-centered computing → Empirical studies in HCI.

Tensorflow: Large-Scale Machine Learning On Heterogeneous Distributed Systems

Abstract: TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms. A computation expressed using TensorFlow can be executed with little or no change on a wide variety of heterogeneous systems, ranging from mobile devices such as phones and tablets up to large-scale distributed systems of hundreds of machines and thousands of computational devices such as GPU cards. The system is flexible and can be used to express a wide variety of algorithms, including training and inference algorithms for deep neural network models, and it has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer science and other fields, including speech recognition, computer vision, robotics, information retrieval, natural language processing, geographic information extraction, and computational drug discovery. This paper describes the TensorFlow interface and an implementation of that interface that we have built at Google. The TensorFlow API and a reference implementation were released as an open-source package under the Apache 2.0 license in November, 2015 and are available at www.tensorflow.org.


More sophisticated methods tend to be based on more complex algorithms such as neural networks, such as the work from #REFR which explores the use of self-organising maps (SOMs) to reduce dimensionality within the data and create an interpretable 2D map of topics.

Topic modelling provides an unsupervised approach to topic allocation in texts, with a broad range of complexities.

It will be explored in sections four and five of this paper Simple approaches such as Latent Semantic Indexing (LSI) #OTHEREFR involve the vectorisation of texts within a corpus and grouping together based on co-sine similarity (an effective measure used to compare similarity of vectors #OTHEREFR ).

Such methods are quick to implement and require little resources but come with the large disadvantage that the nature of the topic groupings remains unknown, making the results very difficult to interpret.

But whilst having the advantage of interpretability these methods are often computationally expensive and can tie up valuable resources within an organisation.

Latent Dirichlet Allocation (LDA) #OTHEREFR provides a middle ground between overly simply non-interpretable and overly-complex resource heavy topic modelling techniques, and is one of the most commonly used methods in the field #OTHEREFR .

LDA involves creating a latent layer of topics within a dataset where words that are likely to be found near each other within texts are grouped.

Each text within a corpus is then evaluated for a percentage match with each of the topics in the latent layer to allow allocation.

One of the drawbacks with LDA is that as a statistical approach, interpretation of the topics is still required to achieve sensible results, just because words are statistically found near each other does not necessarily mean they will be considered related by a human observer.

Natural Language Processing On Customer Note Data

Abstract: Automatic analysis of customer data for businesses is an area that is of interest to companies. Business to business data is studied rarely in academia due to the sensitive nature of such information. Applying natural language processing can speed up the analysis of prohibitively large sets of data. This paper addresses this subject and applies sentiment analysis, topic modelling and keyword extraction to a B2B data set. We show that accurate sentiment can be extracted from the notes automatically and the notes can be sorted by relevance into different topics. We see that without clear separation topics can lack relevance to a business context.

Cuda-Self-Organizing Feature Map Based Visual Sentiment Analysis Of Bank Customer Complaints For Analytical Crm

Abstract: With the widespread use of social media, companies now have access to a wealth of customer feedback data which has valuable applications to Customer Relationship Management (CRM). Analyzing customer grievances data, is paramount as their speedy non-redressal would lead to customer churn resulting in lower profitability. In this paper, we propose a descriptive analytics framework using Self-organizing feature map (SOM), for Visual Sentiment Analysis of customer complaints. The network learns the inherent grouping of the complaints automatically which can then be visualized too using various techniques. Analytical Customer Relationship Management (ACRM) executives can draw useful business insights from the maps and take timely remedial action. We also propose a high-performance version of the algorithm CUDASOM (CUDA based Self Organizing feature Map) implemented using NVIDIA parallel computing platform, CUDA, which speeds up the processing of high-dimensional text data and generates fast results. The efficacy of the proposed model has been demonstrated on the customer complaints data regarding the products and services of four leading Indian banks. CUDASOM achieved an average speed up of 44 times. Our approach can expand research into intelligent grievance redressal system to provide rapid solutions to the complaining customers.


TVM #REFR is a compiler that exposes graph-level and operatorlevel optimizations to provide performance portability for DL workloads across diverse hardware backends.

HLO #OTHEREFR is an single assignment-based intermediate representation(IR) for tensor computations in XLA.

MLIR #OTHEREFR is a reusable and extensible compiler infrastructure that standardizes the static single Assignment-based IR data structures and provides a declarative system to define IR dialects.

Gshard #OTHEREFR and GSPMD #OTHEREFR introduce collective communication primitives on HLO IR and provides convenient APIs for sharding large models.

Relay #OTHEREFR presents a compiler framework to unify and generalize IR in existing frameworks.

Ada-Grouper: Accelerating Pipeline Parallelism In Preempted Network By Adaptive Group-Scheduling For Micro-Batches

Abstract: Pipeline parallelism has been demonstrated to be a remarkable approach to improve throughput for training deep neural networks with billions of parameters over heterogeneous clusters. The 1F1B scheduling plan is a widely adopted strategy for memory and performance optimization, which interchanges the forward and backward stage computations of different micro-batches. On the other hand, a common issue in using the 1F1B scheduling is that stage computation is delayed due to the data transfer when network resources are preempted by other tasks, even with the minimum communication between stages. The exclusive access of these network resources cannot be guaranteed in cloud offerings. We present a general scheduling technique to accommodate pipeline parallelism to preempted network environments at the expense of a certain amount of memory pressure. The core concept is to extend 1F1B schedule scheme to kFkB, which groups k micro-batches, and alternately executes k forward and backward computations. We propose Ada-Grouper, an adaptive kFkB scheduler which regularly adjusts the number of group members k to maintain an optimal balance between communication and computation efficiency correspond to changes in a changing network environment under the memory limit. Experimental results demonstrate that our design maintains stable performance for pipeline parallelism, yielding a performance increase of up from 4% to 30%, compared with 1F1B in preempted network scenarios.

Tvm: An Automated End-To-End Optimizing Compiler For Deep Learning

Abstract: There is an increasing need to bring machine learning to a wide diversity of hardware devices. Current frameworks rely on vendor-specific operator libraries and optimize for a narrow range of server-class GPUs. Deploying workloads to new platforms -- such as mobile phones, embedded devices, and accelerators (e.g., FPGAs, ASICs) -- requires significant manual effort. We propose TVM, a compiler that exposes graph-level and operator-level optimizations to provide performance portability to deep learning workloads across diverse hardware back-ends. TVM solves optimization challenges specific to deep learning, such as high-level operator fusion, mapping to arbitrary hardware primitives, and memory latency hiding. It also automates optimization of low-level programs to hardware characteristics by employing a novel, learning-based cost modeling method for rapid exploration of code optimizations. Experimental results show that TVM delivers performance across hardware back-ends that are competitive with state-of-the-art, hand-tuned libraries for low-power CPU, mobile GPU, and server-class GPUs. We also demonstrate TVM's ability to target new accelerator back-ends, such as the FPGA-based generic deep learning accelerator. The system is open sourced and in production use inside several major companies.

DL workloads
operatorlevel optimizations


I. Is Sargan Effective For Facial Attribute Manipulation?

For comparison, StarGAN results were obtained via pre-trained model provided by their authors #REFR . The results in Fig.

We trained the proposed SARGAN model on CelebA dataset to assess the performance of SARGAN on facial attribute manipulation.

13 demonstrate that StarGAN is unable to recover the true input image overall color as well as facial details such as eye and skin colors.

In contrast, SARGAN recovers true input image colors and facial details.

To confirm our analysis, we compute the distance between histograms of input and StarGAN-synthesized images and input and SARGAN-synthesized images, respectively. Fig.

13 shows that SARGAN results are closer to the input images compared to StarGAN. Further, as shown in Fig.

13 , Column 3, StarGAN modifies the input face's hairstyle in addition to the hair color, whereas SARGAN only modifies the hair color.

Sargan: Spatial Attention-Based Residuals For Facial Expression Manipulation

Abstract: Encoder-decoder based architecture has been widely used in the generator of generative adversarial networks for facial manipulation. However, we observe that the current architecture fails to recover the input image color, rich facial details such as skin color or texture and introduces artifacts as well. In this paper, we present a novel method named SARGAN that addresses the above-mentioned limitations from three perspectives. First, we employed spatial attention-based residual block instead of vanilla residual blocks to properly capture the expressionrelated features to be changed while keeping the other features unchanged. Second, we exploited a symmetric encoder-decoder network to attend facial features at multiple scales. Third, we proposed to train the complete network with a residual connection which relieves the generator of pressure to generate the input face image thereby producing the desired expression by directly feeding the input image towards the end of the generator. Both qualitative and quantitative experimental results show that our proposed model performs significantly better than state-ofthe-art methods. In addition, existing models require much larger datasets for training but their performance degrades on out-ofdistribution images. In contrast, SARGAN can be trained on smaller facial expressions datasets, which generalizes well on outof-distribution images including human photographs, portraits, avatars and statues.

Stargan: Unified Generative Adversarial Networks For Multi-Domain Image-To-Image Translation

Abstract: Recent studies have shown remarkable success in image-to-image translation for two domains. However, existing approaches have limited scalability and robustness in handling more than two domains, since different models should be built independently for every pair of image domains. To address this limitation, we propose StarGAN, a novel and scalable approach that can perform image-to-image translations for multiple domains using only a single model. Such a unified model architecture of StarGAN allows simultaneous training of multiple datasets with different domains within a single network. This leads to StarGAN's superior quality of translated images compared to existing models as well as the novel capability of flexibly translating an input image to any desired target domain. We empirically demonstrate the effectiveness of our approach on a facial attribute transfer and a facial expression synthesis tasks.


Trained using Reinforcement Learning from Human Feedback (RLHF) #REFR , ChatGPT has been shown to excel at generating human-like responses to natural language prompts, and has garnered widespread attention for its powerful understanding, reasoning, and dialogue abilities.

These two filtering steps have removed about 265 000 data samples from FreeSound.

ChatGPT-based Transformation To transform the raw descriptions into audio captions, we propose that a well-formed audio caption should possess the following characteristics:

• Be a single, accurate description of the audio content using concise syntax; • Avoid the use of named entities such as people's names, locations, and recording devices that cannot be inferred from the audio signal alone; • Exclude any subjective sound-unrelated information such as personal feelings or opinions; However, online-harvested descriptions are extremely noisy and most of them fail to meet above requirements, particularly those from FreeSound.

Due to the varying characteristics of raw descriptions, it is challenging to design rules that accurately convert them into captions, and doing so would result in a high discard rate similar to what was observed in CC3M.

To tackle the challenge of converting raw descriptions into captions, we propose using ChatGPT, a powerful large language model trained by OpenAI 6 to perform this task automatically.

By designing prompts that account for the characteristics of different data sources, ChatGPT can effectively filter out sound-unrelated information and rewrite raw descriptions in to audio caption-like sentences that meet the requirements we proposed in prompts.

This approach has the potential to significantly reduce the discard rate of raw descriptions and improve the quality of converted captions. Prompts we used are shown in Table I .

In order to make use of ChatGPT's in-context learning ability, several transformation examples are also included in the prompts and they are different for each data sources (ignored in Table I ). These examples can significantly improve the caption quality.

Table II presents examples of the raw descriptions and final processed captions.

It can be observed that ChatGPT can transform non-sentence descriptions (i.e., nouns and phrases) into sentences, remove redundant information that is too specific or is not related to sound, and summarize long sentences into one-sentence high-level audio captions.

Wavcaps: A Chatgpt-Assisted Weakly-Labelled Audio Captioning Dataset For Audio-Language Multimodal Research

Abstract: The advancement of audio-language (AL) multimodal learning tasks has been significant in recent years. However, researchers face challenges due to the costly and timeconsuming collection process of existing audio-language datasets, which are limited in size. To address this data scarcity issue, we introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions. We sourced audio clips and their raw descriptions from web sources and a sound event detection dataset. However, the online-harvested raw descriptions are highly noisy and unsuitable for direct use in tasks such as automated audio captioning. To overcome this issue, we propose a three-stage processing pipeline for filtering noisy data and generating highquality captions, where ChatGPT, a large language model, is leveraged to filter and transform raw descriptions automatically. We conduct a comprehensive analysis of the characteristics of WavCaps dataset and evaluate it on multiple downstream audiolanguage multimodal learning tasks. The systems trained on WavCaps outperform previous state-of-the-art (SOTA) models by a significant margin. Our aspiration is for the WavCaps dataset we have proposed to facilitate research in audio-language multimodal learning and demonstrate the potential of utilizing ChatGPT to enhance academic research. Our dataset and codes are available at https://github.com/XinhaoMei/WavCaps.

Deep Reinforcement Learning From Human Preferences

Abstract: For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any that have been previously learned from human feedback.

natural language prompts
Reinforcement Learning


Inspired by InfoVAE #REFR , we introduce an additional pair of condition-specific MMD regularization loss to learn mutual information between the sampled noise distribution and the Gaussian distribution.

Maximum-Mean Discrepancy (MMD) is to quantify the similarity between two distributions by comparing all of their moments #OTHEREFR . It can be efficiently implemented using a kernel trick.

To be specific, we sample the noisy variable y g t from the diffusion process at time step t conditioned only by the global prior and then compute an MMDregularization loss as:

where K(•, •) is a positive definite kernel to reproduce distributions in the Hilbert space.

The condition-specific MMD regularization is also applied on the local prior, as shown in Figure 1 (a) .

While the general noise estimation loss L captures the complementary information from both priors, the condition-specific MMD regularization maintains the mutual information between each prior and target distribution.

Diffmic: Dual-Guidance Diffusion Network For Medical Image Classification

Abstract: Diffusion Probabilistic Models have recently shown remarkable performance in generative image modeling, attracting significant attention in the computer vision community. However, while a substantial amount of diffusion-based research has focused on generative tasks, few studies have applied diffusion models to general medical image classification. In this paper, we propose the first diffusion-based model (named DiffMIC) to address general medical image classification by eliminating unexpected noise and perturbations in medical images and robustly capturing semantic representation. To achieve this goal, we devise a dual conditional guidance strategy that conditions each diffusion step with multiple granularities to improve step-wise regional attention. Furthermore, we propose learning the mutual information in each granularity by enforcing Maximum-Mean Discrepancy regularization during the diffusion forward process. We evaluate the effectiveness of our DiffMIC on three medical classification tasks with different image modalities, including placental maturity grading on ultrasound images, skin lesion classification using dermatoscopic images, and diabetic retinopathy grading using fundus images. Our experimental results demonstrate that DiffMIC outperforms state-of-the-art methods by a significant margin, indicating the universality and effectiveness of the proposed model.

Infovae: Information Maximizing Variational Autoencoders

Abstract: It has been previously observed that variational autoencoders tend to ignore the latent code when combined with a decoding distribution that is too flexible. This undermines the purpose of unsupervised representation learning. We identify the reason for this short-coming in the regularization term used in the ELBO criterion to match the variational posterior to the latent prior distribution. We show that removing this regularization term leads to a model that can still discover meaningful latent features. Even though ancestral sampling is no longer tractable, sampling is possible using a Markov chain. Furthermore, we propose a class of training criteria that use alternative divergences for the regularization term, generalizing the standard ELBO which employs KL divergence. These models can discover meaningful latent features and allow for tractable ancestral sampling. In particular, we propose an alternative based on Maximum Mean Discrepancy (MMD) that is simple to implement, robust, and has similar or better performance in every quantitative and qualitative metric we experimented on.


The accuracy of vision transformers exceeds that of convolutional neural networks by a considerable margin when the model is pre-trained on huge datasets, such as JFT-300M #REFR .

Vision transformers #OTHEREFR have made a significant impact on the entire field of computer vision, and state of the art models in classification #OTHEREFR , object detection #OTHEREFR , and segmentation #OTHEREFR are now based on vision transformers.

However, the JFT-300M dataset contains 300M images and 375M labels.

It is impossible to sin #OTHEREFR manually label all of these images.

Efforts to automatically label such datasets is still not as accurate as manual labeling.

Self-supervised learning (SSL) is increasing in popularity, as datasets do not need to be labeled for this mode of training #OTHEREFR .

Although SSL removes the burden of labeling large datasets, the effort to collect/download, store, and load these large datasets remains a challenge.

Visual Atoms: Pre-Training Vision Transformers With Sinusoidal Waves

Abstract: Formula-driven supervised learning (FDSL) has been shown to be an effective method for pre-training vision transformers, where ExFractalDB-21k was shown to exceed the pre-training effect of ImageNet-21k. These studies also indicate that contours mattered more than textures when pre-training vision transformers. However, the lack of a systematic investigation as to why these contour-oriented synthetic datasets can achieve the same accuracy as real datasets leaves much room for skepticism. In the present work, we develop a novel methodology based on circular harmonics for systematically investigating the design space of contour-oriented synthetic datasets. This allows us to efficiently search the optimal range of FDSL parameters and maximize the variety of synthetic images in the dataset, which we found to be a critical factor. When the resulting new dataset VisualAtom-21k is used for pre-training ViT-Base, the top-1 accuracy reached 83.7% when fine-tuning on ImageNet-1k. This is close to the top-1 accuracy (84.2%) achieved by JFT-300M pre-training, while the number of images is 1/14. Unlike JFT-300M which is a static dataset, the quality of synthetic datasets will continue to improve, and the current work is a testament to this possibility. FDSL is also free of the common issues associated with real images, e.g. privacy/copyright issues, labeling costs/errors, and ethical biases.

Revisiting Unreasonable Effectiveness Of Data In Deep Learning Era

Abstract: The success of deep learning in vision can be attributed to: (a) models with high capacity; (b) increased computational power; and (c) availability of large-scale labeled data. Since 2012, there have been significant advances in representation capabilities of the models and computational capabilities of GPUs. But the size of the biggest dataset has surprisingly remained constant. What will happen if we increase the dataset size by 10x or 100x? This paper takes a step towards clearing the clouds of mystery surrounding the relationship between `enormous data' and visual deep learning. By exploiting the JFT-300M dataset which has more than 375M noisy labels for 300M images, we investigate how the performance of current vision tasks would change if this data was used for representation learning. Our paper delivers some surprising (and some expected) findings. First, we find that the performance on vision tasks increases logarithmically based on volume of training data size. Second, we show that representation learning (or pre-training) still holds a lot of promise. One can improve performance on many vision tasks by just training a better base model. Finally, as expected, we present new state-of-the-art results for different vision tasks including image classification, object detection, semantic segmentation and human pose estimation. Our sincere hope is that this inspires vision community to not undervalue the data and develop collective efforts in building larger datasets.


The results are still significantly inferior to that with the original high-resolution images #REFR .

Statistical analysis suggests that when image resolution is below 20 × 20 pixels, superresolution techniques can significantly improve human recognition performance on both activity recognition and privacy recognition tasks.

But it is worth noting that the improvement in recognition performance brought about by super-resolution technology is still less than that brought about by increasing the resolution itself.

Such a finding reveals that super-resolution techniques do not provide enough additional information for humans to enhance their perception ability in both activity recognition and visual privacy awareness tasks.

In terms of the impact of the super-resolution technique on the machine's recognition performance, researchers have proved that super-resolution can slightly facilitate vision-based recognition task such as activity recognition #OTHEREFR , object and text recognition #OTHEREFR .

However, the influence of the super-resolution technique is very limited.

In conclusion, the additional visual information introduced by the image super-resolution technique is insufficient to overcome the effect of resolution on the recognition performance of humans and machines.

Therefore, we believe that the effects of image resolution on human (section 6) and the machine's (section 7) ADLs and visual privacy recognition performance are robust against image superresolution techniques.

Modeling The Trade-Off Of Privacy Preservation And Activity Recognition On Low-Resolution Images

Abstract: A computer vision system using low-resolution image sensors can provide intelligent services (e.g., activity recognition) but preserve unnecessary visual privacy information from the hardware level. However, preserving visual privacy and enabling accurate machine recognition have adversarial needs on image resolution. Modeling the trade-off of privacy preservation and machine recognition

Is Image Super-Resolution Helpful For Other Vision Tasks?

Abstract: Despite the great advances made in the field of image super-resolution (ISR) during the last years, the performance has merely been evaluated perceptually. Thus, it is still unclear whether ISR is helpful for other vision tasks. In this paper, we present the first comprehensive study and analysis of the usefulness of ISR for other vision applications. In particular, six ISR methods are evaluated on four popular vision tasks, namely edge detection, semantic image segmentation, digit recognition, and scene recognition. We show that applying ISR to input images of other vision systems does improve their performance when the input images are of low-resolution. We also study the correlation between four standard perceptual evaluation criteria (namely PSNR, SSIM, IFC, and NQM) and the usefulness of ISR to the vision tasks. Experiments show that they correlate well with each other in general, but perceptual criteria are still not accurate enough to be used as full proxies for the usefulness. We hope this work will inspire the community to evaluate ISR methods also in real vision applications, and to adopt ISR as a pre-processing step of other vision tasks if the resolution of their input images is low.