Simple Batch Size + Learning Rate Scaling With Μp

Further, when training models with a batch size smaller than the critical batch size, learning rate should be reduced linearly proportional to the reduction in batch size-consistent with the findings in #REFR .

More precisely, we find that µP learning rate transfers as long as each model size is trained with a batch size roughly consistent with or larger than the critical batch size.

The closer the batch size is to the critical batch size for a given model, the better the loss will be when using the µTransferred learning rate.

We believe the LLaMa 13B model is better than Cerebras-GPT on downstream tasks because it was trained for 4× more tokens, but were unable to get access to test the model ourselves.©2023 Cerebras Systems Inc. All Rights Reserved.

All Cerebras-GPT development and hyperparameter tuning was evaluated using the Pile validation set.3 Pile test loss is crossentropy in nats/token.

We correct all crossentropy results for different vocabularies to be comparable to the GPT-2 vocabulary.©2023 Cerebras Systems Inc. All Rights Reserved.

Here, we report accuracy result from each model predictions using token-level probability, consistent with reported results in the GPT-NeoX paper.

We report additional accuracy measures in Appendix C.2.©2023 Cerebras Systems Inc. All Rights Reserved.

Cerebras-Gpt: Open Compute-Optimal Language Models Trained On The Cerebras Wafer-Scale Cluster

Abstract: We study recent research advances that improve large language models through efficient pretraining and scaling, and open datasets and tools. We combine these advances to introduce Cerebras-GPT, a family of open compute-optimal language models scaled from 111M to 13B parameters. We train Cerebras-GPT models on the Eleuther Pile dataset following DeepMind Chinchilla scaling rules for efficient pre-training (highest accuracy for a given compute budget). We characterize the predictable power-law scaling and compare Cerebras-GPT with other publicly-available models to show all Cerebras-GPT models have state-ofthe-art training efficiency on both pre-training and downstream objectives. We describe our learnings including how Maximal Update Parameterization (µP) can further improve large model scaling, improving accuracy and hyperparameter predictability at scale. We release our pre-trained models and code, making this paper the first open and reproducible work comparing compute-optimal model scaling to models trained on fixed dataset sizes. Cerebras-GPT models are available on HuggingFace: https://huggingface.co/cerebras. Recent research in large language models (LLMs) shows important advances that can improve LLM quality and efficiency. Scaling law studies show predictable and significant improvements in model performance by increasing model and dataset size (Hestness et al., 2017; Kaplan et al., 2020). Language models can also be improved just by training on more data (Hoffmann et al., 2022; Touvron et al., 2023). Recent works, such as Maximal Update Parameterization (µP), also show techniques to improve training stability and performance as models scale up (e.g., Bachlechner et al. (2020); Yang et al. (2021)). Concurrently with these advances, the research community has trained and released many open-source models. Models like GPT-J, GPT-NeoX, OPT, and Pythia have each held state-of-the-art accuracy for open source models for their size, and these models can be tested and used simply by downloading the pretrained weights (

Measuring The Effects Of Data Parallelism On Neural Network Training

Abstract: Recent hardware developments have made unprecedented amounts of data parallelism available for accelerating neural network training. Among the simplest ways to harness next-generation accelerators is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally characterize the effects of increasing the batch size on training time, as measured in the number of steps necessary to reach a goal out-of-sample error. Eventually, increasing the batch size will no longer reduce the number of training steps required, but the exact relationship between the batch size and how many training steps are necessary is of critical importance to practitioners, researchers, and hardware designers alike. We study how this relationship varies with the training algorithm, model, and data set and find extremely large variation between workloads. Along the way, we reconcile disagreements in the literature on whether batch size affects model quality. Finally, we discuss the implications of our results for efforts to train neural networks much faster in the future.


Bottleneck Attention module (BAM) #REFR also uses double attention modules to add their attention matrices to get the final attention map.

Attention mechanisms have attracted great attention since it was successfully applied in natural language processing tasks #OTHEREFR , which is mainly used to help networks concentrating on partial information of data flow and generating the corresponding dimensional weight maps.

When it comes to the image processing task, there are mainly three types of attention mechanisms: channel attention (CA) #OTHEREFR , spatial attention (SA) #OTHEREFR and pixel attention (PA) #OTHEREFR , depending on their generated weight maps' dimension level.

Self-Attention #OTHEREFR is also very popular apart from the three.

Among the above mechanisms, channel attention is the first to propose in SE-Net #OTHEREFR and then polished in SK-Net #OTHEREFR .

After that, attempts to incorporate both channel attention and spatial attention for stronger attention capability in Convolutional Block Attention Module (CBAM) #OTHEREFR are tried several times.

SA-Net #OTHEREFR splits the channel feature maps separately into the channel attention module and spatial attention module.

Apart from these, there also exists the temporal attention, i.e. the attention in time level #OTHEREFR .

Parameter-Free Channel Attention For Image Classification And Super-Resolution

Abstract: The channel attention mechanism is a useful technique widely employed in deep convolutional neural networks to boost the performance for image processing tasks, e.g., image classification and image super-resolution. It is usually designed as a parameterized sub-network and embedded into the convolutional layers of the network to learn more powerful feature representations. However, current channel attention induces more parameters and therefore leads to higher computational costs. To deal with this issue, in this work, we propose a Parameter-Free Channel Attention (PFCA) module to boost the performance of popular image classification and image super-resolution networks, but completely sweep out the parameter growth of channel attention. Experiments on CIFAR-100, ImageNet, and DIV2K validate that our PFCA module improves the performance of ResNet on image classification and improves the performance of MSRResNet on image super-resolution tasks, respectively, while bringing little growth of parameters and FLOPs.

Bam: Bottleneck Attention Module

Abstract: Recent advances in deep neural networks have been developed via architecture search for stronger representational power. In this work, we focus on the effect of attention in general deep neural networks. We propose a simple and effective attention module, named Bottleneck Attention Module (BAM), that can be integrated with any feed-forward convolutional neural networks. Our module infers an attention map along two separate pathways, channel and spatial. We place our module at each bottleneck of models where the downsampling of feature maps occurs. Our module constructs a hierarchical attention at bottlenecks with a number of parameters and it is trainable in an end-to-end manner jointly with any feed-forward models. We validate our BAM through extensive experiments on CIFAR-100, ImageNet-1K, VOC 2007 and MS COCO benchmarks. Our experiments show consistent improvement in classification and detection performances with various models, demonstrating the wide applicability of BAM. The code and models will be publicly available.

double attention modules
Bottleneck Attention module


There is a lot of community interest in distributed and federated problems in machine learning #REFR .

We believe that the inference setting is under-studied despite a clear need in a number of different domains.

This paper focuses its discussion on the execution layer for decentralized prediction systems.

The paper contributes the communication primitives needed to make such systems work in practice.

We find that decentralized prediction can give system designers new knobs to enable real-time predictions.

Our results show: (1) a clear value for decentralization, (2) that EdgeServe is an effective architecture for supporting decentralization, and (3) core techniques in EdgeServe, such as lazy data routing, are broadly applicable to modern stream processing.

Edgeserve: An Execution Layer For Decentralized Prediction

Abstract: The relevant features for a machine learning task may be aggregated from data sources collected on different nodes in a network. This problem, which we call decentralized prediction, creates a number of interesting systems challenges in managing data routing, placing computation, and time-synchronization. This paper presents EdgeServe, a machine learning system that can serve decentralized predictions. EdgeServe relies on a low-latency message broker to route data through a network to nodes that can serve predictions. EdgeServe relies on a series of novel optimizations that can trade-off computation, communication, and accuracy. We evaluate EdgeServe on three decentralized prediction tasks: (1) multi-camera object tracking, (2) network intrusion detection, and (3) human activity recognition.

Federated Optimization:Distributed Optimization Beyond The Datacenter

Abstract: We introduce a new and increasingly relevant setting for distributed optimization in machine learning, where the data defining the optimization are distributed (unevenly) over an extremely large number of \nodes, but the goal remains to train a high-quality centralized model. We refer to this setting as Federated Optimization. In this setting, communication efficiency is of utmost importance. ::: A motivating example for federated optimization arises when we keep the training data locally on users' mobile devices rather than logging it to a data center for training. Instead, the mobile devices are used as nodes performing computation on their local data in order to update a global model. We suppose that we have an extremely large number of devices in our network, each of which has only a tiny fraction of data available totally; in particular, we expect the number of data points available locally to be much smaller than the number of devices. Additionally, since different users generate data with different patterns, we assume that no device has a representative sample of the overall distribution. ::: We show that existing algorithms are not suitable for this setting, and propose a new algorithm which shows encouraging experimental results. This work also sets a path for future research needed in the context of federated optimization.


Comparison With State-Of-The-Art Methods

Finally, on the FER+ #REFR dataset, ARBEx acquires an accuracy score of 93.09%, which significantly outperforms every other model tested.

Using five separate datasets -AffWild2, RAF-DB, JAFFE, FERG-DB, and FER+ (explained in 4.1), the table shows the comparison of the accuracy of multiple State-of-the-Art facial expression learning methods.

In this study, the models SCN #OTHEREFR , RAN #OTHEREFR , RUL #OTHEREFR , EfficientFace #OTHEREFR , POSTER #OTHEREFR , POSTER++ #OTHEREFR , and ARBEx are compared.

Upon investigation of the results, it is apparent that AR-BEx outperforms all other models across all datasets, attaining the highest accuracy scores for each dataset.

Specifically, ARBEx earns an accuracy score of 72.48% on the AffWild2 #OTHEREFR dataset, which is significantly higher than POSTER++, which has an accuracy score of 69.18%.

ARBEx outperforms every other model in the study, with accuracy scores on the RAF-DB #OTHEREFR , #OTHEREFR , FERG-DB #OTHEREFR and JAFFE #OTHEREFR , #OTHEREFR datasets of 92.47%, 98.18% and 96.67%, respectively.

Our novel reliability balancing section reduces all kinds of biases, resulting in exceptional performance in all circumstances.

Arbex: Attentive Feature Extraction With Reliability Balancing For Robust Facial Expression Learning

Abstract: In this paper, we introduce a framework ARBEx, a novel attentive feature extraction framework driven by Vision Transformer with reliability balancing to cope against poor class distributions, bias, and uncertainty in the facial expression learning (FEL) task. We reinforce several data pre-processing and refinement methods along with a window-based cross-attention ViT to squeeze the best of the data. We also employ learnable anchor points in the embedding space with label distributions and multi-head self-attention mechanism to optimize performance against weak predictions with reliability balancing, which is a strategy that leverages anchor points, attention scores, and confidence values to enhance the resilience of label predictions. To ensure correct label classification and improve the model's discriminative power, we introduce anchor loss, which encourages large margins between anchor points. Additionally, the multi-head self-attention mechanism, which is also trainable, plays an integral role in identifying accurate labels. This approach provides critical elements for improving the reliability of predictions and has a substantial positive effect on final prediction capabilities. Our adaptive model can be integrated with any deep neural network to forestall challenges in various recognition tasks. Our strategy outperforms current state-of-the-art methodologies, according to extensive experiments conducted in a variety of contexts.

Training Deep Networks For Facial Expression Recognition With Crowd-Sourced Label Distribution

Abstract: Crowd sourcing has become a widely adopted scheme to collect ground truth labels. However, it is a well-known problem that these labels can be very noisy. In this paper, we demonstrate how to learn a deep convolutional neural network (DCNN) from noisy labels, using facial expression recognition as an example. More specifically, we have 10 taggers to label each input image, and compare four different approaches to utilizing the multiple labels: majority voting, multi-label learning, probabilistic label drawing, and cross-entropy loss. We show that the traditional majority voting scheme does not perform as well as the last two approaches that fully leverage the label distribution. An enhanced FER+ data set with multiple labels for each face image will also be shared with the research community.


Second, a diverse of ASG functions can be produced to achieve much richer encoding compared with a few fixed basis encoding functions in existing methods #REFR .

Finally, we apply the sigmoid activation function to the combined color to obtain the viewdependent color:

We can also interpret the proposed rendering equation encoding as a more advanced view direction encoding method.

Our rendering equation encoding has two-fold benefits compared with popular frequency encoding #OTHEREFR and sphere harmonics #OTHEREFR .

First, every point now has its own independent encoding functions controlled by the predicted bandwidths in the ASGs, while the encoding functions are fixed for all points in existing works.

Multiscale Tensor Decomposition And Rendering Equation Encoding For View Synthesis

Abstract: Rendering novel views from captured multi-view images has made considerable progress since the emergence of the neural radiance field. This paper aims to further advance the quality of view rendering by proposing a novel approach dubbed the neural radiance feature field (NRFF) which represents scenes in the feature space. We first propose a multiscale tensor decomposition scheme to organize learnable features so as to represent scenes from coarse to fine scales. We demonstrate many benefits of the proposed multiscale representation, including more accurate scene shape and appearance reconstruction, and faster convergence compared with the single-scale representation. Instead of encoding view directions to model view-dependent effects, we further propose to encode the rendering equation in the feature space by employing the anisotropic spherical Gaussian mixture predicted from the proposed multiscale representation. The proposed NRFF improves state-of-the-art rendering results by over 1 dB in PSNR on both the NeRF and NSVF synthetic datasets. A significant improvement has also been observed on the real-world Tanks & Temples dataset.

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.


We use an off-the-shelf multi-label classifier based on ResNet50 #REFR to obtain predicted logits.

w/o L multiID denotes the model without multi-view ID loss.

ated by the average pose distance(APD), which is root mean squared error between the pose encodings estimated by the pretrained 3D face detector #OTHEREFR .

Time metric indicates the average inference time (encoding time and generation time) for one image computed on one Tesla V100 GPU.

Our method outperforms baselines on ID and APD metrics and is significantly faster than IDE-3D and 3D-Inv when inference.

We show the comparison of face attribute editing against the baselines in Table 3 .

Attribute altering (AA) measures the change of the desired attribute, which is the attribute logit change ∆l t #OTHEREFR when detecting attribute t by the classifier (pretrained on CelebA #OTHEREFR ).

∆l t is normalized by σ(l) #OTHEREFR , which is the standard deviation calculated from the logits of CelebA-HQ dataset.

We evaluate the precision of attribute editing with attribute dependency (AD) #OTHEREFR , which measures the degree of change on other attributes when modifying along a certain attribute editing direction, as measured by classifiers. Our method performs better than the previous method. More attribute editing results are provided in Appendix. Qualitative Evaluation.

We present examples of the inversion and editing results in Figure 4 .

We sample 9 images for each source image with yaw = [−30

Preim3D: 3D Consistent Precise Image Attribute Editing From A Single Image

Abstract: 'an Satellite Control Center 3 RealAI black hair inversion smile age input eyeglasses shape Figure 1. 3D consistent precise inversion and editing. Our method enables reconstructing texture and geometry from a single real image and allows one to perform a list of attributes editing sequentially. The yaw angles of the second to sixth columns are [−30 • , −20 • , 0 • , 20 • , 30 • ]. The last column is the shape of the sixth column.

Arbitrary Style Transfer In Real-Time With Adaptive Instance Normalization

Abstract: Gatys et al. recently introduced a neural algorithm that renders a content image in the style of another image, achieving so-called style transfer. However, their framework requires a slow iterative optimization process, which limits its practical application. Fast approximations with feed-forward neural networks have been proposed to speed up neural style transfer. Unfortunately, the speed improvement comes at a cost: the network is usually tied to a fixed set of styles and cannot adapt to arbitrary new styles. In this paper, we present a simple yet effective approach that for the first time enables arbitrary style transfer in real-time. At the heart of our method is a novel adaptive instance normalization (AdaIN) layer that aligns the mean and variance of the content features with those of the style features. Our method achieves speed comparable to the fastest existing approach, without the restriction to a pre-defined set of styles. In addition, our approach allows flexible user controls such as content-style trade-off, style interpolation, color & spatial controls, all using a single feed-forward neural network.


It is found that the transmit power is non-increasing after each iteration, which can be proved in a similar way as that in #REFR .

The proposed BCD-based algorithm for solving P1 is summarized in Algorithm 1.

Thus, the value of the objective function in P1 will converge to a stationary point. Then, we analyze the computational complexity of Algorithm 1.

According to #OTHEREFR , the complexities of solving P2.1 and P3.4 are

2 )), respectively, where n 1 = N 2 and n 2 = 2M + K + 2Q.

Therefore, the overall computational complexity of Algorithm 1 is O(

Transmit Power Minimization For Star-Ris Empowered Symbiotic Radio Communications

Abstract: In this paper, we propose a simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR-RIS) empowered transmission scheme for symbiotic radio (SR) systems to make more flexibility for network deployment and enhance system performance. The STAR-RIS is utilized to not only beam the primary signals from the base station (BS) towards multiple primary users on the same side of the STAR-RIS, but also achieve the secondary transmission to the secondary users on another side. We consider both the broadcasting signal model and unicasting signal model at the BS. For each model, we aim for minimizing the transmit power of the BS by designing the active beamforming and simultaneous reflection and transmission coefficients under the practical phase correlation constraint. To address the challenge of solving the formulated problem, we propose a block coordinate descent based algorithm with the semidefinite relaxation, penalty dual decomposition and successive convex approximation methods, which decomposes the original problem into one sub-problem about active beamforming and the other sub-problem about simultaneous reflection and transmission coefficients, and iteratively solve them until the convergence is achieved. Numerical results indicate that the proposed scheme can reduce up to 150.6% transmit power compared to the backscattering device enabled scheme.

Intelligent Reflecting Surface Assisted Non-Orthogonal Multiple Access

Abstract: Intelligent reflecting surface (IRS) which consists of a large number of low-cost passive reflecting elements and can digitally manipulating electromagnetic waves, is a new and disruptive technology to achieve spectrum- and energy-efficient as well as cost-efficient wireless networks. In this paper, we consider an IRS-assisted non-orthogonal-multiple-access (NOMA) system in which a base station (BS) transmits superposed downlink signals to multiple users. In order to optimize the rate performance and ensure user fairness, we maximize the minimum decoding signal-to-interference-plus-noise-ratio (SINR) (i.e., equivalently the rate) of all users, by jointly optimizing the power allocation at the BS and the phase shifts at the passive IRS. However, the formulated problem is non-convex and difficult to be solved optimally. By leveraging the block coordinated decent, successive convex optimization and semidefinite relaxation techniques, an efficient algorithm is further proposed to obtain a sub-optimal solution. The convergence is proved and the complexity is analyzed for the proposed algorithm. Also, a low-complexity solving scheme is proposed. Simulation results show that the IRS can enhance the rate performance for downlink NOMA systems significantly even for the scenario in which users have the same or comparable channel strength(es), and the practical IRS with a 3-bit phase quantizer is sufficient to ensure the rate degradation of less than 3.4% compared to the case of continuous phase shifts.


They are shown to over-emphasize proximity information at the expense of structural information #REFR 38] .

Early unsupervised methods for learning node representation are traditionally based on matrix factorization and random walks.

Matrix factorization methods calculate losses with handcrafted similarity metrics to build vector representations for each node with latent features #OTHEREFR .

The inspiration for random walk-based unsupervised methods for learning node representation comes from the effectiveness of the NLP method.

DeepWalk #OTHEREFR and Node2Vec #OTHEREFR optimize node embeddings by matching nodes' co-occurrence rate on short random walks over graphs.

Also, they are limited to preserving the similarity of adjacent nodes and cannot extend to heterophily settings. AutoEncoder-based.

Graph autoencoders, e.g., GAE and VGAE [19] and their follow-up work ARVGE #OTHEREFR with an adversarial regularization framework use two-layer GCN as their encoder and consider that impose the topological closeness of nodes in the graph structure on the latent space by predicting the first-order neighbors.

GAEs over-emphasize proximity information and are also based on the assumption that connected nodes should be more similar. Unsupervised GNNs.

Unsupervised GNN-based methods #OTHEREFR , on the other hand, propagate features and aggregate them within various graph neighborhoods via different mechanisms (e.g., averaging, LSTM), where the node representations evolve over multiple rounds of propagation with becoming prohibitively similar.

Uniform aggregation and update in GNNs ignore the difference in information between similar and dissimilar neighbors.

Multi-View Graph Representation Learning Beyond Homophily

Abstract: Unsupervised graph representation learning(GRL) aims to distill diverse graph information into task-agnostic embeddings without label supervision. Due to a lack of support from labels, recent representation learning methods usually adopt self-supervised learning, and embeddings are learned by solving a handcrafted auxiliary task(so-called pretext task). However, partially due to the irregular non-Euclidean data in graphs, the pretext tasks are generally designed under homophily assumptions and cornered in the low-frequency signals, which results in significant loss of other signals, especially high-frequency signals widespread in graphs with heterophily. Motivated by this limitation, we propose a multi-view perspective and the usage of diverse pretext tasks to capture different signals in graphs into embeddings. A novel framework, denoted as Multi-view Graph Encoder(MVGE), is proposed, and a set of key designs are identified. More specifically, a set of new pretext tasks are designed to encode different types of signals, and a straightforward operation is propxwosed to maintain both the commodity and personalization in both the attribute and the structural levels. Extensive experiments on synthetic and real-world network datasets show that the node representations learned with MVGE achieve significant performance improvements in three different downstream tasks, especially on graphs with heterophily.

Struc2Vec: Learning Node Representations From Structural Identity

Abstract: Structural identity is a concept of symmetry in which network nodes are identified according to the network structure and their relationship to other nodes. Structural identity has been studied in theory and practice over the past decades, but only recently has it been addressed with representational learning techniques. This work presents struc2vec, a novel and flexible framework for learning latent representations for the structural identity of nodes. struc2vec uses a hierarchy to measure node similarity at different scales, and constructs a multilayer graph to encode structural similarities and generate structural context for nodes. Numerical experiments indicate that state-of-the-art techniques for learning node representations fail in capturing stronger notions of structural identity, while struc2vec exhibits much superior performance in this task, as it overcomes limitations of prior approaches. As a consequence, numerical experiments indicate that struc2vec improves performance on classification tasks that depend more on structural identity.


In addition, we conduct experiments with TinyBERT (2 layers version) #REFR to assess the generalizability of our results.

For the lexical retrieval with BM25, we use the similarity function of Elasticsearch [9] .

We cap the query length at 30 tokens and the passage length at 200 tokens following prior work #OTHEREFR .

Table 2 shows a comparison of the effectiveness of CE human and CE ChatGPT .

Please note that for both models, during inference, we evaluate their effectiveness in retrieving human responses in the supervised or zero-shot settings.

We choose MiniLM (12 layers version) #OTHEREFR for the experiments due to its competitive results in comparison to BERT re-ranker #OTHEREFR while being three times smaller and six times faster.

Generating Synthetic Documents For Cross-Encoder Re-Rankers: A Comparative Study Of Chatgpt And Human Experts

Abstract: We investigate the usefulness of generative Large Language Models (LLMs) in generating training data for cross-encoder re-rankers in a novel direction: generating synthetic documents instead of synthetic queries. We introduce a new dataset, ChatGPT-RetrievalQA, and compare the effectiveness of models fine-tuned on LLMgenerated and human-generated data. Data generated with generative LLMs can be used to augment training data, especially in domains with smaller amounts of labeled data. We build ChatGPT-RetrievalQA based on an existing dataset, human ChatGPT Comparison Corpus (HC3), consisting of public question collections with human responses and answers from ChatGPT. We fine-tune a range of cross-encoder re-rankers on either human-generated or ChatGPTgenerated data. Our evaluation on MS MARCO DEV, TREC DL'19, and TREC DL'20 demonstrates that cross-encoder re-ranking models trained on ChatGPT responses are statistically significantly more effective zero-shot re-rankers than those trained on human responses. In a supervised setting, the human-trained re-rankers outperform the LLM-trained re-rankers. Our novel findings suggest that generative LLMs have high potential in generating training data for neural retrieval models. Further work is needed to determine the effect of factually wrong information in the generated responses and test our findings' generalizability with open-source LLMs. We release our data, code, and cross-encoders checkpoints for future work. 1 2 We refer to the cross-encoders fine-tuned on ChatGPT-generated and humangenerated responses as CE ChatGPT and CE human , respectively.

Tinybert: Distilling Bert For Natural Language Understanding

Abstract: Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive and memory intensive, so it is difficult to effectively execute them on some resource-restricted devices. To accelerate inference and reduce model size while maintaining accuracy, we firstly propose a novel transformer distillation method that is a specially designed knowledge distillation (KD) method for transformer-based models. By leveraging this new KD method, the plenty of knowledge encoded in a large teacher BERT can be well transferred to a small student TinyBERT. Moreover, we introduce a new two-stage learning framework for TinyBERT, which performs transformer distillation at both the pre-training and task-specific learning stages. This framework ensures that TinyBERT can capture both the general-domain and task-specific knowledge of the teacher BERT.TinyBERT is empirically effective and achieves more than 96% the performance of teacher BERTBASE on GLUE benchmark while being 7.5x smaller and 9.4x faster on inference. TinyBERT is also significantly better than state-of-the-art baselines on BERT distillation, with only about 28% parameters and about 31% inference time of them.


The forward gain matrix K k+1 in case of I-KF is deterministic #REFR .

The I-CKF recursions follow from the non-additive noise formulation of CKF #OTHEREFR with a higher (n x + n y )-dimensional cubature points.

However, unlike forward CKF, these cubature points are generated only once for the time update taking into account the process noise statistics (covariance R).

For I-EKF #OTHEREFR , this matrix depends on only the linearized system model at the state estimates.

Hence, I-KF and I-EKF treat K k+1 as a time-varying parameter of the inverse filter's state transition.

But the gain matrix in CKF explicitly depends on the state estimates through the covariances computed from the generated cubature points.

Thus, it is not treated as a parameter of (10).

Inverse Cubature And Quadrature Kalman Filters

Abstract: Recent developments in counter-adversarial system research have led to the development of inverse stochastic filters that are employed by a defender to infer the information its adversary may have learned. Prior works addressed this inverse cognition problem by proposing inverse Kalman filter (I-KF) and inverse extended KF (I-EKF), respectively, for linear and non-linear Gaussian state-space models. However, in practice, many counter-adversarial settings involve highly non-linear system models, wherein EKF's linearization often fails. In this paper, we consider the efficient numerical integration techniques to address such nonlinearities and, to this end, develop inverse cubature KF (I-CKF) and inverse quadrature KF (I-QKF). We derive the stochastic stability conditions for the proposed filters in the exponential-mean-squared-boundedness sense. Numerical experiments demonstrate the estimation accuracy of our I-CKF and I-QKF with the recursive Cramér-Rao lower bound as a benchmark.

How To Calibrate Your Adversary'S Capabilities? Inverse Filtering For Counter-Autonomous Systems

Abstract: We consider an adversarial Bayesian signal processing problem involving “us” and an “adversary”. The adversary observes our state in noise; updates its posterior distribution of our state and then chooses an action based on this posterior. Given knowledge of “our” state and sequence of adversary's actions observed in noise, we consider three problems: (i) How can the adversary's posterior distribution be estimated? Estimating the posterior is an inverse filtering problem involving a random measure - we formulate and solve several versions of this problem in a Bayesian setting. (ii) How can the adversary's observation likelihood be estimated? This tells us how accurate the adversary's sensors are. We compute the maximum likelihood estimator for the adversary's observation likelihood given our measurements of the adversary's actions where the adversary's actions are in response to estimating our state. (iii) How can the state be chosen by us to minimize the covariance of the estimate of the adversary's observation likelihood? “Our” state can be viewed as a probe signal which causes the adversary to act; so choosing the optimal state sequence is an input design problem. The above questions are motivated by the design of counter-autonomous systems: given measurements of the actions of a sophisticated autonomous adversary, how can our counter-autonomous system estimate the underlying belief of the adversary, predict future actions and therefore guard against these actions.


To analyze the state distribution discrepancy in BC, we first introduce a useful lemma #REFR . Lemma 3.1.

Let T (s) = 1 denotes the teacher policy π t takes control and T (s) = 0 means otherwise. Then π b can be represented as

One issue with the joint control is that the student policy π s is trained with samples collected by the behavior policy π b , whose action distribution is not always aligned with π s .

A large state distribution discrepancy between two policies d π b − d πs 1 can cause distributional shift and ruin the training.

A similar problem exists in behavior cloning (BC), though in BC no intervention is involved and π s learns from samples all collected by the teacher policy π t .

The state distribution discrepancy between the teacher policy π t and the student policy π s is bounded by their expected policy discrepancy:

We apply the lemma to the setting of intervention-based RL and derive a bound for

Guarded Policy Optimization With Imperfect Online Demonstrations

Abstract: The Teacher-Student Framework (TSF) is a reinforcement learning setting where a teacher agent guards the training of a student agent by intervening and providing online demonstrations. Assuming optimal, the teacher policy has the perfect timing and capability to intervene in the learning process of the student agent, providing safety guarantee and exploration guidance. Nevertheless, in many real-world settings it is expensive or even impossible to obtain a well-performing teacher policy. In this work, we relax the assumption of a well-performing teacher and develop a new method that can incorporate arbitrary teacher policies with modest or inferior performance. We instantiate an Off-Policy Reinforcement Learning algorithm, termed Teacher-Student Shared Control (TS2C), which incorporates teacher intervention based on trajectory-based value estimation. Theoretical analysis validates that the proposed TS2C algorithm attains efficient exploration and substantial safety guarantee without being affected by the teacher's own performance. Experiments on various continuous control tasks show that our method can exploit teacher policies at different performance levels while maintaining a low training cost. Moreover, the student policy surpasses the imperfect teacher policy in terms of higher accumulated reward in held-out testing environments. Code is available at https://metadriverse.github.io/TS2C.

Constrained Policy Optimization

Abstract: For many applications of reinforcement learning it can be more convenient to specify both a reward function and constraints, rather than trying to design behavior through the reward function. For example, systems that physically interact with or around humans should satisfy safety constraints. Recent advances in policy search algorithms (Mnih et al., 2016, Schulman et al., 2015, Lillicrap et al., 2016, Levine et al., 2016) have enabled new capabilities in high-dimensional control, but do not consider the constrained setting. ::: We propose Constrained Policy Optimization (CPO), the first general-purpose policy search algorithm for constrained reinforcement learning with guarantees for near-constraint satisfaction at each iteration. Our method allows us to train neural network policies for high-dimensional control while making guarantees about policy behavior all throughout training. Our guarantees are based on a new theoretical result, which is of independent interest: we prove a bound relating the expected returns of two policies to an average divergence between them. We demonstrate the effectiveness of our approach on simulated robot locomotion tasks where the agent must satisfy constraints motivated by safety.


Furthermore, subsequently VGG #REFR This research was supported in part by the National Science Foundation under grant number CCF-1954749.

T ENSORS are multidimensional arrays indexed by three or more indices.

An N th -order tensor is the tensor product of N vector spaces. Third-order tensors have three indices as shown in Fig. 1 .

In special cases, first-order tensors represent vectors, and second-order tensors represent matrices.

Convolutional neural networks (CNNs) have outperformed traditional techniques for image recognition tasks.

In 2012, AlexNet #OTHEREFR achieved about 80% top-5 accuracy on the ImageNet dataset #OTHEREFR .

and GoogleNet #OTHEREFR achieved about 90% top-5 accuracy with the same dataset.

On the ImageNet dataset, ResNet #OTHEREFR with a depth of up to 152 layers achieved 3.57% top-5 error.

Executing CNNs for computer vision tasks on mobile devices is gaining more and more attention.

Common methods to reduce the size of CNNs include: sparsification #OTHEREFR - #OTHEREFR , quantization #OTHEREFR , #OTHEREFR , structural pruning #OTHEREFR - #OTHEREFR , and low-rank approximation #OTHEREFR - #OTHEREFR .

The use of low-rank approximations is inspired by #OTHEREFR which showed that the neural network parameters are highly redundant.

Tensor Decomposition For Model Reduction In Neural Networks: A Review

Abstract: Modern neural networks have revolutionized the fields of computer vision (CV) and Natural Language Processing (NLP). They are widely used for solving complex CV tasks and NLP tasks such as image classification, image generation, and machine translation. Most state-of-the-art neural networks are over-parameterized and require a high computational cost. One straightforward solution is to replace the layers of the networks with their low-rank tensor approximations using different tensor decomposition methods. This paper reviews six tensor decomposition methods and illustrates their ability to compress model parameters of convolutional neural networks (CNNs), recurrent neural networks (RNNs) and Transformers. The accuracy of some compressed models can be higher than the original versions. Evaluations indicate that tensor decompositions can achieve significant reductions in model size, run-time and energy consumption, and are well suited for implementing neural networks on edge devices.

Very Deep Convolutional Networks For Large-Scale Image Recognition

Abstract: In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

subsequently VGG
National Science Foundation


Thus, in Phase I, we solve problems (7), #REFR , and (8) to obtain the minimum cost, hence the optimal trajectory for Phase I.

The disruption in (5c) is evaluated with parameters γ x = 0.5, γ v = 0.5.

When any of the problems (13), #OTHEREFR , or (15) is infeasible or whenever the optimal trajectory of C does not converge, we relax the terminal time with a relaxation rate λ = 1.8.

The numerical solutions to the optimization problems are obtained using an interior point optimizer (IPOPT) on an Intel(R) Core(TM) i7-8700 3.20GHz.

As discussed in Section III, in order for CAV C to evaluate the cost of this policy, it breaks down its trajectory into two phases if its initial position is behind the HDV.

The weights α {t,u,v} in (7) and (9) are set to 0.55, 0.2, and 0.25, respectively.

The maximum maneuver time is set as T = 15s.

If any of the OCPs is infeasible in this phase, its corresponding cost is set as "Inf ".

The results are shown in Table I , where we see that, in this case, it is optimal for CAV C to travel with constant acceleration and t * 1 = 3.53s. Proceeding to Phase II, the initial conditions are

Cooperative Lane Changing In Mixed Traffic Can Be Robust To Human Driver Behavior

Abstract: We derive time and energy-optimal control policies for a Connected Autonomous Vehicle (CAV) to complete lane change maneuvers in mixed traffic. The interaction between CAVs and Human-Driven Vehicles (HDVs) requires designing the best possible response of a CAV to actions by its neighboring HDVs. This interaction is formulated using a bilevel optimization setting with an appropriate behavioral model for an HDV's. Then, an iterated best response (IBR) method is used to determine a Nash equilibrium. However, we also show that when a common and simple-to-detect condition applies, the optimal lane-changing policy is in fact independent of HDV behavior with a CAV changing lanes by cooperating with another CAV in the target lane and always merging ahead of it. Thus, the dependence on the interaction between CAVs and HDVs may be eliminated in such cases. Simulation results are included to show the effectiveness of our controllers in terms of cost, safety guarantees, and disruption to the traffic flow when uncontrollable HDVs are present.

Nonconvex Consensus Admm For Cooperative Lane Change Maneuvers Of Connected Automated Vehicles

Abstract: Connected automated vehicles (CAV) offer huge potential to improve the performance of automated vehicles (AV) without communication capabilities, especially in situations when the vehicles (or agents) need to be cooperative to accomplish their maneuver. Lane change maneuvers in dense traffic, e.g., are very challenging for non-connected AVs. To alleviate this problem, we propose a holistic distributed lane change control scheme for CAVs which relies on vehicle-to-vehicle communication. The originally centralized optimal control problem is embedded into a consensus-based Alternating Direction Method of Multipliers framework to solve it in a distributed receding horizon fashion. Although agent dynamics render the underlying optimal control problem nonconvex, we propose a problem reformulation that allows to derive convergence guarantees. In the distributed setting, every agent needs to solve a nonlinear program (NLP) locally. To obtain a real time solution of the local NLPs, we utilize the optimization engine OpEn which implements the proximal averaged Newton method for optimal control (PANOC). Simulation results prove the efficacy and real time capability of our approach.


Appendix C. Detailed Network Architecture

Arrows indicate the diseased areas in the entire reconstructions, which are diminished in the pseudo-normal reconstructions. Gd, gadolinium. et al., 2021; #REFR .

Moreover, the fourth row presents the ground-truth segmentation labels for the tumor-associated regions, and the fifth row shows the predicted segmentation labels that were decoded by the label decoder taking the abnormal ACs as input.

Note that the segmentation was trained only for the tumor-associated regions and not for the normal anatomy-associated regions, as shown in the results.

One may argue that the quality of the reconstructed images and segmentation labels was insufficient, as observed from the blurred and rounded appearance that did not recover the detailed image characteristics.

These tendencies are reasonable because the image information is compressed because of the limited size of the latent representation, which raises a trade-off between the reconstruction qualities and latent size (Kobayashi The input images, the entire reconstructions from entire anatomy codes (ACs), and the pseudo-normal reconstructions from normal ACs are presented in the first, second, and third rows, respectively.

The ground-truth and predicted segmentation labels are shown in the fourth and fifth rows, respectively.

Indeed, we did not pursue the generation quality of the reconstructions as a primary purpose because the lower dimension of the latent representations can be advantageous for computational efficiency in similarity search.

Besides, although the detailed part of the image characteristics was not perfectly reproduced, it was still sufficient for recognizing the anatomical location and presence of abnormalities in the reconstructed images.

Qualitative evaluation of the semantically organized latent space We applied t-distributed stochastic neighbor embedding (t-SNE) analysis (van der Maaten and Hinton, 2011) to evaluate how the latent space was organized according to the semantics by changing the margin parameter (see Section 2.2.3).

We randomly selected 50 individual volumes from each dataset and extracted entire ACs and normal ACs in an image-wise manner.

Sketch-Based Medical Image Retrieval

Abstract: The amount of medical images stored in hospitals is increasing faster than ever; however, utilizing the accumulated medical images has been limited. This is because existing content-based medical image retrieval (CBMIR) systems usually require example images to construct query vectors; nevertheless, example images cannot always be prepared. Besides, there can be images with rare characteristics that make it difficult to find similar example images, which we call isolated samples. Here, we introduce a novel sketch-based medical image retrieval (SBMIR) system that enables users to find images of interest without example images. The key idea lies in feature decomposition of medical images, whereby the entire feature of a medical image can be decomposed into and reconstructed from normal and abnormal features. By extending this idea, our SBMIR system provides an easy-to-use two-step graphical user interface: users first select a template image to specify a normal feature and then draw a semantic sketch of the disease on the template image to represent an abnormal feature. Subsequently, it integrates the two kinds of input to construct a query vector and retrieves reference images with the closest reference vectors. Using two datasets, ten healthcare professionals with various clinical backgrounds participated in the user test for evaluation. As a result, our SBMIR system enabled users to overcome previous challenges, including image retrieval based on fine-grained image characteristics, image retrieval without example images, and image retrieval for isolated samples. Our SBMIR system achieves flexible medical image retrieval on demand, thereby expanding the utility of medical image databases.

Generating Diverse High-Fidelity Images With Vq-Vae-2

Abstract: We explore the use of Vector Quantized Variational AutoEncoder (VQ-VAE) models for large scale image generation. To this end, we scale and enhance the autoregressive priors used in VQ-VAE to generate synthetic samples of much higher coherence and fidelity than possible before. We use simple feed-forward encoder and decoder networks, making our model an attractive candidate for applications where the encoding and/or decoding speed is critical. Additionally, VQ-VAE requires sampling an autoregressive model only in the compressed latent space, which is an order of magnitude faster than sampling in the pixel space, especially for large images. We demonstrate that a multi-scale hierarchical organization of VQ-VAE, augmented with powerful priors over the latent codes, is able to generate samples with quality that rivals that of state of the art Generative Adversarial Networks on multifaceted datasets such as ImageNet, while not suffering from GAN's known shortcomings such as mode collapse and lack of diversity.


In this paper, we propose a joint rendering mechanism that takes the MPI strategy for coarse sampling proposals and the MLP&volume-based rendering #REFR for fine sampling and rendering.

However, the rendering quality heavily relies on the number of image planes.

It needs high-resolution 4D volumes to store the 4channel (RGB and volume density) image planes that cost a large amount of GPU memory in both training and prediction. Li Zhang is the corresponding author. (e-mail:lizhangfd@fudan.edu.cn).

Yurui Chen, Chun Gu and Li Zhang are with the School of Data Science, Fudan University. Feihu Zhang is with University of Oxford. #OTHEREFR .

It produces many blurs and floats in the occluded regions and at the object/depth edges.

(c-d) Our method employs a joint rendering mechanism that preserves more image details and predicts sharp depth edges.

Then, both the coarse point samples and the fine samples are combined according to their geometry distribution to realize a more accurate joint rendering.

More importantly, we introduce a depth teacher net that serves as the guidance for the joint rendering.

The monocular depth teacher predicts dense pseudo depth maps that assist the consistent 3D geometry learning between the MPI, the fine volume, and the joint rendering.

It also boosts the multi-view geometry consistency between the source view and the target novel views that helps handle the occlusions, reduce the blurs and floats, and render accurate depths.

In the experiments, we verify the effectiveness of our method on three challenging real-scene datasets (RealEstate10K #OTHEREFR , NYU #OTHEREFR and NeRF-LLFF #OTHEREFR ) for novel view synthesis or depth estimation.

Single-View Neural Radiance Fields With Depth Teacher

Abstract: Neural Radiance Fields (NeRF) have been proposed for photorealistic novel view rendering. However, it requires many different views of one scene for training. Moreover, it has poor generalizations to new scenes and requires retraining or finetuning on each scene. In this paper, we develop a new NeRF model for novel view synthesis using only a single image as input. We propose to combine the (coarse) planar rendering and the (fine) volume rendering to achieve higher rendering quality and better generalizations. We also design a depth teacher net that predicts dense pseudo depth maps to supervise the joint rendering mechanism and boost the learning of consistent 3D geometry. We evaluate our method on three challenging datasets. It outperforms state-of-the-art single-view NeRFs by achieving 5∼20% improvements in PSNR and reducing 20∼50% of the errors in the depth rendering. It also shows excellent generalization abilities to unseen data without the need to finetune on each new scene. Index Terms-single-view, novel view synthesis, multi-plane images, neural radiance field, volume rendering.

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.


Contrary to its unimodal counterpart, CEM requires an additional relationship modeling step #REFR that represents data elements from different modalities on a shared embedding space before similarities can be established.

posters, banners), this results in a limited set of queries as the target fields needed to gather actionable insights do not appear in the document in the first place.

A cross-modal entity matching (CEM) framework (defined in Section 2) can accelerate Alice's decision-making process in the previous example by augmenting the menu with supplementary data e.g., allergen information from the nutritional table in near-real-time.

Briefly, a CEM framework maps a data element e 1 from one modality (e.g.

printed menu) to a data element e 2 from another modality (e.g. relational table) if they represent the same real-world object(s).

What makes this task challenging is measuring similarity between data elements across modalities.

This is particularly challenging as VRDs can be multimodal and extremely diverse in terms of their layout, format and content (see Fig. 2 for an example).

Now let's consider a straightforward solution for the aforementioned task. Using a publicly available document understanding model (e.g.

LayoutLMv2 #OTHEREFR ), we can extract a structured record from the menu corresponding to each item.

We can subsequently identify matching tuples for a menu-item by performing fuzzyjoin between the nutritional table and its corresponding record.

Unfortunately, this method requires significant effort at every intermediate step.

Cross-Modal Entity Matching For Visually Rich Documents

Abstract: Visually rich documents (VRD) are physical/digital documents that utilize visual cues to augment their semantics. The information contained in these documents are often incomplete. Existing works that enable automated querying on VRDs do not take this aspect into account. Consequently, they support a limited set of queries. In this paper, we describe Juno-a multimodal framework that identifies a set of tuples from a relational database to augment an incomplete VRD with supplementary information. Our main contribution in this is an end-to-end-trainable neural network with bi-directional attention that executes this crossmodal entity matching task without any prior knowledge about the document type or the underlying database-schema. Exhaustive experiments on two heteroegeneous datasets show that Juno outperforms state-of-theart baselines by more than 6% in F1-score, while reducing the amount of human-effort in its workflow by more than 80%. To the best of our knowledge, ours is the first work that investigates the incompleteness of VRDs and proposes a robust framework to address it in a seamless way.

A Comprehensive Survey On Cross-Modal Retrieval

Abstract: In recent years, cross-modal retrieval has drawn much attention due to the rapid growth of multimodal data. It takes one type of data as the query to retrieve relevant data of another type. For example, a user can use a text to retrieve relevant pictures or videos. Since the query and its retrieved results can be of different modalities, how to measure the content similarity between different modalities of data remains a challenge. Various methods have been proposed to deal with such a problem. In this paper, we first review a number of representative methods for cross-modal retrieval and classify them into two main groups: 1) real-valued representation learning, and 2) binary representation learning. Real-valued representation learning methods aim to learn real-valued common representations for different modalities of data. To speed up the cross-modal retrieval, a number of binary representation learning methods are proposed to map different modalities of data into a common Hamming space. Then, we introduce several multimodal datasets in the community, and show the experimental results on two commonly used multimodal datasets. The comparison reveals the characteristic of different kinds of cross-modal retrieval methods, which is expected to benefit both practical applications and future research. Finally, we discuss open problems and future research directions.


In particular, machine learning techniques which were limited to Kitaev's codes in #REFR could be expanded to a much broader range of codes and circuits. Proof.

In the future, our scheme may be improved by designing a decoder that exploits the equivalence between different fault configurations like in the work of Pryadko #OTHEREFR .

Design of better decoders for quantum LDPC codes may also improve our scheme.

Conceptually, our work emphasizes and formally captures a circuit-centric approach of quantum error correction and fault tolerance.

This circuit-centric approach is central to Floquet codes [17] and has also lead to new ideas for surface codes #OTHEREFR .

Our formalism could be used to take this approach further by, for example, searching over the space of quantum codes and circuits.

Because S j belongs to the stabilizer group of the system right after level j , the faults corresponding to η j +0.5 (S j ) do not flip any of the checks of the outcome code.

Based on Lemma 4, this leads to the lemma for the sign j + 0.5.

The same result holds for j − 0.5 because a fault S j after the measurement of S j is equivalent to a fault S j before this measurement.

A Pauli error on the input state also keeps the check operator outcomes trivial.

Spacetime Codes Of Clifford Circuits

Abstract: We propose a scheme for detecting and correcting faults in any Clifford circuit. The scheme is based on the observation that the set of all possible outcome bit-strings of a Clifford circuit is a linear code, which we call the "outcome code". From the outcome code we construct a corresponding stabilizer code, the "spacetime code". Our construction extends the circuit-tocode construction of Bacon, Flammia, Harrow and Shi [2], revisited recently by Gottesman [16], to include intermediate and multi-qubit measurements. With this correspondence, we reduce the problem of correcting faults in a circuit to the well-studied problem of correcting errors in a stabilizer code. More precisely, a most likely error decoder for the spacetime code can be transformed into a most likely fault decoder for the circuit. We give efficient algorithms to construct the outcome and spacetime codes. We also identify conditions under which these codes are LDPC, and give an algorithm to generate low-weight checks, which can then be combined with efficient LDPC code decoders.

Optimizing Quantum Error Correction Codes With Reinforcement Learning

Abstract: Quantum error correction is widely thought to be the key to fault-tolerant quantum computation. However, determining the most suited encoding for unknown error channels or specific laboratory setups is highly challenging. Here, we present a reinforcement learning framework for optimizing and fault-tolerantly adapting quantum error correction codes. We consider a reinforcement learning agent tasked with modifying a family of surface code quantum memories until a desired logical error rate is reached. Using efficient simulations with about 70 data qubits with arbitrary connectivity, we demonstrate that such a reinforcement learning agent can determine near-optimal solutions, in terms of the number of data qubits, for various error models of interest. Moreover, we show that agents trained on one setting are able to successfully transfer their experience to different settings. This ability for transfer learning showcases the inherent strengths of reinforcement learning and the applicability of our approach for optimization from off-line simulations to on-line laboratory settings.


Optimization is always conducted with Adam #REFR to make the comparison more straightforward, although L-BFGS leads to fewer iterations.

To increase the ratio between the number of indicator voxels and network parameters, twice as many filters are employed in the three-dimensional case. For more details on both networks, c.f. Appendix B.

Nearest-Neighbor interpolation was used for upsampling, as it performed better than polynomial interpolation, as it preserves discontinuities needed to represent the voids.

Due to their convergence accelerating behavior #OTHEREFR , adaptive activation functions are used.

Specifically, PReLU #OTHEREFR was used for all layers except the last layer, where an adaptive Sigmoid was used to ensure an output in the range [0, 1].

The output was then rescaled to the range [ , 1] to ensure numerical stability.

Gradient clipping #OTHEREFR was found to be essential to ensure an early convergence.

Without it, a plateau was observed, followed by a strongly delayed breakthrough.

In addition, a learning rate scheduler with a polynomial decay (b • epoch + 1) a was employed with the values a = −0.5, b = 0.2.

The initial network weights are set with Glorot initialization #OTHEREFR .

On The Use Of Neural Networks For Full Waveform Inversion

Abstract: Neural networks have recently gained attention in solving inverse problems. One prominent methodology are Physics-Informed Neural Networks (PINNs) which can solve both forward and inverse problems. In the paper at hand, full waveform inversion is the considered inverse problem. The performance of PINNs is compared against classical adjoint optimization, focusing on three key aspects: the forward-solver, the neural network Ansatz for the inverse field, and the sensitivity computation for the gradient-based minimization. Starting from PINNs, each of these key aspects is adapted individually until the classical adjoint optimization emerges. It is shown that it is beneficial to use the neural network only for the discretization of the unknown material field, where the neural network produces reconstructions without oscillatory artifacts as typically encountered in classical full waveform inversion approaches. Due to this finding, a hybrid approach is proposed. It exploits both the efficient gradient computation with the continuous adjoint method as well as the neural network Ansatz for the unknown material field. This new hybrid approach outperforms Physics-Informed Neural Networks and the classical adjoint optimization in settings of two and three-dimensional examples.

Adam: A Method For Stochastic Optimization

Abstract: We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.


H.4. Experiment Results Of The Robustness Of The Harsanyinets

Table 6 shows that the classification accuracy of the adversarial examples of HarsanyiNet is slightly higher than that of ResNet-18 #REFR .

We conducted more experiments to analyze the robustness of the HarsanyiNet.

Specifically, we estimate the adversarial robustness of the classification performance and the adversarial robustness of the estimated Shapley values #OTHEREFR .

To estimate the adversarial robustness of the classification performance on HarsanyiNet, we conducted experiments on the CIFAR-10 dataset to evaluate the model robustness by examining its classification accuracy on the test set of adversarial examples.

To generate adversarial examples, we used the FGSM attack #OTHEREFR , a gradient-based method, with a maximum perturbation of 8/255 #OTHEREFR .

To estimate the adversarial robustness of the estimated Shapley values on HarsanyiNet, we assessed the robustness of its estimated Shapley values by computing the 2 norm of the difference in Shapley values between the adversarial and natural examples, i.e., ||φ nat − φ adv || 2 , where φ nat denotes the Shapley values of natural examples, and φ adv denotes the Shapley values of adversarial examples.

To calculate the Shapley values of the ResNet-18 model, we estimate Shapley values using the sampling algorithm with 1000 iterations.

Table 6 shows that the adversarial robustness of the estimated Shapley values on HarsanyiNet (estimated by the 2 norm of the difference of Shapley values, the lower the better) is slightly higher than that of Both experiments indicate that HarsanyiNet has a robustness close to, or even slightly higher than, that of the traditional model.

Harsanyinet: Computing Accurate Shapley Values In A Single Forward Propagation

Abstract: The Shapley value is widely regarded as a trustworthy attribution metric. However, when people use Shapley values to explain the attribution of input variables of a deep neural network (DNN), it usually requires a very high computational cost to approximate relatively accurate Shapley values in real-world applications. Therefore, we propose a novel network architecture, the HarsanyiNet, which makes inferences on the input sample and simultaneously computes the exact Shapley values of the input variables in a single forward propagation. The HarsanyiNet is designed on the theoretical foundation that the Shapley value can be reformulated as the redistribution of Harsanyi interactions encoded by the network.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


For example, in image classification applications #REFR , a partial area of the image is enough for interpretation of the classification results, not specific to each feature (pixel).

Therefore, interpretable machine learning has attracted more and more attention in the past years.

The usual way to obtain interpretability is to build a black-box model first and then explain its behavior using the post-hoc methods.

However, many researchers are skeptical about the reliability of this direction. Considering interpretability before modeling is encouraged #OTHEREFR .

Some works of interpretable machine learning tend to propose a general model or algorithm #OTHEREFR , but it is unknown whether that model meets the requirements in the actual application scenarios.

In application scenarios with high requirements for interpretation, it is unrealistic to seek a one-for-all model or interpretation technology, because different application scenarios need different interpretations.

For healthcare and criminal justice, the traditional scorecard is more acceptable for practitioners #OTHEREFR , but for credit evaluation, while the rule-based interpretation may be more user-friendly.

RRL (Rule Representation Learner) #OTHEREFR provides a rule representation and learning framework, which has performance advantages over traditional decision tree methods.

So we design a credit evaluation model based on RRL. The main contributions of the paper are as follows:

• Focusing on the credit evaluation, we designed an interpretable loan credit evaluation model based on RRL.

Our model can naturally extract accurate global and local explanations without using the post-hoc methods.

An Interpretable Loan Credit Evaluation Method Based On Rule Representation Learner

Abstract: The interpretability of model has become one of the obstacles to its wide application in the high-stake fields. The usual way to obtain interpretability is to build a black-box first and then explain it using the post-hoc methods. However, the explanations provided by the posthoc method are not always reliable. Instead, we design an intrinsically interpretable model based on RRL(Rule Representation Learner) for the Lending Club dataset. Specifically, features can be divided into three categories according to their characteristics of themselves and build three sub-networks respectively, each of which is similar to a neural network with a single hidden layer but can be equivalently converted into a set of rules. During the training, we learned tricks from previous research to effectively train binary weights. Finally, our model is compared with the tree-based model. The results show that our model is much better than the interpretable decision tree in performance and close to other blackbox, which is of practical significance to both financial institutions and borrowers. More importantly, our model is used to test the correctness of the explanations generated by the post-hoc method, the results show that the post-hoc method is not always reliable.

This Looks Like That: Deep Learning For Interpretable Image Recognition

Abstract: When we are faced with challenging image classification tasks, we often explain our reasoning by dissecting the image, and pointing out prototypical aspects of one class or another. The mounting evidence for each of the classes helps us make our final decision. In this work, we introduce a deep network architecture -- prototypical part network (ProtoPNet), that reasons in a similar way: the network dissects the image by finding prototypical parts, and combines evidence from the prototypes to make a final classification. The model thus reasons in a way that is qualitatively similar to the way ornithologists, physicians, and others would explain to people on how to solve challenging image classification tasks. The network uses only image-level labels for training without any annotations for parts of images. We demonstrate our method on the CUB-200-2011 dataset and the Stanford Cars dataset. Our experiments show that ProtoPNet can achieve comparable accuracy with its analogous non-interpretable counterpart, and when several ProtoPNets are combined into a larger network, it can achieve an accuracy that is on par with some of the best-performing deep models. Moreover, ProtoPNet provides a level of interpretability that is absent in other interpretable deep models.

classification results
image classification applications


To do this, we build upon the COMA RL algorithm of Foerster et al. #REFR to our IPP application for UAV teams.

Our goal is to learn a policy enabling cooperative UAV team behaviour for the adaptive monitoring task.

COMA is an actor-critic algorithm using a centralised critic network to evaluate each agent's behaviour, described by the policy π(• | ω t i ) and parameterised by an actor network, and to optimise the policy accordingly.

The critic evaluates the current policy π by estimating the agent's i state-action value

given the other agents' actions u t −i .

The critic network is trained on-policy via TD(λ) #OTHEREFR to estimate the discounted return G t , introduced in Sec.

Multi-Uav Adaptive Path Planning Using Deep Reinforcement Learning

Abstract: Efficient aerial data collection is important in many remote sensing applications. In large-scale monitoring scenarios, deploying a team of unmanned aerial vehicles (UAVs) offers improved spatial coverage and robustness against individual failures. However, a key challenge is cooperative path planning for the UAVs to efficiently achieve a joint mission goal. We propose a novel multi-agent informative path planning approach based on deep reinforcement learning for adaptive terrain monitoring scenarios using UAV teams. We introduce new network feature representations to effectively learn path planning in a 3D workspace. By leveraging a counterfactual baseline, our approach explicitly addresses credit assignment to learn cooperative behaviour. Our experimental evaluation shows improved planning performance, i.e. maps regions of interest more quickly, with respect to non-counterfactual variants. Results on synthetic and real-world data show that our approach has superior performance compared to state-of-theart non-learning-based methods, while being transferable to varying team sizes and communication constraints.

Counterfactual Multi-Agent Policy Gradients

Abstract: Cooperative multi-agent systems can be naturally used to model many real world problems, such as network packet routing and the coordination of autonomous vehicles. There is a great need for new reinforcement learning methods that can efficiently learn decentralised policies for such systems. To this end, we propose a new multi-agent actor-critic method called counterfactual multi-agent (COMA) policy gradients. COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies. In addition, to address the challenges of multi-agent credit assignment, it uses a counterfactual baseline that marginalises out a single agent's action, while keeping the other agents' actions fixed. COMA also uses a critic representation that allows the counterfactual baseline to be computed efficiently in a single forward pass. We evaluate COMA in the testbed of StarCraft unit micromanagement, using a decentralised variant with significant partial observability. COMA significantly improves average performance over other multi-agent actor-critic methods in this setting, and the best performing agents are competitive with state-of-the-art centralised controllers that get access to the full state.


Note that in "ED-ATT", Q is set to the output of the former decoder block while K, V are set to the output of the visual encoder, and such operation is called encoder-decoder attention #REFR . After stacking 6 blocks defined in Eq. (4), a language decoder is built.

For the first block, its input is the extracted visual feature set of the given image.

The output of the last block will be input into the language decoder.

For the decoder, it also stacks 6 identical blocks where each one contains two MHAs and an FFN.

Given the output of the former decoder block X D and the output of the visual encoder X E , the next decoder block calculates its output:

For the first block in the decoder, X D in Eq.

(4) is set to the word embedding set of the partially generated captions S = {s 1 , ..., s t } at the t-th time step.

For all the decoder blocks, the input X E is set to the same value, which is the output of the visual encoder.

The output of the last decoder block Z = {z 1 , ..., z t } is used to calculate the word distribution of the next word:

Transforming Visual Scene Graphs To Image Captions

Abstract: We propose to Transform Scene Graphs (TSG) into more descriptive captions. In TSG, we apply multi-head attention (MHA) to design the Graph Neural Network (GNN) for embedding scene graphs. After embedding, different graph embeddings contain diverse specific knowledge for generating the words with different part-of-speech, e.g., object/attribute embedding is good for generating nouns/adjectives. Motivated by this, we design a Mixtureof-Expert (MOE)-based decoder, where each expert is built on MHA, for discriminating the graph embeddings to generate different kinds of words. Since both the encoder and decoder are built based on the MHA, as a result, we construct a homogeneous encoderdecoder unlike the previous heterogeneous ones which usually apply Fully-Connectedbased GNN and LSTM-based decoder. The homogeneous architecture enables us to unify the training configuration of the whole model instead of specifying different training strategies for diverse sub-networks as in the heterogeneous pipeline, which releases the training difficulty. Extensive experiments on the MS-COCO captioning benchmark validate the effectiveness of our TSG. The code is in: https: //anonymous.4open.science/r/ACL23_TSG.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


Here, we perform retrieval by first translating a query sketch into the photo domain, and then finding its nearest neighbourhood feature match in the entire photo gallery using an Im-ageNet pre-trained VGG-16 #REFR feature extractor.

Fine-Grained SBIR: Fine-grained SBIR aims at retrieving a particular image given a query sketch #OTHEREFR .

Hence, we essentially convert the sketch-based retrieval task into an image-based retrieval task.

As seen in Table 4 , our method beats SOTA FG-SBIR schemes #OTHEREFR in terms of Acc.@q, which measures the percentage of sketches having a true-paired photo in the top-q retrieved list.

Precise Semantic Editing: Local semantic image editing is a popular application of GAN inversion #OTHEREFR .

Our method enables realistic semantic editing, where modifying one region of an input sketch, yields seamless local alterations in the generated images. Fig.

7 depicts one such sketch editing episode where the user gradually changes the heel length via sketch, to observe consistent local changes in the output photo domain.

Picture That Sketch: Photorealistic Image Generation From Abstract Sketches

Abstract: Edgemap Sketch Sketch Sketch Sketch Sketch Existing Methods Proposed Method Figure 1. (a) Set of photos generated by the proposed method. (b) While existing methods can generate faithful photos from perfectly pixel-aligned edgemaps, they fall short drastically in case of highly deformed and sparse free-hand sketches. In contrast, our autoregressive sketch-to-photo generation model produces highly photorealistic outputs from highly abstract sketches.

Very Deep Convolutional Networks For Large-Scale Image Recognition

Abstract: In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

photo domain
Im-ageNet pre-trained VGG-16


The architecture and performance comparison among TVG methods: a) 3D TVG methods [14, 16, 18, 34, 43, 60-62, 64, 67, 69, 71, 73] , b) 2D TVG methods #REFR 7] , and c) TVP-based 2D TVG (Ours), d) overall performance comparison.

In recent years, we have witnessed great progress on temporal video grounding (TVG) #OTHEREFR .

One key to this success comes from the fine-grained dense 3D visual features extracted by 3D convolutional neural networks (CNNs) (e.g., C3D #OTHEREFR and I3D #OTHEREFR ) since TVG tasks demand spatial-temporal context to locate the temporal interval of the moments described by the text query.

However, due to the high cost of the dense 3D feature extraction, most existing TVG models only take these 3D visual features ex- Figure 1 .

Ours is the most efficient (least inference time) and achieves competitive performance compared to 3D TVG methods.

In contrast to existing TVG methods, which utilize dense video features extracted by non-trainable offline 3D CNNs and textual features, our proposed framework utilizes a trainable 2D CNN as the vision encoder to extract features from sparsely-sampled video frames with a universal set of frame-aware visual prompts and adds text prompts in textual feature space for end-to-end regression-based modeling.

tracted by offline 3D CNNs as inputs instead of co-training during TVG model training.

Although models using 3D visual features (that we call '3D methods or models') outperform these using the 2D features (that we call '2D methods or models'), a unique advantage of 2D methods is that extracting 2D visual features can significantly reduce the cost in TVG tasks #OTHEREFR .

An efficient and lightweight solution with reasonable performance is also demanded in computer vision, NLP, and video-language tasks #OTHEREFR .

Text-Visual Prompting For Efficient 2D Temporal Video Grounding

Abstract: In this paper, we study the problem of temporal video grounding (TVG), which aims to predict the starting/ending time points of moments described by a text sentence within a long untrimmed video. Benefiting from fine-grained 3D visual features, the TVG techniques have achieved remarkable progress in recent years. However, the high complexity of 3D convolutional neural networks (CNNs) makes extracting dense 3D visual features time-consuming, which calls for intensive memory and computing resources. Towards efficient TVG, we propose a novel text-visual prompting (TVP) framework, which incorporates optimized perturbation patterns (that we call 'prompts') into both visual inputs and textual features of a TVG model. In sharp contrast to 3D CNNs, we show that TVP allows us to effectively co-train vision encoder and language encoder in a 2D TVG model and improves the performance of crossmodal feature fusion using only low-complexity sparse 2D visual features. Further, we propose a Temporal-Distance IoU (TDIoU) loss for efficient learning of TVG. Experiments on two benchmark datasets, Charades-STA and Ac-tivityNet Captions datasets, empirically show that the proposed TVP significantly boosts the performance of 2D TVG (e.g., 9.79% improvement on Charades-STA and 30.77% improvement on ActivityNet Captions) and achieves 5× inference acceleration over TVG using 3D visual features. Codes are available at Open.Intel. "Person pours water into a glass." Text Query Video 2D CNN Textual Features

Localizing Moments In Video With Natural Language

Abstract: We consider retrieving a specific temporal segment, or moment, from a video given a natural language text description. Methods designed to retrieve whole video clips with natural language determine what occurs in a video but not when. To address this issue, we propose the Moment Context Network (MCN) which effectively localizes natural language queries in videos by integrating local and global video features over time. A key obstacle to training our MCN model is that current video datasets do not include pairs of localized video segments and referring expressions, or text descriptions which uniquely identify a corresponding moment. Therefore, we collect the Distinct Describable Moments (DiDeMo) dataset which consists of over 10,000 unedited, personal videos in diverse visual settings with pairs of localized video segments and referring expressions. We demonstrate that MCN outperforms several baseline methods and believe that our initial results together with the release of DiDeMo will inspire further research on localizing video moments with natural language.


While these techniques allowed for generating images with creative and novel artistic styles #REFR , none provided significant affordances to end-users for controlling what was generated outside of the training data scope.

A particularly relevant variation of this architecture is the Creative Adversarial Network (CAN) from Elgammal et al.

#OTHEREFR , designed to generate images with novel artistic styles.

This artwork was subsequently featured in multiple exhibitions #OTHEREFR where human observers could not distinguish the CAN-generated art from human-authored artwork. Outside of GANs, Gatys et al.

#OTHEREFR introduced a method to apply learned artistic styles to random images (a technique now known as Neural Style Transfer #OTHEREFR ).

Additionally, work originally designed to make convolutional neural networks more explainable, now referred to as Deep Dreams, became popular for generating art #OTHEREFR due to its ability to generate psychedelic versions of images #OTHEREFR .

#OTHEREFR addressed this issue by showing that a generative model could produce novel images from natural text when conditioned on image captions.

As text-to-image models rely on language modeling techniques, recent advances in the scaling of large language models #OTHEREFR , and Imagen #OTHEREFR .

Fueled by the latest advances in text-to-image models, current image generation applications are becoming mainstream.

With this broader adoption comes the question of how these new models' capabilities impact art practices, which we examine in this paper.

The Prompt Artists

Abstract: Fig. 1. Prompt artists develop descriptive text-based prompts that are rendered by text-to-image models. Highly skilled prompt artists will develop 1) distinct visual concepts and styles (S1), 2) prompts that can also serve as titles of the art piece ("prompts as art", A1), and 3) "prompt templates" (A2), which encapsulate specific visual concepts to be customized by others. Artists strive to discover unique natural language that produces unique visual outputs (G1), and/or model "glitches" (G2) that can be elevated to artistic styles in their own right. Finally, some prompt artists validate the novelty of their work by conducting an image search for similar images (C1). This paper examines the art practices, artwork, and motivations of prolific users of the latest generation of text-to-image models. Through interviews, observations, and a user survey, we present a sampling of the artistic styles and describe the developed community of practice around generative AI. We find that: 1) the text prompt and the resulting image can be considered collectively as an art piece (prompts as art), and 2) prompt templates (prompts with "slots" for others to fill in with their own words) are developed to create generative art styles. We discover that the value placed by this community on unique outputs leads to artists seeking specialized vocabulary to produce distinctive art pieces (e.g., by reading architectural blogs to find phrases to describe images). We also find that some artists use "glitches" in the model that can be turned into artistic styles of their own right. From these findings, we outline specific implications for design regarding future prompting and image editing options.

Can: Creative Adversarial Networks, Generating"Art"By Learning About Styles And Deviating From Style Norms

Abstract: We propose a new system for generating art. The system generates art by looking at art and learning about style; and becomes creative by increasing the arousal potential of the generated art by deviating from the learned styles. We build over Generative Adversarial Networks (GAN), which have shown the ability to learn to generate novel images simulating a given distribution. We argue that such networks are limited in their ability to generate creative products in their original design. We propose modifications to its objective to make it capable of generating creative art by maximizing deviation from established styles and minimizing deviation from art distribution. We conducted experiments to compare the response of human subjects to the generated art with their response to art created by artists. The results show that human subjects could not distinguish art generated by the proposed system from art generated by contemporary artists and shown in top art fairs. Human subjects even rated the generated images higher on various scales.


We use the implementation of Residual Dense Block from #REFR , where "ResidualDenseBlock-32" refers to a Residual Dense Block with minimum channel 32.

In this section, we describe the details of our architecture and training settings. Network Architecture.

Table 5 and Table 6 respectively show details of our encoder and decoder.

In Table 5 , we describe the architecture of our UNet-based encoder #OTHEREFR .

"PixelUnshuffle 4x" stands for the rearrangement of elements #OTHEREFR which downsamples the HR image by a factor of 4.

"3×3, 64, LeakyReLU" denotes a 2d convolution operation of kernel size 3, output channel 64, followed by a LeakyReLU operation.

We save the produced quantization tables and output coefficients into a single JPEG file using TorchJPEG #OTHEREFR .

Building blocks are shown in brackets, with the number of blocks stacked.

Downsampling is performed at the beginning of the downsampling block using max pooling of stride 2.

After the first convolution in the upsampling block, we upsample the decoder feature with Pixel Shuffling Operation. Then, skip connections with the encoder features are conducted.

In Table 6 , we show the details of our efficient decoder, which is developed based on EDSR #OTHEREFR .

Hyperthumbnail: Real-Time 6K Image Rescaling With Rate-Distortion Optimization

Abstract: Contemporary image rescaling aims at embedding a high-resolution (HR) image into a low-resolution (LR) thumbnail image that contains embedded information for HR image reconstruction. Unlike traditional image superresolution, this enables high-fidelity HR image restoration faithful to the original one, given the embedded information in the LR thumbnail. However, state-of-the-art image rescaling methods do not optimize the LR image file size for efficient sharing and fall short of real-time performance for ultra-high-resolution (e.g., 6K) image reconstruction. To address these two challenges, we propose a novel framework (HyperThumbnail) for real-time 6K rate-distortionaware image rescaling. Our framework first embeds an HR image into a JPEG LR thumbnail by an encoder with our proposed quantization prediction module, which minimizes the file size of the embedding LR JPEG thumbnail while maximizing HR reconstruction quality. Then, an efficient frequency-aware decoder reconstructs a high-fidelity HR image from the LR one in real time. Extensive experiments demonstrate that our framework outperforms previous image rescaling baselines in rate-distortion performance and can perform 6K image reconstruction in real time.

Esrgan: Enhanced Super-Resolution Generative Adversarial Networks

Abstract: The Super-Resolution Generative Adversarial Network (SRGAN) is a seminal work that is capable of generating realistic textures during single image super-resolution. However, the hallucinated details are often accompanied with unpleasant artifacts. To further enhance the visual quality, we thoroughly study three key components of SRGAN - network architecture, adversarial loss and perceptual loss, and improve each of them to derive an Enhanced SRGAN (ESRGAN). In particular, we introduce the Residual-in-Residual Dense Block (RRDB) without batch normalization as the basic network building unit. Moreover, we borrow the idea from relativistic GAN to let the discriminator predict relative realness instead of the absolute value. Finally, we improve the perceptual loss by using the features before activation, which could provide stronger supervision for brightness consistency and texture recovery. Benefiting from these improvements, the proposed ESRGAN achieves consistently better visual quality with more realistic and natural textures than SRGAN and won the first place in the PIRM2018-SR Challenge. The code is available at this https URL .