The white-box robustness of VGG-16 #REFR models trained using AT, ALP, TRADES, and CAT are reported in Tab. A1.

The setting for VGG-16 is the same as ResNet-18 models, i.e., α = 1.0/20 and β = 1.0/20.

The improvement for CAT with VGG-16 models is as consistent with ResNet-18 models.

CAT can boost model's robustness under AutoAttack with 2.0 points.

Cat: Collaborative Adversarial Training

Abstract: Adversarial training can improve the robustness of neural networks. Previous methods focus on a single adversarial training strategy and do not consider the model property trained by different strategies. By revisiting the previous methods, we find different adversarial training methods have distinct robustness for sample instances. For example, a sample instance can be correctly classified by a model trained using standard adversarial training (AT) but not by a model trained using TRADES, and vice versa. Based on this observation, we propose a collaborative adversarial training framework to improve the robustness of neural networks. Specifically, we use different adversarial training methods to train robust models and let models interact with their knowledge during the training process. Collaborative Adversarial Training (CAT) can improve both robustness and accuracy. Extensive experiments on various networks and datasets validate the effectiveness of our method. CAT achieves state-of-the-art adversarial robustness without using any additional data on CIFAR-10 under the Auto-Attack benchmark. Code is available at https://github.com/liuxingbin/CAT.

Very Deep Convolutional Networks For Large-Scale Image Recognition

Abstract: In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.


In #REFR , the authors jointly minimize the total transmit power and the circuit power consumption for MRT and ZF precoders.

The reason is that #OTHEREFR , #OTHEREFR consider the minimization of (2) from the precoding problem definition, not assuming any particular precoding scheme.

When other BS components are included in the power consumption model, the optimal number of BS antennas to be utilized decreases.

Similarly to #OTHEREFR , several studies have investigated the whole BS power consumption for energy-efficient mMIMO communications.

A power consumption model for a mMIMO BS is developed in #OTHEREFR , where the total consumed power is the sum of the contributions from the PAs and from the BS circuits, plus a static consumption term.

The circuit power consumption considers the radio frequency transceivers, the baseband processing, the backhauling, the cooling, and the power supply, and typically scales linearly with M #OTHEREFR .

Differently from #OTHEREFR , they consider different traffic situations and they show that the number of activated antennas increases as a function of the traffic.

Energy-Saving Precoder Design For Narrowband And Wideband Massive Mimo

Abstract: In this work, we study massive multiple-input multiple-output (MIMO) precoders optimizing power consumption while achieving the users' rate requirements. We first characterize analytically the solutions for narrowband and wideband systems minimizing the power amplifiers (PAs) consumption in low system load, where the per-antenna power constraints are not binding. After, we focus on the asymptotic wideband regime. The power consumed by the whole base station (BS) and the high-load scenario are then also investigated. We obtain simple solutions, and the optimal strategy in the asymptotic case reduces to finding the optimal number of active antennas, relying on known precoders among the active antennas. Numerical results show that large savings in power consumption are achievable in the narrowband system by employing antenna selection, while all antennas need to be activated in the wideband system when considering only the PAs consumption, and this implies lower savings. When considering the overall BS power consumption and a large number of subcarriers, we show that significant savings are achievable in the low-load regime by using a subset of the BS antennas. While optimization based on transmit power pushes to activate all antennas, optimization based on consumed power activates a number of antennas proportional to the load.

Joint Transmit And Circuit Power Minimization In Massive Mimo With Downlink Sinr Constraints: When To Turn On Massive Mimo?

Abstract: In this paper, we consider the downlink of a multi-cell multiple-input multiple-output system and find the jointly optimal number of base station (BS) antennas and transmission powers that minimize the power consumption while satisfying each user’s effective signal-to-interference-and-noise-ratio constraint and the BSs’ power constraints. Different from prior work, we consider a power consumption model that takes both transmitted and hardware-consumed power into account. We formulate the joint optimization problem for both single-cell and multi-cell systems. The closed-form expressions for the optimal number of BS antennas and transmission powers are derived for the single-cell case. The analysis for the multi-cell case reveals that increasing the number of BS antennas in any cell always improves the performance of the overall system in terms of both feasibility and total radiated power. A key contribution of this paper is to show that the joint optimization problem can be relaxed as a geometric programming problem that can be solved efficiently. The solution can be utilized in practice to turn on and off antennas depending on the traffic load variations. The substantial power savings are demonstrated by simulation.


AudioSet has been documented by prior work #REFR to be much noisier than the videos in the other two datasets, which explains the lower performance across all approaches.

Similar to the reported performance on the MUSIC dataset, we also observe that VAST is comparable to the strongly supervised AVSGS approach on the SOLOS dataset.

One possible reason behind the lower performance of AVSGS on videos with single instrument is that it is less critical to reason about context for videos with a single sounding object.

Our better performance as compared to Co-Separation suggests that learning to perform audio separation at the coarse video region level under an MIL formulation can serve as an effective replacement for training object detectors.

These results also indicate the great promise of latent captions in replacing ground-truth object annotations for supervision. Audio-visual source separation on AudioSet.

Finally, we report the results of our evaluation on the AudioSet dataset in Table 3 .

Unlike the SOP model, we observe that our training approach is more robust to noise in the dataset where the sounding object may not always be visible.

Language-Guided Audio-Visual Source Separation Via Trimodal Consistency

Abstract: We propose a self-supervised approach for learning to perform audio source separation in videos based on natural language queries, using only unlabeled video and audio pairs as training data. A key challenge in this task is learning to associate the linguistic description of a soundemitting object to its visual features and the corresponding components of the audio waveform, all without access to annotations during training. To overcome this challenge, we adapt off-the-shelf vision-language foundation models to provide pseudo-target supervision via two novel loss functions and encourage a stronger alignment between the audio, visual and natural language modalities. During inference, our approach can separate sounds given text, video and audio input, or given text and audio input alone. We demonstrate the effectiveness of our self-supervised approach on three audiovisual separation datasets, including MUSIC, SOLOS and AudioSet, where we outperform state-of-the-art strongly supervised approaches despite not using object detectors or text labels during training. Our project page including publicly available code can be found at https://cs-people.bu.edu/rxtan/projects/VAST.

Co-Separating Sounds Of Visual Objects

Abstract: Learning how objects sound from video is challenging, since they often heavily overlap in a single audio channel. Current methods for visually-guided audio source separation sidestep the issue by training with artificially mixed video clips, but this puts unwieldy restrictions on training data collection and may even prevent learning the properties of "true" mixed sounds. We introduce a co-separation training paradigm that permits learning object-level sounds from unlabeled multi-source videos. Our novel training objective requires that the deep neural network's separated audio for similar-looking objects be consistently identifiable, while simultaneously reproducing accurate video-level audio tracks for each source training pair. Our approach disentangles sounds in realistic test videos, even in cases where an object was not observed individually during training. We obtain state-of-the-art results on visually-guided audio source separation and audio denoising for the MUSIC, AudioSet, and AV-Bench datasets.


Function f is realized through a CNN with an ImageNet pretrained ResNet-34 backbone #REFR .

We provide details about the initialization set Θ, function f , training data, and loss function to train f .

Initialization set Θ comes from the Cartesian product of a set of robot base positions in R 3 and a set of 10 arm configurations.

We use 704 base positions (sampled in a uniform 1 m × 1.5 m 2D grid of base positions at a 10 cm resolution at 4 different heights) and 10 arm configurations, resulting in Θ having 7040 elements.

The 10 arm configurations are selected in a data-driven manner, we sample 20 random configurations that satisfy the joint limits, and then select the 10 which when used with SeqIK give us the most successes across the training set.

We add 2 fully connected layers on the conv5 output to produce a 7040 dimensional representation.

This is reshaped into an 80-dimensional spatial output of size 11 × 16.

This is processed through another 3 convolutional layers to produce a (10 • 4) × 11 × 16 logits tensor containing 11 × 16 spatial output for each of the 10 arm configurations at each of the 4 heights.

Training labels are generated by decoding each candidate θ 0 into motion plans using SeqIK, and testing them for endeffector pose deviation, self-collision, collision with the static environment, and collision with the articulating object in our simulator from Section III.

Note that while testing the decoded motion plans, we interpolate between consecutive states to simulate how the plan will be executed in practice.

Predicting Motion Plans For Articulating Everyday Objects

Abstract: Mobile manipulation tasks such as opening a door, pulling open a drawer, or lifting a toilet lid require constrained motion of the end-effector under environmental and task constraints. This, coupled with partial information in novel environments, makes it challenging to employ classical motion planning approaches at test time. Our key insight is to cast it as a learning problem to leverage past experience of solving similar planning problems to directly predict motion plans for mobile manipulation tasks in novel situations at test time. To enable this, we develop a simulator, ArtObjSim, that simulates articulated objects placed in real scenes. We then introduce SeqIK+θ0, a fast and flexible representation for motion plans. Finally, we learn models that use SeqIK+θ0 to quickly predict motion plans for articulating novel objects at test time. Experimental evaluation shows improved speed and accuracy at generating motion plans than pure search-based methods and pure learning methods.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


We can show however that in our setting, the well-known "barren plateau" problem #REFR does not arise.

From the definition in Section 4.4, each of the neural networks has O(T 2 ) parameters.

As a consequence of results due to Caro and Datta #OTHEREFR , the pseudo-dimension of our parameterized model is polynomial in T .

Therefore poly(T ) episodes suffice to ensure that empirical risk minimization over our sample converges with high probability to the true optimal expressible model over the whole distribution of futures.

• Trainability: Finally we consider whether the task of optimizing the parameters for our model can be performed efficiently.

The associated optimization problem is non-convex and thus training convergence cannot be guaranteed.

Each Quantum Compound Neural Network that we use is on O(T ) qubits and has O(T ) depth.

Furthermore the loss function we measure can be constructed as a function of measurements in the computational basis (corresponding to a measuring the vector of observables Z on each qubit i).

We initialize the parameters of the model as normal random variables with variance O(1/T ).

Theorem 2 ensures that the gradients decay only polynomially with the time horizon T .

Quantum Deep Hedging

Abstract: Quantum machine learning has the potential for a transformative impact across industry sectors and in particular in finance. In our work we look at the problem of hedging where deep reinforcement learning offers a powerful framework for real markets. We develop quantum reinforcement learning methods based on policy-search and distributional actor-critic algorithms that use quantum neural network architectures with orthogonal and compound layers for the policy and value functions. We prove that the quantum neural networks we use are trainable, and we perform extensive simulations that show that quantum models can reduce the number of trainable parameters while achieving comparable performance and that the distributional approach obtains better performance than other standard approaches, both classical and quantum. We successfully implement the proposed models on a trapped-ion quantum processor, utilizing circuits with up to 16 qubits, and observe performance that agrees well with noiseless simulation. Our quantum techniques are general and can be applied to other reinforcement learning problems beyond hedging.

Barren Plateaus In Quantum Neural Network Training Landscapes

Abstract: Many experimental proposals for noisy intermediate scale quantum devices involve training a parameterized quantum circuit with a classical optimization loop. Such hybrid quantum-classical algorithms are popular for applications in quantum simulation, optimization, and machine learning. Due to its simplicity and hardware efficiency, random circuits are often proposed as initial guesses for exploring the space of quantum states. We show that the exponential dimension of Hilbert space and the gradient estimation complexity make this choice unsuitable for hybrid quantum-classical algorithms run on more than a few qubits. Specifically, we show that for a wide class of reasonable parameterized quantum circuits, the probability that the gradient along any reasonable direction is non-zero to some fixed precision is exponentially small as a function of the number of qubits. We argue that this is related to the 2-design characteristic of random circuits, and that solutions to this problem must be studied.


In our evaluation, we show the integration of our implemented prototype with Caterpillar #REFR , a state-ofthe-art process execution engine, to demonstrate how our approach can complement a business process management system to secure its data flow.

To overcome these limitations, we here propose a Multi-Authority Approach to Transaction Systems for Interoperating Applications (MARTSIA).

In MARTSIA, encrypted data is persisted in decentralized storage, which is connected to a public permissionless blockchain system supporting process execution.

Data owners define access policies to regulate which users are able to view specific parts of the information. No central authority can generate decryption keys alone.

The encryption and decryption of messages are left to the individual nodes.

To attain the desired characteristics, this approach employs hybrid encryption in combination with the Cyphertext-Policy variant of Multi-Authority Attribute-Based Encryption (henceforth, MA-ABE for the sake of conciseness), smart contracts, and InterPlanetary File System (IPFS).

2 presents a running example, to which we will refer throughout the paper, and illustrates the problem we tackle.

Section 3 outlines the fundamental notions that our solution is based upon. In Sect. 4, we describe our approach in detail. In Sect.

5, we present our proof-of-concept implementation and the results of the experiments we conducted therewith. Section 6 reviews related work before Sect. 7 concludes the paper and outlines future works.

2 Example, problem illustration, and requirements Figure 1 shows a Business Process Model and Notation (BPMN) collaboration diagram illustrating a supply chain in the automotive area: the production of a special car for a person with paraplegia.

Martsia: Enabling Data Confidentiality For Blockchain-Based Process Execution

Abstract: Multi-party business processes rely on the collaboration of various players in a decentralized setting. Blockchain technology can facilitate the automation of these processes, even in cases where trust among participants is limited. Transactions are stored in a ledger, a replica of which is retained by every node of the blockchain network. The operations saved thereby are thus publicly accessible. While this enhances transparency, reliability, and persistence, it hinders the utilization of public blockchains for process automation as it violates typical confidentiality requirements in corporate settings. In this paper, we propose MARTSIA: A Multi-Authority Approach to Transaction Systems for Interoperating Applications. MARTSIA enables precise control over process data at the level of message parts. Based on Multi-Authority Attribute-Based Encryption (MA-ABE), MARTSIA realizes a number of desirable properties, including confidentiality, transparency, and auditability. We implemented our approach in proof-of-concept prototypes, with which we conduct a case study in the area of supply chain management. Also, we show the integration of MARTSIA with a state-of-the-art blockchain-based process execution engine to secure the data flow.

Caterpillar: A Business Process Execution Engine On The Ethereum Blockchain

Abstract: Blockchain platforms, such as Ethereum, allow a set of actors to maintain a ledger of transactions without relying on a central authority and to deploy scripts, called smart contracts, that are executed whenever certain transactions occur. These features can be used as basic building blocks for executing collaborative business processes between mutually untrusting parties. However, implementing business processes using the low-level primitives provided by blockchain platforms is cumbersome and error-prone. In contrast, established business process management systems, such as those based on the standard Business Process Model and Notation (BPMN), provide convenient abstractions for rapid development of process-oriented applications. This article demonstrates how to combine the advantages of a business process management system with those of a blockchain platform. The article introduces a blockchain-based BPMN execution engine, namely Caterpillar. Like any BPMN execution engine, Caterpillar supports the creation of instances of a process model and allows users to monitor the state of process instances and to execute tasks thereof. The specificity of Caterpillar is that the state of each process instance is maintained on the (Ethereum) blockchain and the workflow routing is performed by smart contracts generated by a BPMN-to-Solidity compiler. The Caterpillar compiler supports a large array of BPMN constructs, including subprocesses, multi-instances activities and event handlers. The paper describes the architecture of Caterpillar, and the interfaces it provides to support the monitoring of process instances, the allocation and execution of work items, and the execution of service tasks.


In principle, CAM roughly indicates the contribution of local image regions to classification confidence #REFR .

The classification prediction scoreŷ ∈ R C can be calculated asŷ = σ(GAP(f (x))), where σ denotes the sigmoid activation function.

To train the network, the classification loss is defined as the binary cross-entropy between prediction y and ground-truthŷ, i.e.,

After training f , the CAM of an image x related to the c-th class, denoted as M c ∈ R C×H×W , c ∈ {1, 2, .., C}, can be directly calculated as, Shortage of CAM.

The discriminating features always play a dominant role in the classification, as can be seen from Figure 2 (a), where the confidence of the bird head regions (highlighted by red boxes) is much ahead of other regions.

The phenomenon results in CAM mainly confined to the small salient regions.

Most methods #OTHEREFR modify the structure of the CAM generating network or introduce well-designed modules, which alter the contribution of image regions to classification confidence, leading to the expansion of the CAM.

However, Figure 2 (a) suggests that foreground points (in red) generally lie above background points (in blue), indicating that CAM still treats non-discriminating features differently from the background. Here, we pose the question Is CAM fully exploited?

Exploit Cam By Itself: Complementary Learning System For Weakly Supervised Semantic Segmentation

Abstract: Weakly Supervised Semantic Segmentation (WSSS) with image-level labels has long been suffering from fragmentary object regions led by Class Activation Map (CAM), which is incapable of generating fine-grained masks for semantic segmentation. To guide CAM to find more non-discriminating object patterns, this paper turns to an interesting working mechanism in agent learning named Complementary Learning System (CLS). CLS holds that the neocortex builds a sensation of general knowledge, while the hippocampus specially learns specific details, completing the learned patterns. Motivated by this simple but effective learning pattern, we propose a General-Specific Learning Mechanism (GSLM) to explicitly drive a coarse-grained CAM to a fine-grained pseudo mask. Specifically, GSLM develops a General Learning Module (GLM) and a Specific Learning Module (SLM). The GLM is trained with image-level supervision to extract coarse and general localization representations from CAM. Based on the general knowledge in the GLM, the SLM progressively exploits the specific spatial knowledge from the localization representations, expanding the CAM in an explicit way. To this end, we propose the Seed Reactivation to help SLM reactivate nondiscriminating regions by setting a boundary for activation values, which successively identifies more regions of CAM. Without extra refinement processes, our method is able to achieve breakthrough improvements for CAM of over 20.0% mIoU on PASCAL VOC 2012 and 10.0% mIoU on MS COCO 2014 datasets, representing a new state-of-the-art among existing WSSS methods.

Weakly Supervised Learning Of Instance Segmentation With Inter-Pixel Relations

Abstract: This paper presents a novel approach for learning instance segmentation with image-level class labels as supervision. Our approach generates pseudo instance segmentation labels of training images, which are used to train a fully supervised model. For generating the pseudo labels, we first identify confident seed areas of object classes from attention maps of an image classification model, and propagate them to discover the entire instance areas with accurate boundaries. To this end, we propose IRNet, which estimates rough areas of individual instances and detects boundaries between different object classes. It thus enables to assign instance labels to the seeds and to propagate them within the boundaries so that the entire areas of instances can be estimated accurately. Furthermore, IRNet is trained with inter-pixel relations on the attention maps, thus no extra supervision is required. Our method with IRNet achieves an outstanding performance on the PASCAL VOC 2012 dataset, surpassing not only previous state-of-the-art trained with the same level of supervision, but also some of previous models relying on stronger supervision.


Style Transfer And Adaptive Instance Normalization

StyleGAN #REFR draws on the idea of AdaIN style transfer, removes the traditional input, and proposes a stylebased generator, which takes a learnable constant as input.

This research found that replacing BN with instance normalization (IN) can significantly improve the convergence speed #OTHEREFR . Dumoulin et al.

#OTHEREFR found that images with different styles can be generated by using different scales and displacements during IN operation, also known as conditional instance normalization (CIN). Huang et al.

#OTHEREFR proposed that the artistic style of the image is the cross-spatial statistical information of each feature channel of the feature graph, such as mean and variance.

Style transfer can be realized by transferring each channel's mean and standard deviation.

This operation is named adaptive instance normalization (AdaIN), which ensures transferring any styles to the feature maps.

It mainly controls the visual features represented by each level by modifying the input of each level separately without affecting other levels.

And StyleGAN2 #OTHEREFR adjusted the use of AdaIN to avoid water droplet artifacts effectively.

With the advantage of GAN network, some scholars try to generate high-quality facial age images.

LATS #OTHEREFR presented a method for synthesizing lifespan age transformations.

RAGAN #OTHEREFR introduces a personalized self-guidance scheme that enables transforming the input face across various target age groups while preserving identity. Our designed network needs two modal inputs.

Daa: A Delta Age Adain Operation For Age Estimation Via Binary Code Transformer

Abstract: Naked eye recognition of age is usually based on comparison with the age of others. However, this idea is ignored by computer tasks because it is difficult to obtain representative contrast images of each age. Inspired by the transfer learning, we designed the Delta Age AdaIN (DAA) operation to obtain the feature difference with each age, which obtains the style map of each age through the learned values representing the mean and standard deviation. We let the input of transfer learning as the binary code of age natural number to obtain continuous age feature information. The learned two groups of values in Binary code mapping are corresponding to the mean and standard deviation of the comparison ages. In summary, our method consists of four parts: FaceEncoder, DAA operation, Binary code mapping, and AgeDecoder modules. After getting the delta age via AgeDecoder, we take the average value of all comparison ages and delta ages as the predicted age. Compared with state-of-the-art methods, our method achieves better performance with fewer parameters on multiple facial age datasets.

A Style-Based Generator Architecture For Generative Adversarial Networks

Abstract: We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.

AdaIN style transfer
stylebased generator


Much work has been done on the optimal polynomial approximation of circular arcs (see, e.g., #REFR and the references therein).

One of the most important issues of computer-aided geometric design (CAGD) is the approximation of curves and surfaces by simple polynomial-based objects.

It is well known that even fundamental geometric objects such as circular arcs or spherical patches do not possess exact polynomial representations.

It is thus an interesting and practical issue to find their best polynomial approximants.

However, much less is known about the optimal approximation of spherical patches.

The main two classes of the polynomial (or spline) surfaces used in CAGD are triangular patches and tensor product patches.

The optimal approximation of equilateral spherical triangles by triangular Bézier patches was studied recently in #OTHEREFR .

Approximation of rational tensor-product biquadratic Bézier surfaces (which also includes the sphere) by polynomial tensor product patches was proposed in #OTHEREFR .

An exact sphere representation by rational S-patches was considered recently in #OTHEREFR .

Optimal Approximation Of Spherical Squares By Tensor Product Quadratic Bézier Patches

Abstract: In [1], the author considered the problem of the optimal approximation of symmetric surfaces by biquadratic Bézier patches. Unfortunately, the results therein are incorrect, which is shown in this paper by considering the optimal approximation of spherical squares. A detailed analysis and a numerical algorithm are given, providing the best approximant according to the (simplified) radial error, which differs from the one obtained in [1]. The sphere is then approximated by the continuous spline of two and six tensor product quadratic Bézier patches. It is further shown that the G 1 smooth spline of six patches approximating the sphere exists, but it is not a good approximation. The problem of an approximation of spherical rectangles is also addressed and numerical examples indicate that several optimal approximants might exist in some cases, making the problem extremely difficult to handle. Finally, numerical examples are provided that confirm theoretical results.

Optimal Parametric Interpolants Of Circular Arcs.

Abstract: The aim of this paper is a construction of quartic parametric polynomial interpolants of a circular arc, where two boundary points of a circular arc are interpolated. For every unit circular arc of inner angle not greater than $\pi$ we find the best interpolant, where the optimality is measured by the simplified radial error.


However, computing a single queryitem score using cross-encoders requires a forward pass of the model which can be computationally expensive as cross-encoders are typically parameterized using deep neural models such as transformers #REFR .

k-nearest neighbor (k-NN) search is core subroutine of a variety of tasks in NLP such as entity linking #OTHEREFR , passage retrieval for QA #OTHEREFR , and more generally, in retrieval-augmented machine learning models (Guu et al., 2020; Izacard et al., 2022) .

Neural models are widely used for parameterizing the similarity function in such applications due to their widespread success #OTHEREFR .

The efficiency of k-NN search depends largely on the architecture and parameterization of the similarity function.

In this work, we focus on k-NN search for cross-encoders, a popular class of neural similarity functions.

Cross-encoder models directly output a scalar similarity score after jointly encoding a given query-item pair, and generalize better to new domains and unseen data as compared alternative choices such as dual-encoders which independently embed query and item and compute similarity using simple functions such as dot-product of these embeddings.

For this reason, k-NN search with cross-encoders typically involves retrieving candidate items using additional models such as dual-encoders or BM25 #OTHEREFR , followed by re-ranking items using the cross-encoder #OTHEREFR .

Much of recent work focuses on improving recall of the first-stage retrieval using distillation-based techniques #OTHEREFR , or on designing new architectures that trade-off effectiveness for higher efficiency while attempting to capture fine-grained interactions between the query and the item #OTHEREFR .

Recent work #OTHEREFR explores matrix factorization based approaches for approximating cross-encoder scores. #OTHEREFR

(2022) proposed ANNCUR, a ma-trix factorization based method, to perform k-NN search with cross-encoders without relying on additional dual-encoder style models for candidate retrieval.

ANNCUR use an approximate-then-retrieve strategy where it approximates query-item scores using latent query and item embeddings, and performs retrieval using the approximate scores followed by optionally re-ranking them using exact CE scores.

Adaptive Selection Of Anchor Items For Cur-Based K-Nn Search With Cross-Encoders

Abstract: Cross-encoder models, which jointly encode and score a query-item pair, are typically prohibitively expensive for k-nearest neighbor search. Consequently, k-NN search is performed not with a cross-encoder, but with a heuristic retrieve (e.g., using BM25 or dualencoder) and re-rank approach. Recent work proposes ANNCUR (Yadav et al., 2022) which uses CUR matrix factorization to produce an embedding space for efficient vector-based search that directly approximates the crossencoder without the need for dual-encoders. ANNCUR defines this shared query-item embedding space by scoring the test query against anchor items which are sampled uniformly at random. While this minimizes average approximation error over all items, unsuitably high approximation error on top-k items remains and leads to poor recall of top-k (and especially top-1) items. Increasing the number of anchor items is a straightforward way of improving the approximation error and hence k-NN recall of ANNCUR but at the cost of increased inference latency. In this paper, we propose a new method for adaptively choosing anchor items that minimizes the approximation error for the practically important topk neighbors for a query with minimal computational overhead. Our proposed method incrementally selects a suitable set of anchor items for a given test query over several rounds, using anchors chosen in previous rounds to inform selection of more anchor items. Empirically, our method consistently improves k-NN recall as compared to both ANNCUR and the widely-used dual-encoder-based retrieve-andrerank approach.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

single queryitem score
deep neural models


We follow the same standard configuration and training process of both models as in #REFR .

The victim class and target class of Cityscapes dataset are set to be class 13 (car) and class 0 (road), respectively.

In this study, we use the classic Hello Kitty pattern as the backdoor trigger.

The trigger size is set to 15 × 15 pixels for the VOC dataset and 55 × 55 for the Cityscapes dataset. Segmentation models.

Two popular semantic segmentation architectures, namely PSPNet #OTHEREFR and DeepLabV3 #OTHEREFR , are adopted in this work.

In both architectures, ResNet-50 #OTHEREFR pre-trained on ImageNet #OTHEREFR is used as an embedding backbone.

Influencer Backdoor Attack On Semantic Segmentation

Abstract: When a small number of poisoned samples are injected into the training dataset of a deep neural network, the network can be induced to exhibit malicious behavior during inferences, which poses potential threats to real-world applications. While they have been intensively studied in classification, backdoor attacks on semantic segmentation have been largely overlooked. Unlike classification, semantic segmentation aims to classify every pixel within a given image. In this work, we explore backdoor attacks on segmentation models to misclassify all pixels of a victim class by injecting a specific trigger on non-victim pixels during inferences, which is dubbed Influencer Backdoor Attack (IBA). IBA is expected to maintain the classification accuracy of non-victim pixels and misleads classifications of all victim pixels in every single inference. Specifically, we consider two types of IBA scenarios, i.e., 1) Free-position IBA: the trigger can be positioned freely except for pixels of the victim class, and 2) Long-distance IBA: the trigger can only be positioned somewhere far from victim pixels, given the possible practical constraint. Based on the context aggregation ability of segmentation models, we propose techniques to improve IBA for the scenarios. Concretely, for free-position IBA, we propose a simple, yet effective Nearest Neighbor trigger injection strategy for poisoned sample creation. For long-distance IBA, we propose a novel Pixel Random Labeling strategy. Our extensive experiments reveal that current segmentation models do suffer from backdoor attacks, and verify that our proposed techniques can further increase attack performance.

Pyramid Scene Parsing Network

Abstract: Scene parsing is challenging for unrestricted open vocabulary and diverse scenes. In this paper, we exploit the capability of global context information by different-region-based context aggregation through our pyramid pooling module together with the proposed pyramid scene parsing network (PSPNet). Our global prior representation is effective to produce good quality results on the scene parsing task, while PSPNet provides a superior framework for pixel-level prediction. The proposed approach achieves state-of-the-art performance on various datasets. It came first in ImageNet scene parsing challenge 2016, PASCAL VOC 2012 benchmark and Cityscapes benchmark. A single PSPNet yields the new record of mIoU accuracy 85.4% on PASCAL VOC 2012 and accuracy 80.2% on Cityscapes.


We also extend π-Tuning on a uni-modal pretrained foundation model, T5 #REFR , to solve natural language understanding tasks.

(c) The test error surface on RefCOCOg #OTHEREFR , as a function of model weights in a two-dimensional subspace 3 .

It shows that checkpoints of similar tasks lie in the same basin of the surface.

Interpolation of those checkpoints could achieve lower test error on the surface. The visualization follows #OTHEREFR

(2018) , which derives an orthonormal basisû,v by three checkpoints.

We demonstrate the results in Table 3 , where we find that RTE is the task that benefits most with an improvement of 9.29% accuracy.

In Table 5 , experiments on RefCOCO, RefCOCO+, and RefCOCOg datasets with the base-size model show that π-Tuning is agnostic to PETL methods, providing significant and consistent performance gain for LoRA and Prompt Tuning.

π-LoRA improves a large margin of 4.20% on average accuracy across three datasets towards original LoRA and 1.94% for π-Prompt towards original Prompt Tuning.

Π-Tuning: Transferring Multimodal Foundation Models With Optimal Multi-Task Interpolation

Abstract: Foundation models have achieved great advances in multi-task learning with a unified interface of unimodal and multimodal tasks. However, the potential of such multi-task learners has not been exploited during transfer learning. In this work, we present a universal parameter-efficient transfer learning method, termed Predict-Interpolate Tuning (π-Tuning), for vision, language, and vision-language tasks. It aggregates the parameters of lightweight task-specific experts learned from similar tasks to aid the target downstream task. The task similarities are predicted in a unified modality-independent space, yielding a scalable graph to demonstrate task relationships. π-Tuning has several appealing benefits. First, it flexibly explores both intra-and intermodal transferability between similar tasks to improve the accuracy and robustness of transfer learning, especially in data-scarce scenarios. Second, it offers a systematical solution for transfer learning with multi-task prediction-and-theninterpolation, compatible with diverse types of parameter-efficient experts, such as prompt and adapter. Third, an extensive study of task-level mutual benefits on 14 unimodal and 6 multimodal datasets shows that π-Tuning surpasses fine-tuning and other parameter-efficient transfer learning methods both in full-shot and low-shot regimes. The task graph also enables an in-depth interpretable analysis of task transferability across modalities. The code will be available at https:

Exploring The Limits Of Transfer Learning With A Unified Text-To-Text Transformer

Abstract: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.

uni-modal pretrained foundation
natural language understanding


NeuralODE #REFR The prediction dynamics corresponding to the latent NeuralODE model is given by dXt dt = F nn (X t , z ≤r ; W 1 ) where z ≤r = F enc (X t0 , . . .

We perform a grid search over the following hyperparameters: regularization strengths λ Φ ∈ {10 −4 , 10 −3 , 5 × 10 −3 , 10 −2 }, λ REx ∈ {0, 10 −3 , 10 −2 }, and learning rates η ∈ {10 −2 , 10 −3 , 10 −4 }.

We choose the hyperparameters that result in sparsest model (i.e., with the least ||Φ|| 0 ) while achieving validation loss within 5% of the best validation loss in held-out in-distribution validation data.

, X tr ; W 2 ) encodes the initial observations using a recurrent neural network F enc (e.g., GRU), and F nn is a feedforward neural network.

The model is trained with an ODE solver (dopri5) and the gradients computed using the adjoint method #OTHEREFR .

We perform a grid search over the following hyperparameters: number of layers for F nn , L ∈ {1, 2, 3}, size of each hidden layer of F nn , d h ∈ {32, 64, 128}, size of the encoder representation z ≤r , d z ∈ {32, 64, 128}, batch sizes B ∈ {32, 64}, and learning rates η ∈ {10 −2 , 10 −3 , 10 −4 }.

Metaphysica: Ood Robustness In Physics-Informed Machine Learning

Abstract: A fundamental challenge in physics-informed machine learning (PIML) is the design of robust PIML methods for out-of-distribution (OOD) forecasting tasks. These OOD tasks require learning-to-learn from observations of the same (ODE) dynamical system with different unknown ODE parameters, and demand accurate forecasts even under out-of-support initial conditions and out-of-support ODE parameters. In this work we propose a solution for such tasks, which we define as a meta-learning procedure for causal structure discovery (including invariant risk minimization). Using three different OOD tasks, we empirically observe that the proposed approach significantly outperforms existing state-of-the-art PIML and deep learning methods.

Neural Ordinary Differential Equations

Abstract: We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.


Authors in #REFR proposed UcoSLAM that employs both natural and artificial landmarks in the environment to reconstruct the map.

VSLAM approaches have matured over the past years and many researchers contributed to this community by proposing efficient solutions to unresolved challenges in this area.

The authors of this paper also presented a comprehensive survey on diverse VSLAM state-of-the-art works and studied their trade-offs in #OTHEREFR .

Accordingly, many recent approaches focus on improving the front-end thread of the VSLAM systems for better pose estimation and visual odometry #OTHEREFR .

It works in keypoints-only, markers-only, and mixed modes and improves the global map builder and loop closure detector modules in case ArUco markers are found in the camera's field of view.

Another marker-based approach titled TagSLAM #OTHEREFR was proposed by Pfrommer and Daniilidis, which uses the capability of AprilTag #OTHEREFR fiducial markers for SLAM tasks.

Although the proposed approaches perform well in many scenarios, they cannot obtain any semantic information from the environment for an improved map representation. Gomez-Ojeda et al.

#OTHEREFR introduced PL-SLAM, which uses Oriented FAST and Rotated BRIEF (ORB) and Line Segment Detector (LSD) algorithms to extract visual features of points and lines in stereo vision cameras.

However, this approach is computationally intensive in feature tracking and extracting data from structural lines.

Marker-Based Visual Slam Leveraging Hierarchical Representations

Abstract: Fiducial markers can encode rich information about the environment and can aid Visual SLAM (VSLAM) approaches in reconstructing maps with practical semantic information. Current marker-based VSLAM approaches mainly utilize markers for improving feature detections in low-feature environments and/or for incorporating loop closure constraints, generating only low-level geometric maps of the environment prone to inaccuracies in complex environments. To bridge this gap, this paper presents a VSLAM approach utilizing a monocular camera along with fiducial markers to generate hierarchical representations of the environment while improving the camera pose estimate. The proposed approach detects semantic entities from the surroundings, including walls, corridors, and rooms encoded within markers, and appropriately adds topological constraints among them. Experimental results on a real-world dataset collected with a robot demonstrate that the proposed approach outperforms a traditional marker-based VSLAM baseline in terms of accuracy, given the addition of new constraints while creating enhanced map representations. Furthermore, it shows satisfactory results when comparing the reconstructed map quality to the one reconstructed using a LiDAR SLAM approach.

Ucoslam: Simultaneous Localization And Mapping By Fusion Of Keypoints And Squared Planar Markers

Abstract: Simultaneous Localization and Mapping is the process of simultaneously creating a map of the environment while navigating in it. Most of the SLAM approaches use natural features (e.g. keypoints) that are unstable over time, repetitive in many cases or their number insufficient for a robust tracking (e.g. in indoor buildings). Other researchers, on the other hand, have proposed the use of artificial landmarks, such as squared fiducial markers, placed in the environment to help tracking and relocalization. This paper proposes a novel SLAM approach by fusing natural and artificial landmarks in order to achieve long-term robust tracking in many scenarios. Our method has been compared to the start-of-the-art methods ORB-SLAM2 [1], LDSO [2] and SPM-SLAM [3] in the public datasets Kitti [4], Euroc-MAV [5], TUM [6] and SPM [3], obtaining better precision, robustness and speed. Our tests also show that the combination of markers and keypoints achieves better accuracy than each one of them independently.


This can be done by appropriately splitting up the prize (Moldovanu and Sela, 2001) , choosing a suitable reserve effort #REFR , setting an entry fee (Taylor, 1995) or by developing a revelation mechanism to select a subset of contestants from a pool of candidates (Mercier, 2018) .

Bug bounty programs are also used by governments and tech companies.

3 There have been comprehensive accounts on the rules of engagement of bug bounty programs (Laszka et al., 2018) , on the effectiveness and best practices of such programs (Walshe and Simpson, 2020; Malladi and Subramanian, 2020) , and on the incentives of researchers to participate in bug bounty schemes (Maillart et al., 2017) .

In this paper, we offer insights on some of the dimensions of bug bounty design, using a game-theoretic model of a simple contest building on the important work of Ghosh and Kleinberg (2016) and Sarne and Lepioshkin (2017) , where agents with different abilities decide on whether or not to exert costly effort for finding bugs.

Several salient features of bug bounty differentiate our approach from that of the standard optimal contest literature.

The design objective in traditional contests is to elicit the highest effort (or sum of efforts) from the contestants.

We focus on the simple problem how to maximize the likelihood to find bugs when a given amount of money is available for rewards.

In particular, we will focus on three design variables for bug bounty systems.

How large should the crowd of agents invited to find bugs be? Should paid experts be added to the crowd of invited bug finders? Should artificial bugs be added to the software to increase participation in bug finding and to increase the likelihood that the real bug is found?

To answer these questions and other, general questions about the nature of equilibria in bug bounty schemes, we develop a simple model of crowd-sourced security.

A group of individuals of arbitrary size is invited to search for a bug. Whether a bug exists is uncertain.

Decentralized Attack Search And The Design Of Bug Bounty Schemes *

Abstract: Systems and blockchains often have security vulnerabilities and can be attacked by adversaries, with potentially significant negative consequences. Therefore, organizations and blockchain infrastructure providers increasingly rely on bug bounty programs, where external individuals probe the system and report any vulnerabilities (bugs) in exchange for monetary rewards (bounty). We develop a contest model for bug bounty programs with an arbitrary number of agents who decide whether to undertake a costly search for bugs or not. Search costs are private information. Besides characterizing the ensuing equilibria, we show that even inviting an unlimited crowd does not guarantee that bugs are found. Adding paid agents can increase the efficiency of the bug bounty scheme although the crowd that is attracted becomes smaller. Finally, adding (known) bugs increases the likelihood that unknown bugs are found, but to limit reward payments it may be optimal to add them only with some probability.

Optimal Crowdsourcing Contests

Abstract: We study the design and approximation of optimal crowdsourcing contests. Crowdsourcing contests can be modeled as all-pay auctions because entrants must exert effort up-front to enter. Unlike all-pay auctions where a usual design objective would be to maximize revenue, in crowdsourcing contests, the principal only benefits from the submission with the highest quality. We give a theory for optimal crowdsourcing contests that mirrors the theory of optimal auction design: the optimal crowdsourcing contest is a virtual valuation optimizer (the virtual valuation function depends on the distribution of contestant skills and the number of contestants). We also compare crowdsourcing contests with more conventional means of procurement. In this comparison, crowdsourcing contests are relatively disadvantaged because the effort of losing contestants is wasted. Nonetheless, we show that crowdsourcing contests are 2-approximations to conventional methods for a large family of "regular" distributions, and 4-approximations, otherwise.


How Do Aleatoric And Epistemic Uncertainty Dif-

This is in line with the findings of #REFR , that Dropout as the epistemic component imitates the aleatoric logit distribution, unable to capture the different types of uncertainty.

Figure 4 shows Precision-Recall curves for the proposed model, as well as the baseline models that use the SSN and therefore model both uncertainties.

Note, that the SSN (diag) + Dropout corresponds to the heteroscedastic Bayesian Neural Network with Dropout derived by #OTHEREFR .

First, we find that the aleatoric component, targeting the data variation, performs better in terms of segmentation performance than the epistemic component for our model.

Further, the epistemic component of the LSN with diagonal covariance matrix performs worse (lower AUC), underpinning that the scaling of the loss has an effect on the post-hoc Laplace approximation.

For the Dropout-based methods, on the other hand, we see virtually no difference between the epistemic and aleatoric components.

What does this tell us? We see that the aleatoric and epistemic components of the LSN clearly capture different aspects of the model's uncertainty.

To better understand where this difference comes from, Figure 5 shows epistemic and aleatoric variance maps of sigmoids for a given ISIC test set sample.

While the epistemic component assigns low variance to this in-distribution sample, the aleatoric component expresses the variation of plausible predictions for the image.

We find this behavior attractive: In the Bayesian framework, the aleatoric component should play the role of the predictor, as it imitates the distribution of annotations Figure 4 : Precision-Recall plots for the ISIC test dataset show that aleatoric and epistemic variance in the LSN models (first row) are not the same, opposed to the SSN + Dropout models.

The better performance of the aleatoric component (higher AUC) indicates that the model learns to separate data variation from model uncertainty.

Laplacian Segmentation Networks: Improved Epistemic Uncertainty From Spatial Aleatoric Uncertainty

Abstract: Out of distribution (OOD) medical images are frequently encountered, e.g. because of site-or scanner differences, or image corruption. OOD images come with a risk of incorrect image segmentation, potentially negatively affecting downstream diagnoses or treatment. To ensure robustness to such incorrect segmentations, we propose Laplacian Segmentation Networks (LSN) that jointly model epistemic (model) and aleatoric (data) uncertainty in image segmentation. We capture data uncertainty with a spatially correlated logit distribution. For model uncertainty, we propose the first Laplace approximation of the weight posterior that scales to large neural networks with skip connections that have high-dimensional outputs. Empirically, we demonstrate that modelling spatial pixel correlation allows the Laplacian Segmentation Network to successfully assign high epistemic uncertainty to out-of-distribution objects appearing within images.

What Uncertainties Do We Need In Bayesian Deep Learning For Computer Vision?

Abstract: There are two major types of uncertainty one can model. Aleatoric uncertainty captures noise inherent in the observations. On the other hand, epistemic uncertainty accounts for uncertainty in the model - uncertainty which can be explained away given enough data. Traditionally it has been difficult to model epistemic uncertainty in computer vision, but with new Bayesian deep learning tools this is now possible. We study the benefits of modeling epistemic vs. aleatoric uncertainty in Bayesian deep learning models for vision tasks. For this we present a Bayesian deep learning framework combining input-dependent aleatoric uncertainty together with epistemic uncertainty. We study models under the framework with per-pixel semantic segmentation and depth regression tasks. Further, our explicit uncertainty formulation leads to new loss functions for these tasks, which can be interpreted as learned attenuation. This makes the loss more robust to noisy data, also giving new state-of-the-art results on segmentation and depth regression benchmarks.


The landscape of neural rendering changed with NeRF #REFR , which proposed a simple, yet revolutionary approach for photorealistic novel-view synthesis of static scenes.

NeRFs attempt to learn a scene representation for novelview synthesis by modeling the radiance field with learnable functions.

A variety of approaches have been proposed recently for neural rendering, exploiting voxel grids #OTHEREFR , neural textures #OTHEREFR , point-clouds #OTHEREFR , and neural implicit functions #OTHEREFR .

NeRF attempts to map from the 5-d light fields to 4-d space consisting of color c (RGB) and density σ: likelihood that the light ray at this 5-d co-ordinate is terminated by occlusion.

Since the introduction of original NeRF formulation, several variations and improvements #OTHEREFR have been proposed.

Flexnerf: Photorealistic Free-Viewpoint Rendering Of Moving Humans From Sparse Views

Abstract: We present FlexNeRF, a method for photorealistic freeviewpoint rendering of humans in motion from monocular videos. Our approach works well with sparse views, which is a challenging scenario when the subject is exhibiting fast/complex motions. We propose a novel approach which jointly optimizes a canonical time and pose configuration, with a pose-dependent motion field and pose-independent temporal deformations complementing each other. Thanks to our novel temporal and cyclic consistency constraints along with additional losses on intermediate representation such as segmentation, our approach provides high quality outputs as the observed views become sparser. We empirically demonstrate that our method significantly outperforms the state-of-the-art on public benchmark datasets as well as a self-captured fashion dataset. The project page is available at: https://flex-nerf.github.io/.

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.


Other successful exploration approaches like Boot-strappedDQN #REFR are unscalable in MARL because of the exponentially large state-action space.

Based on the Optimism in the Face of Uncertainty (OFU) principle, the high-level idea is to capture some notion of uncertainty or novelty, and augment the extrinsic reward the environment emits with an intrinsic reward that quantifies the uncertainty.

For instance, the count-based method #OTHEREFR measures novelty through the number of times the agent observes a state-action tuple. #OTHEREFR

[2016] propose an intrinsic bonus based on the maximization of information gain about the agent's belief of environment dynamics.

Despite their recent success, simply applying bonus-based methods to MARL does not guarantee effective exploration.

As the reward signal is shared across all agents, adding one additional centralized intrinsic reward may still be inefficient to learn structured cooperation due to the multi-agent credit-assignment challenge.

A recent branch of research proposes to drive multi-agent exploration by promoting col-laboration among agents through the maximization of the correlation or influence of agents.

The correlation is commonly realized by the mutual information (MI) of quantities that define or reflect agents' behaviour, such as the trajectory history of each agent.

For instance, MAVEN #OTHEREFR ] learns a hierarchical policy to produce a latent variable that encodes the information about the joint policy, and maximizes the MI between this latent variable and the joint trajectories to encourage the correlation of agents' behaviour.

Some other methods try to promote collaboration by maximizing pairwise MI between every two agents.

Conditionally Optimistic Exploration For Cooperative Deep Multi-Agent Reinforcement Learning

Abstract: Efficient exploration is critical in cooperative deep Multi-Agent Reinforcement Learning (MARL). In this paper, we propose an exploration method that efficiently encourages cooperative exploration based on the idea of the theoretically justified tree search algorithm UCT (Upper Confidence bounds applied to Trees). The high-level intuition is that to perform optimism-based exploration, agents would achieve cooperative strategies if each agent's optimism estimate captures a structured dependency relationship with other agents. At each node (i.e., action) of the search tree, UCT performs optimismbased exploration using a bonus derived by conditioning on the visitation count of its parent node. We provide a perspective to view MARL as tree search iterations and develop a method called Conditionally Optimistic Exploration (COE). We assume agents take actions following a sequential order, and consider nodes at the same depth of the search tree as actions of one individual agent. COE computes each agent's state-action value estimate with an optimistic bonus derived from the visitation count of the state and joint actions taken by agents up to the current agent. COE is adaptable to any value decomposition method for centralized training with decentralized execution. Experiments across various cooperative MARL benchmarks show that COE outperforms current stateof-the-art exploration methods on hard-exploration tasks. Preprint. Under review.

Deep Exploration Via Bootstrapped Dqn

Abstract: Efficient exploration in complex environments remains a major challenge for reinforcement learning. We propose bootstrapped DQN, a simple algorithm that explores in a computationally and statistically efficient manner through use of randomized value functions. Unlike dithering strategies such as epsilon-greedy exploration, bootstrapped DQN carries out temporally-extended (or deep) exploration; this can lead to exponentially faster learning. We demonstrate these benefits in complex stochastic MDPs and in the large-scale Arcade Learning Environment. Bootstrapped DQN substantially improves learning times and performance across most Atari games.


To trigger a liquidation, the liquidator needs to send a transaction #REFR to the augmented contract along with parameters (e.g., the flash loan size and liquidation amount).

Such protection corresponds to an authentication mechanism, a common practice that the practitioners' community employs. We proceed with a real-world authentication example.

In Figure 2 , we show how a liquidation contract 3 attempts to prevent a transaction imitation attack.

The solidity code of Figure 2 's liquidation contract is not open-source, and we therefore decompile the #OTHEREFR A liquidation refers to the process of selling collateral to secure debt #OTHEREFR . #OTHEREFR Address: 0x18C0cA3947E255881f94DE50B3a906Fc2759F7FE.

contract bytecode with a state-of-the-art EVM contract decompiler #OTHEREFR .

If there is no protection mechanism, an adversary might frontrun the liquidator, by calling the same contract with identical parameters.

However, the presented contract requires the transaction sender to match a specific address (in line 3).

Specifically, the liquidation function printMoney() is only callable by a hard-coded address.

If this condition is not met, the naive imitation attack from Section 2.3 fails.

To circumvent authentication protections, one approach is to synthesize an adversarial contract that replicates the liquidator's customized contract.

The Blockchain Imitation Game

Abstract: The use of blockchains for automated and adversarial trading has become commonplace. However, due to the transparent nature of blockchains, an adversary is able to observe any pending, not-yet-mined transactions, along with their execution logic. This transparency further enables a new type of adversary, which copies and front-runs profitable pending transactions in real-time, yielding significant financial gains. Shedding light on such "copy-paste" malpractice, this paper introduces the Blockchain Imitation Game and proposes a generalized imitation attack methodology called APE. Leveraging dynamic program analysis techniques, APE supports the automatic synthesis of adversarial smart contracts. Over a timeframe of one year (1st of August, 2021 to 31st of July, 2022), APE could have yielded 148.96M USD in profit on Ethereum, and 42.70M USD on BNB Smart Chain (BSC). Not only as a malicious attack, we further show the potential of transaction and contract imitation as a defensive strategy. Within one year, we find that APE could have successfully imitated 13 and 22 known Decentralized Finance (DeFi) attacks on Ethereum and BSC, respectively. Our findings suggest that blockchain validators can imitate attacks in real-time to prevent intrusions in DeFi.

Sok: Consensus In The Age Of Blockchains

Abstract: The core technical component of blockchains is consensus: how to reach agreement among a distributed network of nodes. A plethora of blockchain consensus protocols have been proposed---ranging from new designs, to novel modifications and extensions of consensus protocols from the classical distributed systems literature. The inherent complexity of consensus protocols and their rapid and dramatic evolution makes it hard to contextualize the design landscape. We address this challenge by conducting a systematization of knowledge of blockchain consensus protocols. After first discussing key themes in classical consensus protocols, we describe: (i) protocols based on proof-of-work; (ii) proof-of-X protocols that replace proof-of-work with more energy-efficient alternatives; and (iii) hybrid protocols that are compositions or variations of classical consensus protocols. This survey is guided by a systematization framework we develop, to highlight the various building blocks of blockchain consensus design, along with a discussion on their security and performance properties. We identify research gaps and insights for the community to consider in future research endeavours.


Other architectures also use a MAC-array structure, such as the Google Edge TPU #REFR architecture.

IoT devices at the edge are resource and energy constrained.

This advocates for efficient hardware ML accelerators, increased but limited memory resources with reduced power consumption, and optimized ML algorithms well suited for these constrained nodes.

In #OTHEREFR an overview of the architectures for efficient deep learning on IoT devices is presented.

Flexible approaches such as GPUs are too power-hungry to be considered for integrating ML in real IoT devices, while FPGAs are not power-efficient enough. Research focuses on specialized accelerators, especially using dataflow paradigms.

Eyeriss #OTHEREFR is an example of such an accelerator where small processing elements using private buffers are interconnected to implement row-stationary dataflow for efficient convolution operations.

These accelerators focus on convolutions, however complete networks may need more flexibility.

This is the domain of heterogeneous accelerators such as the Orlando platform from STMicroelectronics #OTHEREFR , in which a dataflow hardware engine is coupled with SIMD DSP clusters.

To be more efficient, algorithm-architecture matching needs to be improved.

In #OTHEREFR algorithmic and processor techniques are studied to address deep learning in the edge.

Quantization and network pruning are popular solutions to reduce the memory footprint of a DNN network #OTHEREFR with reduced classification accuracy degradation.

Abstract: Increased capabilities such as recognition and selfadaptability are now required from IoT applications. While IoT node power consumption is a major concern for these applications, cloud-based processing is becoming unsustainable due to continuous sensor or image data transmission over the wireless network. Thus optimized ML capabilities and data transfers should be integrated in the IoT node. Moreover, IoT applications are torn between sporadic data-logging and energy-hungry data processing (e.g. image classification). Thus, the versatility of the node is key in addressing this wide diversity of energy and processing needs. This paper presents SamurAI, a versatile IoT node bridging this gap in processing and in energy by leveraging two on-chip subsystems: a low power, clock-less, event-driven Always-Responsive (AR) part and an energy-efficient On-Demand (OD) part. AR contains a 1.7MOPS event-driven, asynchronous Wake-up Controller (WuC) with a 207ns wake-up time optimized for sporadic computing, while OD combines a deep-sleep RISC-V CPU and 1.3TOPS/W Machine Learning (ML) for more complex tasks up to 36GOPS. This architecture partitioning achieves best in class versatility metrics such as peak performance to idle power ratio. On an applicative classification scenario, it demonstrates system power gains, up to 3.5x compared to cloud-based processing, and thus extended battery lifetime.

In-Datacenter Performance Analysis Of A Tensor Processing Unit

Abstract: Many architects believe that major improvements in cost-energy-performance must now come from domain-specific hardware. This paper evaluates a custom ASIC---called a Tensor Processing Unit (TPU) --- deployed in datacenters since 2015 that accelerates the inference phase of neural networks (NN). The heart of the TPU is a 65,536 8-bit MAC matrix multiply unit that offers a peak throughput of 92 TeraOps/second (TOPS) and a large (28 MiB) software-managed on-chip memory. The TPU's deterministic execution model is a better match to the 99th-percentile response-time requirement of our NN applications than are the time-varying optimizations of CPUs and GPUs that help average throughput more than guaranteed latency. The lack of such features helps explain why, despite having myriad MACs and a big memory, the TPU is relatively small and low power. We compare the TPU to a server-class Intel Haswell CPU and an Nvidia K80 GPU, which are contemporaries deployed in the same datacenters. Our workload, written in the high-level TensorFlow framework, uses production NN applications (MLPs, CNNs, and LSTMs) that represent 95% of our datacenters' NN inference demand. Despite low utilization for some applications, the TPU is on average about 15X -- 30X faster than its contemporary GPU or CPU, with TOPS/Watt about 30X -- 80X higher. Moreover, using the CPU's GDDR5 memory in the TPU would triple achieved TOPS and raise TOPS/Watt to nearly 70X the GPU and 200X the CPU.


For Aachen #REFR , parallel lines from the line-VP association graph are colored the same.

From the recovered structures, we can clearly perceive the scene and easily recognize the main Manhattan directions #OTHEREFR .

To demonstrate the scalability of the proposed system, we also run our method on two large-scale datasets: Aachen (6,697 images) #OTHEREFR and Rome city (16,179 images) #OTHEREFR . Fig.

7 shows that our method produces reliable line maps with clear structures.

Note that the camera poses from Bundler #OTHEREFR on Rome city are far from perfect, while our mapping still works reasonably well. The efficiency Figure 7 .

Scalability to large-scale datasets: Aachen (6,697 images) #OTHEREFR and Rome (16,179 images) #OTHEREFR .

For Rome #OTHEREFR , we visualize 10 representative components individually. Figure 8 .

Uncertainty in line triangulation measured by the largest eigenvalue of the covariance (Sec. D in supp.).

Left: Each segment is colored by the uncertainty in the triangulation.

Lines that align with the epipolar lines (shown in blue) exhibit higher (red) uncertainty.

Right: We perform a small synthetic experiment to illustrate this.

3D Line Mapping Revisited

Abstract: In contrast to sparse keypoints, a handful of line segments can concisely encode the high-level scene layout, as they often delineate the main structural elements. In addition to offering strong geometric cues, they are also omnipresent in urban landscapes and indoor scenes. Despite their apparent advantages, current line-based reconstruction methods are far behind their point-based counterparts. In this paper we aim to close the gap by introducing LIMAP, a library for 3D line mapping that robustly and efficiently creates 3D line maps from multi-view imagery. This is achieved through revisiting the degeneracy problem of line triangulation, carefully crafted scoring and track building, and exploiting structural priors such as line coincidence, parallelism, and orthogonality. Our code integrates seamlessly with existing point-based Structure-from-Motion methods and can leverage their 3D points to further improve the line reconstruction. Furthermore, as a byproduct, the method is able to recover 3D association graphs between lines and points / vanishing points (VPs). In thorough experiments, we show that LIMAP significantly outperforms existing approaches for 3D line mapping. Our robust 3D line maps also open up new research directions. We show two example applications: visual localization and bundle adjustment, where integrating lines alongside points yields the best results. Code is available at https://github.com/cvg/limap.

Benchmarking 6Dof Outdoor Visual Localization In Changing Conditions

Abstract: Visual localization enables autonomous vehicles to navigate in their surroundings and augmented reality applications to link virtual to real worlds. Practical visual localization approaches need to be robust to a wide variety of viewing condition, including day-night changes, as well as weather and seasonal variations, while providing highly accurate 6 degree-of-freedom (6DOF) camera pose estimates. In this paper, we introduce the first benchmark datasets specifically designed for analyzing the impact of such factors on visual localization. Using carefully created ground truth poses for query images taken under a wide variety of conditions, we evaluate the impact of various factors on 6DOF camera pose estimation accuracy through extensive experiments with state-of-the-art localization approaches. Based on our results, we draw conclusions about the difficulty of different conditions, showing that long-term localization is far from solved, and propose promising avenues for future work, including sequence-based localization approaches and the need for better local features. Our benchmark is available at visuallocalization.net.


For a fair comparison, we use the same experimental setups and compare these models with and without our proposed modules integrated. The default backbone is ResNet-50 #REFR . We summarize the results in Table 1 .

In this section, we conduct experiments on the dynamic query aggregation modules with the current Transformer-based object detectors on the ImageNet VID benchmark #OTHEREFR .

The experiments of vanilla query aggregation will be provided in the following section.

For most of the DETR-based object detectors, the performance can be improved by at least 2.4% on the metric of mAP compared with those not integrated with our proposed modules.

The performance is even better than those integrated with TransVOD [100] by a large margin, which validates the effectiveness of our dynamic query aggregation modules.

When considering the objects' sizes, we notice that the performance of large or medium objects is much better than that of small objects.

We argue that this is because the process to generate the dynamic queries Q d i is only based on the global features, which lack enough information for the small objects.

Faq: Feature Aggregated Queries For Transformer-Based Video Object Detectors

Abstract: Video object detection needs to solve feature degradation situations that rarely happen in the image domain. One solution is to use the temporal information and fuse the features from the neighboring frames. With Transformerbased object detectors getting a better performance on the image domain tasks, recent works began to extend those methods to video object detection. However, those existing Transformer-based video object detectors still follow the same pipeline as those used for classical object detectors, like enhancing the object feature representations by aggregation. In this work, we take a different perspective on video object detection. In detail, we improve the qualities of queries for the Transformer-based models by aggregation. To achieve this goal, we first propose a vanilla query aggregation module that weighted averages the queries according to the features of the neighboring frames. Then, we extend the vanilla module to a more practical version, which generates and aggregates queries according to the features of the input frames. Extensive experimental results validate the effectiveness of our proposed methods: On the challenging ImageNet VID benchmark, when integrated with our proposed modules, the current state-of-the-art Transformerbased object detectors can be improved by more than 2.4% on mAP and 4.2% on AP 50 .

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


In #REFR , an extensive evaluation of multiple combinations of feature extractors and metric learning approaches is discussed.

There are many difficulties for Re-ID retrieval #OTHEREFR , among them: (i) different viewpoints, (ii) possible low-image resolutions, (iii) illumination changes, (iv) occlusions, (v) difficulty of manually labeling data for training, (vi) large amount of data to be processed.

The challenge of improving the effectiveness of these systems, especially in open-world scenarios, has attracted a lot of research efforts from the scientific community #OTHEREFR .

Initially, person Re-ID retrieval systems were mainly based on the use of handcrafted feature representations #OTHEREFR .

Besides that, other strategies commonly used to generic image retrieval, like bag of visual words #OTHEREFR , have also been employed.

Aiming at further improving the quality of the results, rather than using traditional distance measures, researchers have proposed metric learning approaches for Re-ID #OTHEREFR , most of them based on supervised models.

Due to the significant impact of deep learning on common image retrieval and machine learning #OTHEREFR , the Convolutional Neural Networks (CNN) also have gained a lot of attention and have been widely employed to solve Re-ID tasks in the recent years #OTHEREFR .

Among the works, existing networks have been trained for Re-ID #OTHEREFR , and some new architectures have been proposed solely focusing on Re-ID #OTHEREFR .

Despite the success of deep learning, one of the main difficulties resides in the lack of large publicly available Re-ID datasets for training #OTHEREFR , mainly be-cause manually labeling images is a very difficulty task, especially in open-world scenarios #OTHEREFR , where the datasets may increase dynamically.

In order to mitigate this problem, many authors have proposed multi-source training, which consists in joining multiple available datasets for training #OTHEREFR , usually increasing the network capacity for generalization.

However, despite several advances in multi-source approaches, it is inevitable the need for intensive manual annotation for obtaining training data.

Person Re-Id Through Unsupervised Hypergraph Rank Selection And Fusion

Abstract: Person Re-ID has been gaining a lot of attention and nowadays is of fundamental importance in many camera surveillance applications. The task consists of identifying individuals across multiple cameras that have no overlapping views. Most of the approaches require labeled data, which is not always available, given the huge amount of demanded data and the difficulty of manually assigning a class for each individual. Recently, studies have shown that re-ranking methods are capable of achieving significant gains, especially in the absence of labeled data. Besides that, the fusion of feature extractors and multiple-source training is another promising research direction not extensively exploited. We aim to fill this gap through a manifold rank aggregation approach capable of exploiting the complementarity of different person Re-ID rankers. In this work, we perform a completely unsupervised selection and fusion of diverse ranked lists obtained from multiple and diverse feature extractors. Among the contributions, this work proposes a query performance prediction measure that models the relationship among images considering a hypergraph structure and does not require the use of any labeled data. Expressive gains were obtained in four datasets commonly used for person Re-ID. We achieved results competitive to the state-of-the-art in most of the scenarios.

A Systematic Evaluation And Benchmark For Person Re-Identification: Features, Metrics, And Datasets

Abstract: Person re-identification (re-id) is a critical problem in video analytics applications such as security and surveillance. The public release of several datasets and code for vision algorithms has facilitated rapid progress in this area over the last few years. However, directly comparing re-id algorithms reported in the literature has become difficult since a wide variety of features, experimental protocols, and evaluation metrics are employed. In order to address this need, we present an extensive review and performance evaluation of single- and multi-shot re-id algorithms. The experimental protocol incorporates the most recent advances in both feature extraction and metric learning. To ensure a fair comparison, all of the approaches were implemented using a unified code library that includes 11 feature extraction algorithms and 22 metric learning and ranking techniques. All approaches were evaluated using a new large-scale dataset that closely mimics a real-world problem setting, in addition to 16 other publicly available datasets: VIPeR, GRID, CAVIAR, DukeMTMC4ReID, 3DPeS, PRID, V47, WARD, SAIVT-SoftBio, CUHK01, CHUK02, CUHK03, RAiD, iLIDSVID, HDA+ and Market1501. The evaluation codebase and results will be made publicly available for community use.


Generative design and machine learning (ML) methods are increasingly being established in fluid dynamics, for example in #REFR , helping engineers to better understand their problem domains earlier on in the design process, and increase the likelihood of making the right design decisions.

(2021) showed that the flow around 3D cylindrical objects can be predicted based on spanwise-averaged Navier-Stokes equations.

However, it is unclear how their method handles the wide diversity of shapes in early design stages.

To perform the computational fluid dynamics (CFD) simulations, the lattice Boltzmann method (LBM) has made considerable progress in the prediction of turbulent flows, including direct numerical simulation (DNS), as in #OTHEREFR , under-resolved DNS, as in #OTHEREFR , and large eddy simulations (LES), as in #OTHEREFR .

The LBM is renowned for its high performance, shown in #OTHEREFR , although 3D simulations remain a costly endeavor.

This increases the demand for efficiency of QD methods applied in these kinds of domains even more.

These methods generally need large data sets with a wide variety of shapes and flow patterns.

Although novel methods like physics-informed neural networks require fewer data points in some cases, for example in work by #OTHEREFR

(2020) , it is unclear how they perform on the wide variety of shapes in early design stages.

The increased integration of ML in the design process is pushing methods in fluid dynamics and optimization to their limits, due to the requirements on the amount and variation of training data points.

Especially when data sets are unavailable for a certain problem, they need to be created first.

Efficient Quality Diversity Optimization Of 3D Buildings Through 2D Pre-Optimization A Preprint

Abstract: Quality diversity algorithms can be used to efficiently create a diverse set of solutions to inform engineers' intuition. But quality diversity is not efficient in very expensive problems, needing 100.000s of evaluations. Even with the assistance of surrogate models, quality diversity needs 100s or even 1000s of evaluations, which can make it use infeasible. In this study we try to tackle this problem by using a pre-optimization strategy on a lowerdimensional optimization problem and then map the solutions to a higher-dimensional case. For a use case to design buildings that minimize wind nuisance, we show that we can predict flow features around 3D buildings from 2D flow features around building footprints. For a diverse set of building designs, by sampling the space of 2D footprints with a quality diversity algorithm, a predictive model can be trained that is more accurate than when trained on a set of footprints that were selected with a space-filling algorithm like the Sobol sequence. Simulating only 16 buildings in 3D, a set of 1024 building designs * This is the final version and has been accepted for publication in Evolutionary Computation (MIT Press)

Deep Learning Observables In Computational Fluid Dynamics

Abstract: Many large scale problems in computational fluid dynamics such as uncertainty quantification, Bayesian inversion, data assimilation and PDE constrained optimization are considered very challenging computationally as they require a large number of expensive (forward) numerical solutions of the corresponding PDEs. We propose a machine learning algorithm, based on deep artificial neural networks, that learns the underlying input parameters to observable map from a few training samples (computed realizations of this map). By a judicious combination of theoretical arguments and empirical observations, we find suitable network architectures and training hyperparameters that result in robust and efficient neural network approximations of the parameters to observable map. Numerical experiments for realistic high dimensional test problems, demonstrate that even with approximately 100 training samples, the resulting neural networks have a prediction error of less than one to two percent, at a computational cost which is several orders of magnitude lower than the cost of the underlying PDE solver. ::: Moreover, we combine the proposed deep learning algorithm with Monte Carlo (MC) and Quasi-Monte Carlo (QMC) methods to efficiently compute uncertainty propagation for nonlinear PDEs. Under the assumption that the underlying neural networks generalize well, we prove that the deep learning MC and QMC algorithms are guaranteed to be faster than the baseline (quasi-) Monte Carlo methods. Numerical experiments demonstrating one to two orders of magnitude speed up over baseline QMC and MC algorithms, for the intricate problem of computing probability distributions of the observable, are also presented.


The fixation object is automatically annotated by a pre-trained object detector #REFR with manual corrections.

To achieve multimodal attention detection, we developed a data annotation pipeline for efficient processing of the multimodal data.

First, we recognized a list of significant primitives in human attention study as detailed in Table  I .

Specifically, the eye state, gaze position, and fixation position can be extracted from Pupil Core's software system outputs but require careful calibration as errors are common in default detectors.

Head pose and surface tracking are derived with the aid of eight Apriltags deployed next to the assembly platform.

Of our greatest interest, the distracted states and in-operation states are manually labelled frame-to-frame to identify subtle attention lapses.

An example of the annotated sequence 002 is shown in Fig. 3 .

Note ensuring ad-hoc occurrence of real-world distractors in an assembly task is critical for attention lapse study.

We curated two pilot data sequences of a human participant assembling a dummy phone in a realistic production laboratory with manufacturing background noises, moving MiR robots, and workers around. According to K. Holmqvist and R.

Detecting Worker Attention Lapses In Human-Robot Interaction: An Eye Tracking And Multimodal Sensing Study

Abstract: The advent of industrial robotics and autonomous systems endow human-robot collaboration in a massive scale. However, current industrial robots are restrained in co-working with human in close proximity due to inability of interpreting human agents' attention. Human attention study is non-trivial since it involves multiple aspects of the mind: perception, memory, problem solving, and consciousness. Human attention lapses are particularly problematic and potentially catastrophic in industrial workplace, from assembling electronics to operating machines. Attention is indeed complex and cannot be easily measured with single-modality sensors. Eye state, head pose, posture, and manifold environment stimulus could all play a part in attention lapses. To this end, we propose a pipeline to annotate multimodal dataset of human attention tracking, including eye tracking, fixation detection, third-person surveillance camera, and sound. We produce a pilot dataset containing two fully annotated phone assembly sequences in a realistic manufacturing environment. We evaluate existing fatigue and drowsiness prediction methods for attention lapse detection. Experimental results show that human attention lapses in production scenarios are more subtle and imperceptible than well-studied fatigue and drowsiness.

Faster R-Cnn: Towards Real-Time Object Detection With Region Proposal Networks

Abstract: State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet [1] and Fast R-CNN [2] have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with ’attention’ mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model [3] , our detection system has a frame rate of 5 fps ( including all steps ) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available.

fixation object
pre-trained object detector