Single-view planar estimation results on ScanNet #REFR by measuring the plane recall with a fixed Intersection over Union threshold 0.5 and a varying depth error threshold (from 0 to 0.8m).

The model is trained on 4 NVIDIA-V100 GPUs for a total of 100k steps.

Before our main training, we add a bootstrap training phase for 50k steps, where we initialize the alpha prediction with the segmentation by turning off L rgb and enforcing a segmentation loss on the alpha channel.

To train the model with multi-view input, we set T = 2 and the trained model can be applied for T ≥ 1.

We cancel L pln for non-planar regions in the second input view. Figure 5 .

Structural Multiplane Image: Bridging Neural View Synthesis And 3D Reconstruction

Abstract: The Multiplane Image (MPI), containing a set of frontoparallel RGBα layers, is an effective and efficient representation for view synthesis from sparse inputs. Yet, its fixed structure limits the performance, especially for surfaces imaged at oblique angles. We introduce the Structural MPI (S-MPI), where the plane structure approximates 3D scenes concisely. Conveying RGBα contexts with geometricallyfaithful structures, the S-MPI directly bridges view synthesis and 3D reconstruction. It can not only overcome the critical limitations of MPI, i.e., discretization artifacts from sloped surfaces and abuse of redundant layers, and can also acquire planar 3D reconstruction. Despite the intuition and demand of applying S-MPI, great challenges are introduced, e.g., high-fidelity approximation for both RGBα layers and plane poses, multi-view consistency, non-planar regions modeling, and efficient rendering with intersected planes. Accordingly, we propose a transformer-based network based on a segmentation model [4]. It predicts compact and expressive S-MPI layers with their corresponding masks, poses, and RGBα contexts. Non-planar regions are inclusively handled as a special case in our unified framework. Multi-view consistency is ensured by sharing global proxy embeddings, which encode plane-level features covering the complete 3D scenes with aligned coordinates. Intensive experiments show that our method outperforms both previous state-of-the-art MPI-based view synthesis methods and planar reconstruction methods.

Scannet: Richly-Annotated 3D Reconstructions Of Indoor Scenes

Abstract: A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets. Unfortunately, in the context of RGB-D scene understanding, very little data is available – current datasets cover a small range of scene views and have limited semantic annotations. To address this issue, we introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations. To collect this data, we designed an easy-to-use and scalable RGB-D capture system that includes automated surface reconstruction and crowdsourced semantic annotation. We show that using this data helps achieve state-of-the-art performance on several 3D scene understanding tasks, including 3D object classification, semantic voxel labeling, and CAD model retrieval.


Following MVF #REFR , we use dynamic voxelization, which establishes the bi-directional mapping between points and voxels without sampling operation.

For images captured by a monocular camera, we train a semantic segmentation backbone, i.e., LR-ASPP #OTHEREFR , to generate image feature maps F img ∈ R H×W ×C .

Each channel in the image feature map represents the confidence score for a specific category, including background.

Notice that the parameters of these layers are frozen during training.

For LiDAR point clouds, voxelization can divide points into an evenly spaced grid of voxels.

The size of a voxel is predefined as (l, w, h), representing length along X, Y, Z axes in 3D space.

For Multi-scale Voxelization Fusion Framework, we define different voxel sizes and apply voxelization within every scale.

Point p i is assigned to voxels {v j0 , . . .

, v js } based on their relative spatial coordinates in different scales.

Sdvrf: Sparse-To-Dense Voxel Region Fusion For Multi-Modal 3D Object Detection

Abstract: In the perception task of autonomous driving, multimodal methods have become a trend due to the complementary characteristics of LiDAR point clouds and image data. However, the performance of previous methods is usually limited by the sparsity of the point cloud or the noise problem caused by the misalignment between LiDAR and the camera. To solve these two problems, we present a new concept, Voxel Region (VR), which is obtained by projecting the sparse local point clouds in each voxel dynamically. And we propose a novel fusion method, named Sparse-to-Dense Voxel Region Fusion (SDVRF). Specifically, more pixels of the image feature map inside the VR are gathered to supplement the voxel feature extracted from sparse points and achieve denser fusion. Meanwhile, different from prior methods, which project the size-fixed grids, our strategy of generating dynamic regions achieves better alignment and avoids introducing too much background noise. Furthermore, we propose a multiscale fusion framework to extract more contextual information and capture the features of objects of different sizes. Experiments on the KITTI dataset show that our method improves the performance of different baselines, especially on classes of small size, including Pedestrian and Cyclist.

End-To-End Multi-View Fusion For 3D Object Detection In Lidar Point Clouds

Abstract: Recent work on 3D object detection advocates point cloud voxelization in birds-eye view, where objects preserve their physical dimensions and are naturally separable. When represented in this view, however, point clouds are sparse and have highly variable point density, which may cause detectors difficulties in detecting distant or small objects (pedestrians, traffic signs, etc.). On the other hand, perspective view provides dense observations, which could allow more favorable feature encoding for such cases. In this paper, we aim to synergize the birds-eye view and the perspective view and propose a novel end-to-end multi-view fusion (MVF) algorithm, which can effectively learn to utilize the complementary information from both. Specifically, we introduce dynamic voxelization, which has four merits compared to existing voxelization methods, i) removing the need of pre-allocating a tensor with fixed size; ii) overcoming the information loss due to stochastic point/voxel dropout; iii) yielding deterministic voxel embeddings and more stable detection outcomes; iv) establishing the bi-directional relationship between points and voxels, which potentially lays a natural foundation for cross-view feature fusion. By employing dynamic voxelization, the proposed feature fusion architecture enables each point to learn to fuse context information from different views. MVF operates on points and can be naturally extended to other approaches using LiDAR point clouds. We evaluate our MVF model extensively on the newly released Waymo Open Dataset and on the KITTI dataset and demonstrate that it significantly improves detection accuracy over the comparable single-view PointPillars baseline.


D. Architecture And Training Of Neural Network

The U-net architecture was used #REFR because of the ability to extract features from the input data at different spatial resolutions which are later used in the decoder part of the network to form a prediction.

All calculations were performed on a DGX station equipped with Tesla V100 GPU cards (Nvidia, Santa Clara, CA, US). The PyTorch DL framework #OTHEREFR was used.

The network had 4 levels, with the encoder part at each level containing the two 3D convolution layers each followed by the leakyRelu activation.

The spatial resolution was decreased with a max-pooling layer by a factor of two.

The bottom of the U-net consists of two 3D convolution and one 3D transposed convolution layers, each followed by leakyRelu activation.

The decoder part, at each level, consists of two 3D transposed convolution each followed by leaky Relu.

The spatial resolution was increase by a trilinear interpolation by factor of 2.

Predicting Dynamic, Motion-Related Changes In B 0 Field In The Brain At A 7 T Mri Using A Subject-Specific Fine-Tuned U-Net

Abstract: Subject movement during the magnetic resonance examination is inevitable and causes not only image artefacts but also deteriorates the homogeneity of the main magnetic field (B0), which is a prerequisite for high quality data. Thus, characterization of changes to B0, e.g. induced by patient movement, is important for MR applications that are prone to B0 inhomogeneities. We propose a deep learning based method to predict such changes within the brain from the change of the head position to facilitate retrospective or even real-time correction. A 3D U-net was trained on in vivo brain 7 T MRI data. The input consisted of B0 maps and anatomical images at an initial position, and anatomical images at a different head position (obtained by applying a rigid-body transformation on the initial anatomical image). The output consisted of B0 maps at the new head positions. We further fine-tuned the network weights to each subject by measuring a limited number of head positions of the given subject, and trained the U-net with these data. Our approach was compared to established dynamic B0 field mapping via interleaved navigators, which suffer from limited spatial resolution and the need for undesirable sequence modifications. Qualitative and quantitative comparison showed similar performance between an interleaved navigator-equivalent method and proposed method. We therefore conclude that it is feasible to predict B0 maps from rigid subject movement and, when combined with external tracking hardware, this information could be used to improve the quality of magnetic resonance acquisitions without the use of navigators.

U-Net: Convolutional Networks For Biomedical Image Segmentation

Abstract: There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .


Huang & Zhang #REFR generated the step using a generator network. These methods properly solved score-based attack using RGF.

While the attacker cannot know the architecture or the gradient of the victim model (black-box attack), the perturbation cannot be updated through gradient-based optimization.

Some researchers solved this problem by using the RGF algorithm proposed by Nesterov #OTHEREFR .

Most RGFbased adversarial attack aims to find ways to find good random steps. M.Chenget al.

#OTHEREFR proposed binary searchbased RGF optimization to build a perturbation efficiently. S.Chenget al.

#OTHEREFR sampled random steps on a hypersphere and adopted the momentum optimization method proposed by Polyak #OTHEREFR .

So far, RGF is not appropriate for decision-based attacks because it cannot evaluate the direction properly, especially for universal attacks.

Decision-Badge: Decision-Based Adversarial Batch Attack With Directional Gradient Estimation

Abstract: The vulnerability of deep neural networks to adversarial examples has led to the rise in the use of adversarial attacks. While various decision-based and universal attack methods have been proposed, none have attempted to create a decision-based universal adversarial attack. This research proposes Decision-BADGE, which uses random gradientfree optimization and batch attack to generate universal adversarial perturbations for decision-based attacks. Multiple adversarial examples are combined to optimize a single universal perturbation, and the accuracy metric is reformulated into a continuous Hamming distance form. The effectiveness of accuracy metric as a loss function is demonstrated and mathematically proven. The combination of Decision-BADGE and the accuracy loss function performs better than both score-based image-dependent attack and white-box universal attack methods in terms of attack time efficiency. The research also shows that Decision-BADGE can successfully deceive unseen victims and accurately target specific classes. The code is available at our GitHub 1 repository.

Black-Box Adversarial Attack With Transferable Model-Based Embedding

Abstract: We present a new method for black-box adversarial attack. Unlike previous methods that combined transfer-based and scored-based methods by using the gradient or initialization of a surrogate white-box model, this new method tries to learn a low-dimensional embedding using a pretrained model, and then performs efficient search within the embedding space to attack an unknown target network. The method produces adversarial perturbations with high level semantic patterns that are easily transferable. We show that this approach can greatly improve the query efficiency of black-box adversarial attack across different target network architectures. We evaluate our approach on MNIST, ImageNet and Google Cloud Vision API, resulting in a significant reduction on the number of queries. We also attack adversarially defended networks on CIFAR10 and ImageNet, where our method not only reduces the number of queries, but also improves the attack success rate.


Rather, it is desirable to design sparse attention algorithms that mimic full self attention's instancedependent nature #REFR , thereby capturing the variable distribution of semantic information in the input image content.

For example, local and strided fixed attention patterns are used #OTHEREFR , in combination with randomlyinitialized connectivities #OTHEREFR .

However, such fixed attention patterns limit the the capacity of the self-attention connections to a fixed subset of tokens (Fig. 1) .

These attention patterns' fixed nature is less effective compared with the direct communication between tokens in full self attention.

For example, Swin transformer #OTHEREFR has a limited receptive field at shallower layers and needs many layers to model long-range dependencies.

And BigBird #OTHEREFR needs to combine multiple fixed attention patterns to achieve good performance.

To address aforementioned challenges, we propose a method called Sparsifiner that learns to compute sparse connectivity patterns over attention that are both instancedependent and unstructured.

The instance-dependent nature of the attention pattern allows each token to use its limited attention budget of nonzero elements more efficiently compared to fixed sparse attention patterns.

For example, in attention heads that attend to semantic rather than positional content #OTHEREFR , tokens containing similar semantic information should be considered to have high connectivity scores despite their spatial distance.

Similarly, nearby tokens with irrelevant semantic relation should have lower connectivity scores despite their spatial proximity.

Furthermore, Sparsifiner improves attention pattern flexibility compared to token pruning by pruning individual connectivities, instead of entire rows and columns of the attention matrix.

Sparsifiner: Learning Sparse Instance-Dependent Attention For Efficient Vision Transformers

Abstract: Vision Transformers (ViT) have shown their competitive advantages performance-wise compared to convolutional neural networks (CNNs) though they often come with high computational costs. To this end, previous methods explore different attention patterns by limiting a fixed number of spatially nearby tokens to accelerate the ViT's multi-head selfattention (MHSA) operations. However, such structured attention patterns limit the token-to-token connections to their spatial relevance, which disregards learned semantic connections from a full attention mask. In this work, we propose a novel approach to learn instance-dependent attention patterns, by devising a lightweight connectivity predictor module to estimate the connectivity score of each pair of tokens. Intuitively, two tokens have high connectivity scores if the features are considered relevant either spatially or semantically. As each token only attends to a small number of other tokens, the binarized connectivity masks are often very sparse by nature and therefore provide the opportunity to accelerate the network via sparse computations. Equipped with the learned unstructured attention pattern, sparse attention ViT (Sparsifiner) produces a superior Pareto-optimal trade-off between FLOPs and top-1 accuracy on ImageNet compared to token sparsity. Our method reduces 48% ∼ 69% FLOPs of MHSA while the accuracy drop is within 0.4%. We also show that combining attention and token sparsity reduces ViT FLOPs by over 60%.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


This is because, unlike mobile/service robots, manipulator robots are expected to physically interact with their environment through precise and fine-grained actions, such as reaching for an object, forming stable grasp or contacts with it, and delivering it to the target location #REFR .

This aspect serves as a key differentiator between human-human and human-robot workers.

In recent years, researchers have explored the use of verbal communication for coordination between mobile robots and humans, as evidenced by studies such as #OTHEREFR .

One such application is in tour guiding, where verbal communication is primarily used to provide the necessary information to the human. Additionally, Zhou et al.

(2021) #OTHEREFR proposed a novel approach for training a reinforcement learning agent using natural language goal-conditioned adversarial learning algorithm and a variational goal generator to sample natural language goals from trajectories of given demonstrations.

Coordinating task execution between a manipulating robot and a human using natural language processing (NLP) is a arXiv:2304.02993v1 [cs.RO] 6 Apr 2023 more challenging task than using NLP for mobile/service robots.

This paper introduces one of the first frameworks for verbally commanding a cobot to perform pick-and-place tasks. Pick-and-place is a common yet straightforward task to automate.

The proposed framework comprises three key components: natural language processing, semantic segmentation, and autonomous grasping.

The autonomous grasping module generates grasping poses for a given scene, allowing a human operator to command the robot to execute a desired grasp.

The contribution of this paper is manifold: (i) We created a dictionary of the common tasks involved in robotic manipulation in our NLP component.

(ii) Our framework includes a translator-middleware capable of translating natural language instructions into low-level joint space or operational space robot commands.

Natural Language Robot Programming: Nlp Integrated With Autonomous Robotic Grasping

Abstract: In this paper, we present a grammar-based natural language framework for robot programming, specifically for pick-and-place tasks. Our approach uses a custom dictionary of action words, designed to store together words that share meaning, allowing for easy expansion of the vocabulary by adding more action words from a lexical database. We validate our Natural Language Robot Programming (NLRP) framework through simulation and real-world experimentation, using a Franka Panda robotic arm equipped with a calibrated camerain-hand and a microphone. Participants were asked to complete a pick-and-place task using verbal commands, which were converted into text using Google's Speech-to-Text API and processed through the NLRP framework to obtain joint space trajectories for the robot. Our results indicate that our approach has a high system usability score. The framework's dictionary can be easily extended without relying on transfer learning or large data sets. In the future, we plan to compare the presented framework with different approaches of humanassisted pick-and-place tasks via a comprehensive user study.

Safe Robotic Grasping: Minimum Impact-Force Grasp Selection

Abstract: This paper addresses the problem of selecting from a choice of possible grasps, so that impact forces will be minimised if a collision occurs while the robot is moving the grasped object along a post-grasp trajectory. Such considerations are important for safety in human-robot interaction, where even a certified"human-safe"(e.g. compliant) arm may become hazardous once it grasps and begins moving an object, which may have significant mass, sharp edges or other dangers. Additionally, minimising collision forces is critical to preserving the longevity of robots which operate in uncertain and hazardous environments, e.g. robots deployed for nuclear decommissioning, where removing a damaged robot from a contaminated zone for repairs may be extremely difficult and costly. Also, unwanted collisions between a robot and critical infrastructure (e.g. pipework) in such high-consequence environments can be disastrous. In this paper, we investigate how the safety of the post-grasp motion can be considered during the pre-grasp approach phase, so that the selected grasp is optimal in terms applying minimum impact forces if a collision occurs during a desired post-grasp manipulation. We build on the methods of augmented robot-object dynamics models and"effective mass"and propose a method for combining these concepts with modern grasp and trajectory planners, to enable the robot to achieve a grasp which maximises the safety of the post-grasp trajectory, by minimising potential collision forces. We demonstrate the effectiveness of our approach through several experiments with both simulated and real robots.


The Gap Between Tree-Form And Efo 1 Queries

The gap is filled by considering those two properties with one assumption, which is a common practice #REFR .

Property 1 (Negation among Existential Variables).

There is a negated atomic formula of the form ¬r(x 1 , x 2 ) or ¬r(x 1 , y) where x 1 , x 2 are existential variables and y is the free variable.

If we expand the query graph by using the free variable as the root, then the leaf node of the query graph can be an existential variable rather than a constant entity.

With assumption 1, any EFO 1 query with a simple acyclic query graph that does not have Property 1 and 2 is a TF query.

The proof of Theorem 3 is detailed in Appendix A.

On Existential First Order Queries Inference On Knowledge Graphs

Abstract: Reasoning on knowledge graphs is a challenging task because it utilizes observed information to predict the missing one. Specifically, answering first-order logic formulas is of particular interest because of its clear syntax and semantics. Recently, the query embedding method has been proposed which learns the embedding of a set of entities and treats logic operations as set operations. Though there has been much research following the same methodology, it lacks a systematic inspection from the standpoint of logic. In this paper, we characterize the scope of queries investigated previously and precisely identify the gap between it and the whole family of existential formulas. Moreover, we develop a new dataset containing ten new formulas and discuss the new challenges coming simultaneously. Finally, we propose a new search algorithm from fuzzy logic theory which is capable of solving new formulas and outperforming the previous methods in existing formulas. Preprint. Under review.

Canonical Tensor Decomposition For Knowledge Base Completion

Abstract: The problem of Knowledge Base Completion can be framed as a 3rd-order binary tensor completion problem. In this light, the Canonical Tensor Decomposition (CP) (Hitchcock, 1927) seems like a natural solution; however, current implementations of CP on standard Knowledge Base Completion benchmarks are lagging behind their competitors. In this work, we attempt to understand the limits of CP for knowledge base completion. First, we motivate and test a novel regularizer, based on tensor nuclear p-norms. Then, we present a refor-mulation of the problem that makes it invariant to arbitrary choices in the inclusion of predicates or their reciprocals in the dataset. These two methods combined allow us to beat the current state of the art on several datasets with a CP decomposition , and obtain even better results using the more advanced ComplEx model.


One solution is to follow a paradigm like DAGGER #REFR to keep exploring the environment and collect new states and labels to update our CBF critics and world model.

We apply our method to two simulated domains and show that our method outperforms existing classical and learningbased safe navigation approaches.

Furthermore, we show that ConBaT can be quickly adapted to new safety constraints from limited demonstrations.

This paradigm of learning safety constraints implicitly from minimal labels reduces training effort in creating safe agents and makes it easier to adapt to new definitions of safety.

Though empirically ConBaT can learn safer policies and quickly adapts to new safety concepts, there are some limitations.

Our method might not guarantee safety if ConBaT makes wrong CBF score predictions due to out-of-distribution data, or if the online action optimization falls into a local minimum.

The runtime for ConBaT in evaluation is also 0.2 ∼ 1.0x higher than other learning-based methods due to the back-propagation step in online optimization. We discuss different ConBaT architectures in Sec.

III-E to weigh the trade-off between the safety performance and the runtime.

Further runtime improvement can be made by faster back-propagation algorithms optimized for GPU hardware. Currently, our framework was only applied in simulated environments.

In the future, we wish to extend our approach to image-based control in 3D and toward more complex robotic platforms in simulation and the real world.

Conbat: Control Barrier Transformer For Safe Policy Learning

Abstract: Large-scale self-supervised models have recently revolutionized our ability to perform a variety of tasks within the vision and language domains. However, using such models for autonomous systems is challenging because of safety requirements: besides executing correct actions, an autonomous agent must also avoid the high cost and potentially fatal critical mistakes. Traditionally, self-supervised training mainly focuses on imitating previously observed behaviors, and the training demonstrations carry no notion of which behaviors should be explicitly avoided. In this work, we propose Control Barrier Transformer (ConBaT), an approach that learns safe behaviors from demonstrations in a self-supervised fashion. ConBaT is inspired by the concept of control barrier functions in control theory and uses a causal transformer that learns to predict safe robot actions autoregressively using a critic that requires minimal safety data labeling. During deployment, we employ a lightweight online optimization to find actions that ensure future states lie within the learned safe set. We apply our approach to different simulated control tasks and show that our method results in safer control policies compared to other classical and learning-based methods such as imitation learning, reinforcement learning, and model predictive control.

A Reduction Of Imitation Learning And Structured Prediction To No-Regret Online Learning

Abstract: Sequential prediction problems such as imitation learning, where future observations depend on previous predictions (actions), violate the common i.i.d. assumptions made in statistical learning. This leads to poor performance in theory and often in practice. Some recent approaches provide stronger guarantees in this setting, but remain somewhat unsatisfactory as they train either non-stationary or stochastic policies and require a large number of iterations. In this paper, we propose a new iterative algorithm, which trains a stationary deterministic policy, that can be seen as a no regret algorithm in an online learning setting. We show that any such no regret algorithm, combined with additional reduction assumptions, must find a policy with good performance under the distribution of observations it induces in such sequential settings. We demonstrate that this new approach outperforms previous approaches on two challenging imitation learning problems and a benchmark sequence labeling problem.


These experiments are conducted on CIFAR-10 with ResNet-18 #REFR and Dirichlet α = 0.6. The norm of local update.

In this subsection, we empirically discuss how SAM mitigates the negative impacts of DP from the aspects of the norm of local update and the visualization of loss landscape and contour.

Meanwhile, we also observe the training performance with SAM under different privacy budgets ϵ compared with all baselines.

To validate the theoretical results for mitigating the adverse impacts of the norm of local updates, we conduct experiments on DP-FedSAM and DP-FedAvg with clipping threshold C = 0.2 as shown in Figure  3 .

We show the norm ∆ t i distribution and average norm ∆ t of local updates before clipping during the communication rounds.

In contrast to DP-FedAvg, most of the norm is distributed at the smaller value in our scheme in Figure 3 (a) , which means that the clipping operation drops less information.

Meanwhile, the on-average norm ∆ t is smaller than DP-FedAvg as shown in Figure 3 (b) .

These observations are also consistent with our theoretical results in Section 5. Performance under different privacy budgets ϵ.

Make Landscape Flatter In Differentially Private Federated Learning

Abstract: To defend the inference attacks and mitigate the sensitive information leakages in Federated Learning (FL), clientlevel Differentially Private FL (DPFL) is the de-facto standard for privacy protection by clipping local updates and adding random noise. However, existing DPFL methods tend to make a sharper loss landscape and have poorer weight perturbation robustness, resulting in severe performance degradation. To alleviate these issues, we propose a novel DPFL algorithm named DP-FedSAM, which leverages gradient perturbation to mitigate the negative impact of DP. Specifically, DP-FedSAM integrates Sharpness Aware Minimization (SAM) optimizer to generate local flatness models with better stability and weight perturbation robustness, which results in the small norm of local updates and robustness to DP noise, thereby improving the performance. From the theoretical perspective, we analyze in detail how DP-FedSAM mitigates the performance degradation induced by DP. Meanwhile, we give rigorous privacy guarantees with Rényi DP and present the sensitivity analysis of local updates. At last, we empirically confirm that our algorithm achieves state-of-the-art (SOTA) performance compared with existing SOTA baselines in DPFL.

Deep Residual Learning For Image Recognition

Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.


How Do Aleatoric And Epistemic Uncertainty Dif-

This is not the case for models with Dropout as an epistemic component, confirming the findings by #REFR .

What does this tell us? We see that the aleatoric and epistemic components of the LSN clearly capture different aspects of the model's uncertainty.

To better understand where this difference comes from, Figure 5 shows epistemic and aleatoric variance maps of sigmoids for a given ISIC test set sample.

While the epistemic component assigns low variance to this in-distribution sample, the aleatoric component expresses the variation of plausible predictions for the image.

We find this behavior attractive: In the Bayesian framework, the aleatoric component should play the role of the predictor, as it imitates the distribution of annotations Figure 4 : Precision-Recall plots for the ISIC test dataset show that aleatoric and epistemic variance in the LSN models (first row) are not the same, opposed to the SSN + Dropout models.

The better performance of the aleatoric component (higher AUC) indicates that the model learns to separate data variation from model uncertainty.

Laplacian Segmentation Networks: Improved Epistemic Uncertainty From Spatial Aleatoric Uncertainty

Abstract: Out of distribution (OOD) medical images are frequently encountered, e.g. because of site-or scanner differences, or image corruption. OOD images come with a risk of incorrect image segmentation, potentially negatively affecting downstream diagnoses or treatment. To ensure robustness to such incorrect segmentations, we propose Laplacian Segmentation Networks (LSN) that jointly model epistemic (model) and aleatoric (data) uncertainty in image segmentation. We capture data uncertainty with a spatially correlated logit distribution. For model uncertainty, we propose the first Laplace approximation of the weight posterior that scales to large neural networks with skip connections that have high-dimensional outputs. Empirically, we demonstrate that modelling spatial pixel correlation allows the Laplacian Segmentation Network to successfully assign high epistemic uncertainty to out-of-distribution objects appearing within images.

What Uncertainties Do We Need In Bayesian Deep Learning For Computer Vision?

Abstract: There are two major types of uncertainty one can model. Aleatoric uncertainty captures noise inherent in the observations. On the other hand, epistemic uncertainty accounts for uncertainty in the model - uncertainty which can be explained away given enough data. Traditionally it has been difficult to model epistemic uncertainty in computer vision, but with new Bayesian deep learning tools this is now possible. We study the benefits of modeling epistemic vs. aleatoric uncertainty in Bayesian deep learning models for vision tasks. For this we present a Bayesian deep learning framework combining input-dependent aleatoric uncertainty together with epistemic uncertainty. We study models under the framework with per-pixel semantic segmentation and depth regression tasks. Further, our explicit uncertainty formulation leads to new loss functions for these tasks, which can be interpreted as learned attenuation. This makes the loss more robust to noisy data, also giving new state-of-the-art results on segmentation and depth regression benchmarks.


Multi-agent reinforcement learning algorithms have contributed to many successes in machine learning, including games such as chess and Go Silver et al. (2016) ; #REFR ; Silver et al.

(2017b,a) , automated warehouses, robotic arms with multiple arms #OTHEREFR , and autonomous traffic control #OTHEREFR ; #OTHEREFR ; see #OTHEREFR ; ; for surveys.

Multi-agent RL can refer to one of many scenarios: (i) where a team of agents work towards a common goal with all the agents have the same information or not #OTHEREFR

(2022) , (ii) non-cooperative games where there are multiple agents with their own objectives #OTHEREFR

(2019) , and (iii) zero-sum games where there are only two players with opposing objectives.

We can further categorize problems as infinite-horizon, discounted reward/cost, finite-horizon, simultaneous move or turn-based games.

A New Policy Iteration Algorithm For Reinforcement Learning In Zero-Sum Markov Games

Abstract: Many model-based reinforcement learning (RL) algorithms can be viewed as having two phases that are iteratively implemented: a learning phase where the model is approximately learned and a planning phase where the learned model is used to derive a policy. In the case of standard MDPs, the learning problem can be solved using either value iteration or policy iteration. However, in the case of zero-sum Markov games, there is no efficient policy iteration algorithm; e.g., it has been shown in Hansen et al. (2013) that one has to solve Ω(1/(1 − α)) MDPs to implement the only known convergent version of policy iteration. Another algorithm for Markov zero-sum games, called naive policy iteration, is easy to implement but is only provably convergent under very restrictive assumptions. Prior attempts to fix naive policy iteration algorithm have several limitations. Here, we show that a simple variant of naive policy iteration for games converges, and converges exponentially fast. The only addition we propose to naive policy iteration is the use of lookahead in the policy improvement phase. This is appealing because lookahead is anyway often used in RL for games. We further show that lookahead can be implemented efficiently in linear Markov games, which are the counterpart of the linear MDPs and have been the subject of much attention recently. We then consider multi-agent reinforcement learning which uses our algorithm in the planning phases, and provide sample and time complexity bounds for such an algorithm.

Asynchronous Methods For Deep Reinforcement Learning

Abstract: We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.


For the federated training of the one-layer 1D CNN after round 1 (line 8), the protocol follows the standard FL algorithm, and we use FedAvg #REFR .

The global federated XGBoost model consists of all clients' locally trained XGBoost tree ensembles and the globally trained one-layer 1D CNN. The detailed procedure is shown in Algorithm. 1.

At round 0 (line 1 to 7), each client first trains its local XGBoost tree ensemble.

The server then conducts tree ensemble aggregation and CNN initialization.

After receiving the aggregated tree ensemble, all clients calculate the prediction outcomes given the aggregated tree ensemble on their local data samples. The calculated prediction outcomes are inputs of the CNN.

It is worth noticing that the clients only build XGBoost models at round 0, and the aggregated tree ensemble is fixed after round 0.

In FedXGBllr, the number of communication rounds is equal to the FL training rounds ( ) because we send the trees (at round 0) and CNN's model parameters (after round 1).

Gradient-Less Federated Gradient Boosting Trees With Learnable Learning Rates

Abstract: The privacy-sensitive nature of decentralized datasets and the robustness of eXtreme Gradient Boosting (XGBoost) on tabular data raise the needs to train XGBoost in the context of federated learning (FL). Existing works on federated XGBoost in the horizontal setting rely on the sharing of gradients, which induce per-node level communication frequency and serious privacy concerns. To alleviate these problems, we develop an innovative framework for horizontal federated XGBoost which does not depend on the sharing of gradients and simultaneously boosts privacy and communication efficiency by making the learning rates of the aggregated tree ensembles learnable. We conduct extensive evaluations on various classification and regression datasets, showing our approach achieves performance comparable to the stateof-the-art method and effectively improves communication efficiency by lowering both communication rounds and communication overhead by factors ranging from 25x to 700x.

Communication-Efficient Learning Of Deep Networks From Decentralized Data

Abstract: Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning. We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.


In this work, we aim to develop a strong system to detect arbitrary objects specified by human language inputs, which we name as open-set object detection #REFR .

Understanding novel concepts is a fundamental capability of visual intelligence.

The task has wide applications for its great potential as a generic object detector.

For example, we can cooperate it with generative models for image editing (as shown in Fig. 1 (b) ).

The key to open-set detection is introducing language for unseen object generalization #OTHEREFR .

For example, GLIP #OTHEREFR reformulates object detection as a phrase grounding task and introduces contrastive training between object regions and language phrases.

It shows a great flexibility for heterogeneous datasets and remarkable performance on both closed-set and open-set detection.

Grounding Dino: Marrying Dino With Grounded Pre-Training For Open-Set Object Detection

Abstract: Standard Object Detection COCO pre-defined categories Zero-Shot Transfer to Novel Categories worldcup Human-input novel categories ear, lion, bench The left lion The bottom man with his head up Referring Object Detection (Referring Expression Comprehension) Human-input reference sentences bench person (c) Application: Image Editing Collaborate with stable diffusion. Prompt (modify background): All people around the world cheer with a worldcup.

Bottom-Up And Top-Down Attention For Image Captioning And Visual Question Answering

Abstract: Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and top-down attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.

open-set object detection
human language inputs


We picked this name for comparison because of two reasons; 1) the number of authors sharing this name is among the top five as shown in Table 3 and 2) All methods cited in #REFR could not achieve a good result.

For example, 1) two authors with the same common name authoring a single publication.

2) more than one author with the same common atomic name variate authoring a single publication, 3) number of authors with the same full name, 4) the uncertainty of the accuracy of the dataset, etc.

Although the comparison is difficult and cannot be completely fair, we compare WhoIs to other state-of-the-art approaches, whose results are reported in #OTHEREFR .

These results are obtained on a collection from CiteSeerX 16 that contains records of authors with the name / atomic name variate 'Y Chen'.

This collection consists of 848 complete documents authored by 71 distinct authors.

We applied WhoIs on this collection by randomly splitting the records into 70% for training, 15% for validation and 15% for testing. The results are shown in Table 5 .

Note that in our collection, we consider way more records and distinct authors (see Table 3 ) and we use only reference attributes (i.e. co-authors, title and source).

As the results presented in Table 5 show, WhoIs outperforms other methods in resolving the disambiguation of the author name 'Y Chen' on the CiteSeerX dataset, which is a relatively small dataset and does not really reflect the performance of all presented approaches in real scenarios.

The disparity between the results shown in Table 4 and Table 5 demonstrates that the existing benchmark datasets are manually prepared for the sake of accuracy.

However, this leads to covering a very small portion of records whose authors share similar Table 5 Comparison between WhoIs and other baseline methods on CiteSeerX dataset in terms of Macro F1 score as reported in #OTHEREFR .

Deep Author Name Disambiguation Using Dblp Data

Abstract: In the academic world, the number of scientists grows every year and so does the number of authors sharing the same names. Consequently, it challenging to assign newly published papers to their respective authors. Therefore, Author Name Ambiguity (ANA) is considered a critical open problem in digital libraries. This paper proposes an Author Name Disambiguation (AND) approach that links author names to their real-world entities by leveraging their co-authors and domain of research. To this end, we use data collected from the DBLP repository that contains more than 5 million bibliographic records authored by around 2.6 million co-authors. Our approach first groups authors who share the same last names and same first name initials. The author within each group is identified by capturing the relation with his/her co-authors and area of research, represented by the titles of the validated publications of the corresponding author. To this end, we train a neural network model that learns from the representations of the co-authors and titles. We validated the effectiveness of our approach by conducting extensive experiments on a large dataset.

Name Disambiguation In Anonymized Graphs Using Network Embedding

Abstract: In real-world, our DNA is unique but many people share names. This phenomenon often causes erroneous aggregation of documents of multiple persons who are namesake of one another. Such mistakes deteriorate the performance of document retrieval, web search, and more seriously, cause improper attribution of credit or blame in digital forensic. To resolve this issue, the name disambiguation task is designed which aims to partition the documents associated with a name reference such that each partition contains documents pertaining to a unique real-life person. Existing solutions to this task substantially rely on feature engineering, such as biographical feature extraction, or construction of auxiliary features from Wikipedia. However, for many scenarios, such features may be costly to obtain or unavailable due to the risk of privacy violation. In this work, we propose a novel name disambiguation method. Our proposed method is non-intrusive of privacy because instead of using attributes pertaining to a real-life person, our method leverages only relational data in the form of anonymized graphs. In the methodological aspect, the proposed method uses a novel representation learning model to embed each document in a low dimensional vector space where name disambiguation can be solved by a hierarchical agglomerative clustering algorithm. Our experimental results demonstrate that the proposed method is significantly better than the existing name disambiguation methods working in a similar setting.


C. Enhancing Personalized Ranking Via Gauc Optimization

In practice, we find that the GAUC #REFR metric is more consistent with the online effect.

The above section details how to construct paired samples within a batch, which do not meet the requirement for personalized recommendation.

Consequently, it is a natural idea to add the GAUC metric to the objective when optimizing the model.

Considering the original calculation of the GAUC metric, the samples are firstly divided into several groups.

In this context, the groups are partitioned by the user ID.

Then the AUC metric is calculated within each group respectively, and the GAUC metric is derived by averaging the AUC metrics from all groups with weight.

The weight is optionally proportional to the impression or click times, here we set the weight to be 1 for all users.

Enhancing Personalized Ranking With Differentiable Group Auc Optimization

Abstract: AUC is a common metric for evaluating the performance of a classifier. However, most classifiers are trained with cross entropy, and it does not optimize the AUC metric directly, which leaves a gap between the training and evaluation stage. In this paper, we propose the PDAOM loss, a Personalized and Differentiable AUC Optimization method with Maximum violation, which can be directly applied when training a binary classifier and optimized with gradient-based methods. Specifically, we construct the pairwise exponential loss with difficult pair of positive and negative samples within sub-batches grouped by user ID, aiming to guide the classifier to pay attention to the relation between hard-distinguished pairs of opposite samples from the perspective of independent users. Compared to the origin form of pairwise exponential loss, the proposed PDAOM loss not only improves the AUC and GAUC metrics in the offline evaluation, but also reduces the computation complexity of the training objective. Furthermore, online evaluation of the PDAOM loss on the 'Guess What You Like' feed recommendation application in Meituan manifests 1.40% increase in click count and 0.65% increase in order count compared to the baseline model, which is a significant improvement in this well-developed online life service recommendation system.

Optimized Cost Per Click In Taobao Display Advertising

Abstract: Taobao, as the largest online retail platform in the world, provides billions of online display advertising impressions for millions of advertisers every day. For commercial purposes, the advertisers bid for specific spots and target crowds to compete for business traffic. The platform chooses the most suitable ads to display in tens of milliseconds. Common pricing methods include cost per mille (CPM) and cost per click (CPC). Traditional advertising systems target certain traits of users and ad placements with fixed bids, essentially regarded as coarse-grained matching of bid and traffic quality. However, the fixed bids set by the advertisers competing for different quality requests cannot fully optimize the advertisers' key requirements. Moreover, the platform has to be responsible for the business revenue and user experience. Thus, we proposed a bid optimizing strategy called optimized cost per click (OCPC) which automatically adjusts the bid to achieve finer matching of bid and traffic quality of page view (PV) request granularity. Our approach optimizes advertisers' demands, platform business revenue and user experience and as a whole improves traffic allocation efficiency. We have validated our approach in Taobao display advertising system in production. The online A/B test shows our algorithm yields substantially better results than previous fixed bid manner.


Compared with state-of-the-art multi-step PGD-AT #REFR , the proposed FGSM-PGK can obtain better adversarial robustness under all attack scenarios with less training cost.

The comparison experiment results on CIFAR-10 are shown in Table 2 .

It can be observed that the proposed FGSM-PGK not only prevents catastrophic overfitting but also achieves excellent adversarial robustness performance improvement.

In particular, under the strongest attack method (AA), PGD-AT achieves the performance of about 48.7%, but the proposed FGSM-PGK achieves the performance of about 49.5%, which indicates that single-step adversarial training with priorguided knowledge has the potential to outperform multistep adversarial training.

More importantly, our FGSM-PGK is about 3.6 times faster than PGD-AT.

Compared with previous fast adversarial training methods, our FGSM-PGK obtains the best adversarial robustness under all attack scenarios on the best and last checkpoints.

For example, under the PGD-50 attack, the most robust one among the previous fast adversarial training methods achieves an accuracy of about 53%, but our FGSM-PGK achieves an accuracy of about 55%.

In terms of training efficiency, our proposed FGSM-PGK only takes 1.2 hours to train, which is faster than previous fast adversarial training methods.

Improving Fast Adversarial Training With Prior-Guided Knowledge

Abstract: Fast adversarial training (FAT) is an efficient method to improve robustness. However, the original FAT suffers from catastrophic overfitting, which dramatically and suddenly reduces robustness after a few training epochs. Although various FAT variants have been proposed to prevent overfitting, they require high training costs. In this paper, we investigate the relationship between adversarial example quality and catastrophic overfitting by comparing the training processes of standard adversarial training and FAT. We find that catastrophic overfitting occurs when the attack success rate of adversarial examples becomes worse. Based on this observation, we propose a positive prior-guided adversarial initialization to prevent overfitting by improving adversarial example quality without extra training costs. This initialization is generated by using high-quality adversarial perturbations from the historical training process. We provide theoretical analysis for the proposed initialization and propose a prior-guided regularization method that boosts the smoothness of the loss function. Additionally, we design a prior-guided ensemble FAT method that averages the different model weights of historical models using different decay rates. Our proposed method, called FGSM-PGK, assembles the prior-guided knowledge, i.e., the prior-guided initialization and model weights, acquired during the historical training process. Evaluations of four datasets demonstrate the superiority of the proposed method.

Overfitting In Adversarially Robust Deep Learning

Abstract: It is common practice in deep learning to use overparameterized networks and train for as long as possible; there are numerous studies that show, both theoretically and empirically, that such practices surprisingly do not unduly harm the generalization performance of the classifier. In this paper, we empirically study this phenomenon in the setting of adversarially trained deep networks, which are trained to minimize the loss under worst-case adversarial perturbations. We find that overfitting to the training set does in fact harm robust performance to a very large degree in adversarially robust training across multiple datasets (SVHN, CIFAR-10, CIFAR-100, and ImageNet) and perturbation models ($\ell_\infty$ and $\ell_2$). Based upon this observed effect, we show that the performance gains of virtually all recent algorithmic improvements upon adversarial training can be matched by simply using early stopping. We also show that effects such as the double descent curve do still occur in adversarially trained models, yet fail to explain the observed overfitting. Finally, we study several classical and modern deep learning remedies for overfitting, including regularization and data augmentation, and find that no approach in isolation improves significantly upon the gains achieved by early stopping. All code for reproducing the experiments as well as pretrained model weights and training logs can be found at this https URL.


We refer the reader to #REFR for a more formal treatment of these games, which we omit for brevity.

We say the algorithm is a regret minimizer, if the external regret

ext = max ∈ {1,..., } =1 ( , ) grows sublinearly in ,

We will use the online algorithm in two-player zero-sum matrix games and sequential imperfect-information games.

We employ the online algorithms at each state a player ∈ {1, 2} can't distinguish (i.e.

at each infostate), as in prior work on counterfactual regret minimization #OTHEREFR .

In a sequential game, the players take turns making decisions at each infostate, until they arrive at a terminal state where the player receives its reward ( ) ∈ R.

We denote the maximum difference in rewards as Δ max = max ( )−min ( ). The expected reward (in the whole game) is

Learning Not To Regret

Abstract: Regret minimization is a key component of many algorithms for finding Nash equilibria in imperfect-information games. To scale to games that cannot fit in memory, we can use search with value functions. However, calling the value functions repeatedly in search can be expensive. Therefore, it is desirable to minimize regret in the search tree as fast as possible. We propose to accelerate the regret minimization by introducing a general "learning not to regret" framework, where we meta-learn the regret minimizer. The resulting algorithm is guaranteed to minimize regret in arbitrary settings and is (meta)-learned to converge fast on a selected distribution of games. Our experiments show that meta-learned algorithms converge substantially faster than prior regret minimization algorithms.

Rethinking Formal Models Of Partially Observable Multiagent Decision Making

Abstract: Multiagent decision-making problems in partially observable environments are usually modeled as either extensive-form games (EFGs) within the game theory community or partially observable stochastic games (POSGs) within the reinforcement learning community. While most practical problems can be modeled in both formalisms, the communities using these models are mostly distinct with little sharing of ideas or advances. The last decade has seen dramatic progress in algorithms for EFGs, mainly driven by the challenge problem of poker. We have seen computational techniques achieving super-human performance, some variants of poker are essentially solved, and there are now sound local search algorithms which were previously thought impossible. While the advances have garnered attention, the fundamental advances are not yet understood outside the EFG community. This can be largely explained by the starkly different formalisms between the game theory and reinforcement learning communities and, further, by the unsuitability of the original EFG formalism to make the ideas simple and clear. This paper aims to address these hindrances, by advocating a new unifying formalism, a variant of POSGs, which we call Factored-Observation Games (FOGs). We prove that any timeable perfect-recall EFG can be efficiently modeled as a FOG as well as relating FOGs to other existing formalisms. Additionally, a FOG explicitly identifies the public and private components of observations, which is fundamental to the recent EFG breakthroughs. We conclude by presenting the two building-blocks of these breakthroughs --- counterfactual regret minimization and public state decomposition --- in the new formalism, illustrating our goal of a simpler path for sharing recent advances between game theory and reinforcement learning community.


Given aggregated features at different pixels in the same sample set, we apply another naïve Transformer #REFR to perform non-local interactions.

To address these issues, we draw inspiration from recent Transformers on 3-D points #OTHEREFR and apply a Transformer on a fixed number (m) of pixel samples (e.g., m = 2048) from random locations in the input coordinate system. We term this the pixel-sampling Transformer.

Unlike image-based approaches, pixel-sampling Transformer's memory consumption is constant per sample set, scaling to arbitrary image sizes.

Moreover, by applying the Transformer to a randomly sampled set of locations, local interactions that may lead to over-smoothing of feature maps (e.g., in CNNs) are almost entirely eliminated.

Concretely, given m random pixels x i=1,...,m from the masked region of the input coordinate system, we interpolate features at those pixels as F 1,...,K (S(x i )), where S is the bilinear interpolation operator.

Then, interpolated features are concatenated with corresponding raw observations I 1,...,K (x i ) and aggregated to A(x i ) with pooling by multihead attention (PMA) #OTHEREFR , as in #OTHEREFR .

Since the goal of this process is to consider surface-level interactions based on physical attributes, pixel coordinate information is unnecessary.

Thus, we don't apply position embeddings to samples, unlike most existing visual Transformer models (e.g.

#OTHEREFR ), allowing the samples to propagate their aggregated features without location information.

After the non-local interaction, we apply a two-layer MLP to predict surface normals at sampled locations.

Finally, surface normals for each set are merged to obtain the final surface normal map at the input image resolution.

Scalable, Detailed And Mask-Free Universal Photometric Stereo

Abstract: Input image (1st out of 9) UniPS [Ikehata2022] 3D Scanner (EinScan-SE) INPUT: Multiple images under varying unknown lighting conditions 1st 2nd 23rd Figure 1. Given multiple images under unknown spatially-varying illuminations, our method can recover the detailed surface normal map of non-convex, non-Lambertian surfaces (Left). Our method even surpasses the level of detail provided by consumer 3-D scanners (Right).

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

aggregated features
another naïve Transformer


In practice, as we do in this paper, we can use sliced Wasserstein distance #REFR to compute Equation 14, avoiding the min-max formulation by directly approximating the Wasserstein distance.

Thus, if θ φ are the parameters of φ, we can re-write the IPM loss in Equation 14as follows

This gives us the following min − max algorithm to solve Equation (6) θNF = arg min

where the loss function is given by Equation (8).

We now discuss our framework in the general case when source and target spaces have different dimensions or are high dimensional.

In such a case, we can project the two data spaces on to a latent space of the same dimension and consider the diffeomorphic pushforward between the latent spaces in exactly the same fashion we have done so far.

Let Ω s ⊂ R ds and Ω t ⊂ R dt . Consider the latent spaces

Ω L,s , Ω L,t ⊂ R d where d ≤ min(d s , d t ).

Associated with each latent space is a pair of encoder and decoder maps, that is enc s : Ω s → Ω L,s and enc t : Ω t → Ω L,t respectively (with the corresponding decoder maps).

Semi-Supervised Learning Of Pushforwards For Domain Translation & Adaptation *

Abstract: Given two probability densities on related data spaces, we seek a map pushing one density to the other while satisfying application-dependent constraints. For maps to have utility in a broad application space (including domain translation, domain adaptation, and generative modeling), the map must be available to apply on out-of-sample data points and should correspond to a probabilistic model over the two spaces. Unfortunately, existing approaches, which are primarily based on optimal transport, do not address these needs. In this paper, we introduce a novel pushforward map learning algorithm that utilizes normalizing flows to parameterize the map. We first re-formulate the classical optimal transport problem to be map-focused and propose a learning algorithm to select from all possible maps under the constraint that the map minimizes a probability distance and application-specific regularizers; thus, our method can be seen as solving a modified optimal transport problem. Once the map is learned, it can be used to map samples from a source domain to a target domain. In addition, because the map is parameterized as a composition of normalizing flows, it models the empirical distributions over the two data spaces and allows both sampling and likelihood evaluation for both data sets. We compare our method (parOT) to related optimal transport approaches in the context of domain adaptation and domain translation on benchmark data sets. Finally, to illustrate the impact of our work on applied problems, we apply parOT to a real scientific application: spectral calibration for high-dimensional measurements from two vastly different environments.

Generative Modeling Using The Sliced Wasserstein Distance

Abstract: Generative Adversarial Nets (GANs) are very successful at modeling distributions from given samples, even in the high-dimensional case. However, their formulation is also known to be hard to optimize and often not stable. While this is particularly true for early GAN formulations, there has been significant empirically motivated and theoretically founded progress to improve stability, for instance, by using the Wasserstein distance rather than the Jenson-Shannon divergence. Here, we consider an alternative formulation for generative modeling based on random projections which, in its simplest form, results in a single objective rather than a saddle-point formulation. By augmenting this approach with a discriminator we improve its accuracy. We found our approach to be significantly more stable compared to even the improved Wasserstein GAN. Further, unlike the traditional GAN loss, the loss formulated in our method is a good measure of the actual distance between the distributions and, for the first time for GAN training, we are able to show estimates for the same.

Wasserstein distance
sliced Wasserstein distance


After that, the compression is done using the knowledge distillation approach #REFR and soft labels; thus, the network reshapes to its original architecture.

In this method, for expanding the network, an LSTM network is used via reinforcement learning and policy gradient to predict the optimal number of nodes that should be added to each layer, with respect to both the detection rate and size of the network.

Although the approaches mentioned above try to expand the network as efficiently as possible, the network's size will still grow after each task, which is considered a drawback in the long run.

An approach for fully compressing the network after the expansion is proposed in regularize, expand, and compress (REC) framework #OTHEREFR .

Similar to RCL, REC exploits reinforcement learning (AutoML #OTHEREFR ) to expand the network.

The whole network is trained on the new task with regularizations based on multi-task learning and the Fisher information matrix.

A Multi-Agent Adaptive Deep Learning Framework For Online Intrusion Detection

Abstract: The network security analyzers use intrusion detection systems (IDSes) to distinguish malicious traffic from benign ones. The deep learning-based (DL-based) IDSes are proposed to auto-extract high-level features and eliminate the time-consuming and costly signature extraction process. However, this new generation of IDSes still suffers from a number of challenges. One of the main issues of an IDS is facing traffic concept drift which manifests itself as new (i.e., zero-day) attacks, in addition to the changing behavior of benign users/applications. Furthermore, a practical DL-based IDS needs to be conformed to a distributed architecture to handle big data challenges. In this paper, we propose a framework for adapting DL-based models to the changing attack/benign traffic behaviors, considering a more practical scenario (i.e., online adaptable IDSes). This framework employs continual deep anomaly detectors in addition to the federated learning approach to solve the above-mentioned challenges. Furthermore, the proposed framework implements sequential packet labeling for each flow, which provides an attack probability score for the flow by gradually observing each flow packet and updating its estimation. We evaluate the proposed framework by employing different deep models (including CNN-based and LSTM-based) over the CIC-IDS2017 and CSE-CIC-IDS2018 datasets. Through extensive evaluations and experiments, we show that the proposed distributed framework is well adapted to the traffic concept drift. More precisely, our results indicate that the CNN-based models are well suited for continually adapting to the traffic concept drift (i.e., achieving an average detection rate of above 95% while needing just 128 new flows for the updating phase), and the LSTM-based models are a good candidate for sequential packet labeling in practical online IDSes (i.e., detecting intrusions by just observing their first 15 packets).

Distilling The Knowledge In A Neural Network

Abstract: A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.


ContactDB #REFR ) is a collection of grasping and manipulation data for applications in human-robot interaction.

The data were collected by thermally imaging 3D printed household objects after they were grasped by human subjects.

We study 8 objects, each handled by a single participant.

The data take the form of triangle meshes with a single feature (intensity) at each vertex, indicating thermal energy transferred to the object.

As in #OTHEREFR , we process the contact maps by passing the intensities through a sigmoid function, assigning 0.999 probability to the maximum intensity and 0.001 to the minimum.

As in #OTHEREFR , we average the corresponding probabilities on each face to obtain an unnormalized probability distribution over faces.

Conformal Generative Modeling On Triangulated Surfaces

Abstract: We propose conformal generative modeling, a framework for generative modeling on 2D surfaces approximated by discrete triangle meshes. Our approach leverages advances in discrete conformal geometry to develop a map from a source triangle mesh to a target triangle mesh of a simple manifold such as a sphere. After accounting for errors due to the mesh discretization, we can use any generative modeling approach developed for simple manifolds as a plug-and-play subroutine. We demonstrate our framework on multiple complicated manifolds and multiple generative modeling subroutines, where we show that our approach can learn good estimates of distributions on meshes from samples, and can also learn simultaneously from multiple distinct meshes of the same underlying manifold.

Contactdb: Analyzing And Predicting Grasp Contact Via Thermal Imaging

Abstract: Grasping and manipulating objects is an important human skill. Since hand-object contact is fundamental to grasping, capturing it can lead to important insights. However, observing contact through external sensors is challenging because of occlusion and the complexity of the human hand. We present ContactDB, a novel dataset of contact maps for household objects that captures the rich hand-object contact that occurs during grasping, enabled by use of a thermal camera. Participants in our study grasped 3D printed objects with a post-grasp functional intent. ContactDB includes 3750 3D meshes of 50 household objects textured with contact maps and 375K frames of synchronized RGB-D+thermal images. To the best of our knowledge, this is the first large-scale dataset that records detailed contact maps for human grasps. Analysis of this data shows the influence of functional intent and object size on grasping, the tendency to touch/avoid ‘active areas’, and the high frequency of palm and proximal finger contact. Finally, we train state-of-the art image translation and 3D convolution algorithms to predict diverse contact patterns from object shape. Data, code and models are available at https://contactdb.cc.gatech.edu.


LOLA do not achieve significant results in MH, and similar results also appear in #REFR .

4 is the evaluation results of ARSP and other baselines in MH.

Global rewards are the summation of all agents' returns in the environment.

In MH, ARSP agents can rapidly discover the high payoff NE where two agents stay together and wait for the monster.

Notably, ARSP agents can stably converge to the coordination strategy in less than 25k time steps, which shows the superiority of ARSP in sample efficiency.

However, most other baseline agents can only converge to some guaranteed but lower payoff NEs, i.e., avoid stag and eat apples alone.

We suspect that LOLA can not scale to high dimensional settings.

R-IQN performs better than other baselines in MH and the second row of Fig 6 shows that R-IQN agents can achieve one or two co-operations in one episode, demonstrating that the risk-seeking policy plays an important role in discovering coordination strategies.

Similar results are also achieved in the Escalation environment, where ARSP significantly outperforms other baselines both in asymptotic performance and sample efficiency.

Although with the increase of cooperation times, the punishment suffered by agents for betrayal is also increasing in the Escalation, agents can still achieve stable cooperation shown in

Learning Adaptable Risk-Sensitive Policies To Coordinate In Multi-Agent General-Sum Games

Abstract: In general-sum games, the interaction of selfinterested learning agents commonly leads to socially worse outcomes, such as defect-defect in the iterated stag hunt (ISH). Previous works address this challenge by sharing rewards or shaping their opponents' learning process, which require too strong assumptions. In this paper, we demonstrate that agents trained to optimize expected returns are more likely to choose a safe action that leads to guaranteed but lower rewards. However, there typically exists a risky action that leads to higher rewards in the long run only if agents cooperate, e.g., cooperate-cooperate in ISH. To overcome this, we propose using action value distribution to characterize the decision's risk and corresponding potential payoffs. Specifically, we present Adaptable Risk-Sensitive Policy (ARSP). ARSP learns the distributions over agent's return and estimates a dynamic risk-seeking bonus to discover risky coordination strategies. Furthermore, to avoid overfitting training opponents, ARSP learns an auxiliary opponent modeling task to infer opponents' types and dynamically alter corresponding strategies during execution. Empirically, agents trained via ARSP can achieve stable coordination during training without accessing opponent's rewards or learning process, and can adapt to noncooperative opponents during execution. To the best of our knowledge, it is the first method to learn coordination strategies between agents both in iterated prisoner's dilemma (IPD) and iterated stag hunt (ISH) without shaping opponents or rewards, and can adapt to opponents with distinct strategies during execution. Furthermore, we show that ARSP can be scaled to high-dimensional settings.

Dice: The Infinitely Differentiable Monte-Carlo Estimator

Abstract: The score function estimator is widely used for estimating gradients of stochastic objectives in Stochastic Computation Graphs (SCG), eg. in reinforcement learning and meta-learning. While deriving the first-order gradient estimators by differentiating a surrogate loss (SL) objective is computationally and conceptually simple, using the same approach for higher-order gradients is more challenging. Firstly, analytically deriving and implementing such estimators is laborious and not compliant with automatic differentiation. Secondly, repeatedly applying SL to construct new objectives for each order gradient involves increasingly cumbersome graph manipulations. Lastly, to match the first-order gradient under differentiation, SL treats part of the cost as a fixed sample, which we show leads to missing and wrong terms for higher-order gradient estimators. To address all these shortcomings in a unified way, we introduce DiCE, which provides a single objective that can be differentiated repeatedly, generating correct gradient estimators of any order in SCGs. Unlike SL, DiCE relies on automatic differentiation for performing the requisite graph manipulations. We verify the correctness of DiCE both through a proof and through numerical evaluation of the DiCE gradient estimates. We also use DiCE to propose and evaluate a novel approach for multi-agent learning. Our code is available at this https URL


Our bounds are comparable to those for non-private synthetic control (e.g., #REFR ), and in Section 5.1.1, we explicitly show that the cost of privacy in synthetic control is O(1/ ).

For privacy (Theorems 3.1 and 3.4), although our algorithmic techniques rely on existing approaches, prior results on privacy do not apply in our setting.

DP methods add noise that scales with the sensitivity of the function being computed, which is defined as the maximum change in the function's output that can be caused by changing a single donor's data.

However, synthetic control performs a regression in a vertical way, treating each time point, rather than each donor's data, as one sample -thus, the transposed setting changes the definition of neighboring databases, completely altering the impact of a single donor's data.

The majority of our privacy analysis is devoted to computing sensitivity of this new method.

We also provide accuracy guarantees for each algorithm (Theorems 3.2 and 3.5), bounding the root mean squared error (RMSE) of the algorithm's output compared to the post-intervention target signal.

That is, the RMSE of our algorithm relative to a non-private version is only greater by a factor of O(1/ ), which is unavoidable in most analysis tasks.

To better interpret our bounds in terms of natural problem parameters such as number of samples and length of observations, we also provide Corollaries 5.5 and 7.2, which give explicit closed-form upper bounds on the RMSE under mild assumptions on the underlying data distribution.

Differentially Private Synthetic Control

Abstract: Synthetic control is a causal inference tool used to estimate the treatment effects of an intervention by creating synthetic counterfactual data. This approach combines measurements from other similar observations (i.e., donor pool) to predict a counterfactual time series of interest (i.e., target unit) by analyzing the relationship between the target and the donor pool before the intervention. As synthetic control tools are increasingly applied to sensitive or proprietary data, formal privacy protections are often required. In this work, we provide the first algorithms for differentially private synthetic control with explicit error bounds. Our approach builds upon tools from non-private synthetic control and differentially private empirical risk minimization. We provide upper and lower bounds on the sensitivity of the synthetic control query and provide explicit error bounds on the accuracy of our private synthetic control algorithms. We show that our algorithms produce accurate predictions for the target unit, and that the cost of privacy is small. Finally, we empirically evaluate the performance of our algorithm, and show favorable performance in a variety of parameter regimes, as well as providing guidance to practitioners for hyperparameter tuning.

Robust Synthetic Control

Abstract: We present a robust generalization of the synthetic control method for comparative case studies. Like the classical method cf. Abadie and Gardeazabal (2003), we present an algorithm to estimate the unobservable counterfactual of a treatment unit. A distinguishing feature of our algorithm is that of de-noising the data matrix via singular value thresholding, which renders our approach robust in multiple facets: it automatically identifies a good subset of donors for the synthetic control, overcomes the challenges of missing data, and continues to work well in settings where covariate information may not be provided. We posit that the setting can be viewed as an instance of the Latent Variable Model and provide the first finite sample analysis (coupled with asymptotic results) for the estimation of the counterfactual. Our algorithm accurately imputes missing entries and filters corrupted observations in producing a consistent estimator of the underlying signal matrix, provided p = Ω(T-1+ζ) for some ζ > 0; here, p is the fraction of observed data and T is the time interval of interest. Under the same proportion of observations, we demonstrate that the mean-squared error in our counterfactual estimation scales as O(σ2/p + 1/√T), where σ2 is the variance of the inherent noise. Additionally, we introduce a Bayesian framework to quantify the estimation uncertainty. Our experiments, using both synthetic and real-world datasets, demonstrate that our robust generalization yields an improvement over the classical synthetic control method.

non-private synthetic control
synthetic control


We call these curated versions SSv2-LT and VideoLT-LT, and resample these following the recipe used in #REFR for ImageNet-LT and Places-LT (sampling from a Pareto distribution with α = 6).

Having identified weaknesses in current benchmarks used for long-tail video recognition, we first propose to use EPIC-KITCHENS-100 as it is naturally-collected and satisfies the long-tail properties (as defined in Sec. 2).

We also propose to resample public video datasets, so their properties are in line with curated long-tail image datasets.

SSv2 #OTHEREFR is chosen as it is widely considered to be a good test of temporal understanding and has previously been re-purposed for evaluating few-shot works #OTHEREFR . Similarly, VideoLT #OTHEREFR targets fine-grained classes.

Table 1 demonstrates these curated versions match the desired properties as visualised in Before proceeding to the method, ablations are first performed at a dataset level, where different curated versions of SSv2-LT are compared to demonstrate the impact on longtail properties and the effect of few-shot classes.

Full implementation details of models and metrics will be given in Sec.

6, but for these ablations it suffices to say that Motionformer #OTHEREFR is trained with cross-entropy, reporting average class accuracy over the test set, as well as over few-shot, tail and head classes.

Use Your Head: Improving Long-Tail Video Recognition

Abstract: This paper presents an investigation into long-tail video recognition. We demonstrate that, unlike naturallycollected video datasets and existing long-tail image benchmarks, current video benchmarks fall short on multiple long-tailed properties. Most critically, they lack few-shot classes in their tails. In response, we propose new video benchmarks that better assess long-tail recognition, by sampling subsets from two datasets: SSv2 and VideoLT. We then propose a method, Long-Tail Mixed Reconstruction (LMR), which reduces overfitting to instances from few-shot classes by reconstructing them as weighted combinations of samples from head classes. LMR then employs label mixing to learn robust decision boundaries. It achieves state-of-the-art average class accuracy on EPIC-KITCHENS and the proposed SSv2-LT and VideoLT-LT.

Large-Scale Long-Tailed Recognition In An Open World

Abstract: Real world data often have a long-tailed and open-ended distribution. A practical recognition system must classify among majority and minority classes, generalize from a few known instances, and acknowledge novelty upon a never seen instance. We define Open Long-Tailed Recognition (OLTR) as learning from such naturally distributed data and optimizing the classification accuracy over a balanced test set which include head, tail, and open classes. OLTR must handle imbalanced classification, few-shot learning, and open-set recognition in one integrated algorithm, whereas existing classification approaches focus only on one aspect and deliver poorly over the entire class spectrum. The key challenges are how to share visual knowledge between head and tail classes and how to reduce confusion between tail and open classes. We develop an integrated OLTR algorithm that maps an image to a feature space such that visual concepts can easily relate to each other based on a learned metric that respects the closed-world classification while acknowledging the novelty of the open world. Our so-called dynamic meta-embedding combines a direct image feature and an associated memory feature, with the feature norm indicating the familiarity to known classes. On three large-scale OLTR datasets we curate from object-centric ImageNet, scene-centric Places, and face-centric MS1M data, our method consistently outperforms the state-of-the-art. Our code, datasets, and models enable future OLTR research and are publicly available at \url{https://liuziwei7.github.io/projects/LongTail.html}.


The former corresponds to solely infer time-dependent traffic flow fields, and the latter corresponds to system identification of traffic states #REFR .

Once the physics model is selected, we need to determine the sequence of parameter calibration prior to, or during the training of PUNN.

1) Sequential training Sequential training aims to first calibrate parameters of the PICG (i.e., parameter calibration) and then encode the known physics into the PUNN for training.

Parameter calibration has been extensively studied in TSE using non-linear programming #OTHEREFR , genetic algorithm #OTHEREFR , least-squares fitting #OTHEREFR - #OTHEREFR , and kernel smoothing #OTHEREFR .

The physics based parameters include ρ max , u max and other nominal parameters. Tab. I summarizes the existing methods for model discovery.

Sequential training is the default paradigm in most PIDLrelated studies, with a focus on how to make the training robust and stable, when large-scale NNs and complicated physicsinformed loss functions are involved. A growing amount of #OTHEREFR .

The adaptive activation function has also been used in DeLISA (deep learning based iteration scheme approximation), which adopts the implicit multistep method and Runge-Kutta method for time iteration scheme to construct the learning loss when training the PUNN #OTHEREFR .

Physics-Informed Deep Learning For Traffic State Estimation: A Survey And The Outlook

Abstract: For its robust predictive power (compared to pure physics-based models) and sample-efficient training (compared to pure deep learning models), physics-informed deep learning (PIDL), a paradigm hybridizing physics-based models and deep neural networks (DNN), has been booming in science and engineering fields. One key challenge of applying PIDL to various domains and problems lies in the design of a computational graph that integrates physics and DNNs. In other words, how physics are encoded into DNNs and how the physics and data components are represented. In this paper, we provide a variety of architecture designs of PIDL computational graphs and how these structures are customized to traffic state estimation (TSE), a central problem in transportation engineering. When observation data, problem type, and goal vary, we demonstrate potential architectures of PIDL computational graphs and compare these variants using the same real-world dataset. Impact Statement-Despite benefits of physics-informed deep learning (PIDL) for robust and efficient pattern learning, how to inject physics into neural networks (NN) remains underexplored and varies case by case and across domains. We will summarize and establish a systematic design pipeline for hybrid computational graphs that facilitates the integration of physics and NNs. The insights into the modular design of the hybrid PIDL paradigm as well as the established visualization tool will not only be useful to guide transportation researchers to pursue PIDL, but can also facilitate researchers at large to better understand and decompose a PIDL problem when applied to their own application domains. It will hopefully open up the conversation across domains about establishing a unified PIDL paradigm.

Hidden Physics Models: Machine Learning Of Nonlinear Partial Differential Equations

Abstract: While there is currently a lot of enthusiasm about"big data", useful data is usually"small"and expensive to acquire. In this paper, we present a new paradigm of learning partial differential equations from {\em small} data. In particular, we introduce \emph{hidden physics models}, which are essentially data-efficient learning machines capable of leveraging the underlying laws of physics, expressed by time dependent and nonlinear partial differential equations, to extract patterns from high-dimensional data generated from experiments. The proposed methodology may be applied to the problem of learning, system identification, or data-driven discovery of partial differential equations. Our framework relies on Gaussian processes, a powerful tool for probabilistic inference over functions, that enables us to strike a balance between model complexity and data fitting. The effectiveness of the proposed approach is demonstrated through a variety of canonical problems, spanning a number of scientific domains, including the Navier-Stokes, Schr\"odinger, Kuramoto-Sivashinsky, and time dependent linear fractional equations. The methodology provides a promising new direction for harnessing the long-standing developments of classical methods in applied mathematics and mathematical physics to design learning machines with the ability to operate in complex domains without requiring large quantities of data.

time-dependent traffic flow
system identification


For achieving leak-proof optimized pistons, the user may refer to #REFR for recommendations on choosing etaf and betaf.

15b , c that are obtained by the following call TOPress (300 , 100 , 0.2 , 3 , 2.4 , 0.1 , 8 , lst , 150) with lst=1 and lst=0, respectively.

We consider the full model to solve this problem to check any deviation from the symmetric nature of the problem.

One notes that the optimized design retains the symmetric nature ( Fig.

15b ) and qualitatively resembles the features of the piston structures reported in the previous literature.

One may also solve this problem using only the symmetric half of the domain.

Different optimized material layouts are obtained with lst=1 and lst=0.

Topress: A Matlab Implementation For Topology Optimization Ofstructures Subjected To Design-Dependent Pressure Loads

Abstract: In a topology optimization setting, design-dependent fluidic pressure loads pose several challenges as their direction, magnitude, and location alter with topology evolution. This paper offers a compact 100-line MATLAB code, TOPress, for topology optimization of structures subjected to fluidic pressure loads using the method of moving asymptotes. The code is intended for pedagogical purposes and aims to ease the beginners' and students' learning toward the topology optimization with designdependent fluidic pressure loads. TOPress is developed per the approach first reported in Kumar et al. (Struct Multidisc Optim 61(4):1637-1655, 2020). The Darcy law, in conjunction with the drainage term, is used to model the applied pressure load. The consistent nodal loads are determined from the obtained pressure field. The employed approach facilitates inexpensive computation of the load sensitivities using the adjoint-variable method. Compliance minimization subject to volume constraint optimization problems are solved. The success and efficacy of the code are demonstrated by solving benchmark numerical examples involving pressure loads, wherein the importance of load sensitivities is also demonstrated. TOPress contains six main parts, is described in detail, and is extended to solve different problems. Steps to include a projection filter are provided to achieve loadbearing designs close to 0-1. The code is provided in Appendix B and can also be downloaded along with its extensions from https://github.com/PrabhatIn/TOPress.

Topology Optimization Of Fluidic Pressure Loaded Structures And Compliant Mechanisms Using The Darcy Method

Abstract: In various applications, design problems involving structures and compliant mechanisms experience fluidic pressure loads. During topology optimization of such design problems, these loads adapt their direction and location with the evolution of the design, which poses various challenges. A new density-based topology optimization approach using Darcy’s law in conjunction with a drainage term is presented to provide a continuous and consistent treatment of design-dependent fluidic pressure loads. The porosity of each finite element and its drainage term are related to its density variable using a Heaviside function, yielding a smooth transition between the solid and void phases. A design-dependent pressure field is established using Darcy’s law and the associated PDE is solved using the finite element method. Further, the obtained pressure field is used to determine the consistent nodal loads. The approach provides a computationally inexpensive evaluation of load sensitivities using the adjoint-variable method. To show the efficacy and robustness of the proposed method, numerical examples related to fluidic pressure-loaded stiff structures and small-deformation compliant mechanisms are solved. For the structures, compliance is minimized, whereas for the mechanisms, a multi-criteria objective is minimized with given resource constraints.