We begin our experiments by rendering synthetic models from the PartNet-Mobility dataset #REFR in various articulation states since this enables us to obtain sufficient annotations to train 2D segmentation networks and thus enable transfer learning applications.

Note that such manual annotations are used only to evaluate our active learning framework.

From Table 1 , we observe that the majority of data samples in OPDReal belong to the Storage category (91.67%), with the rest distributed among the remaining categories.

The difference in distribution between the largest and the second largest category is 87.74%, and between the largest and smallest categories is 91.14%.

With such data skewness towards one category, models trained on OPDReal will likely overfit to the dominant category.Our dataset, on the other hand, contains smaller variations in data distributions across the six object categories, where the difference in data distribution between the largest and second largest categories is 5.88%, and between the largest and smallest categories is 29.47%.

Our synthetic dataset contains around 32K articulation images, with equal data samples for each object category.

We use a 90%-10% train-test split to train on this synthetic dataset, and use this trained model for fine-tuning on real images.

To this end, we use all data samples from both datasets, OPDReal and ours, with 80%-20% train-test split.

We implement our network in PyTorch on a Nvidia RTX 2080 Ti GPU. All images are resized to 256×256 for training.

During pre-training on the PartNet-Mobility dataset, we use the Adam optimizer with a learning rate (lr) of 2.5e-4, and train for 2K epochs.

Coarse-To-Fine Active Segmentation Of Interactable Parts In Real Scene Images

Abstract: OPDRCNN-C Mask2Former ours w/o AL GT ours with AL input Figure 1: 2D segmentation results on interactable object parts in real-world indoor scene images. We show part segmentation masks obtained using similar methods, viz. OPD [15] and Mask2Former [7]. We also contrast the results of our proposed pose-aware masked-attention model, with and without Active Learning (AL) framework.

Sapien: A Simulated Part-Based Interactive Environment

Abstract: Building home assistant robots has long been a pursuit for vision and robotics researchers. To achieve this task, a simulated environment with physically realistic simulation, sufficient articulated objects, and transferability to the real robot is indispensable. Existing environments achieve these requirements for robotics simulation with different levels of simplification and focus. We take one step further in constructing an environment that supports household tasks for training robot learning algorithm. Our work, SAPIEN, is a realistic and physics-rich simulated environment that hosts a large-scale set for articulated objects. Our SAPIEN enables various robotic vision and interaction tasks that require detailed part-level understanding.We evaluate state-of-the-art vision algorithms for part detection and motion attribute recognition as well as demonstrate robotic interaction tasks using heuristic approaches and reinforcement learning algorithms. We hope that our SAPIEN can open a lot of research directions yet to be explored, including learning cognition through interaction, part motion discovery, and construction of robotics-ready simulated game environment.


In this work, we propose to use the weakly supervised anomaly detection #REFR to detect baby cries on audio signals.

On the other hand, the complexity of the CNN may be high, preventing it be used in common embedded devices, like low-cost IP cameras or tablets.

This paper will address this issue by using a super lightweight CNN.

Most, if not all, existing CNN methods use supervised learning. Therefore, frame-level annotation is needed.

This annotation is very time-consuming and prone to human mistakes.

In the few datasets available online, some of the audio files [8, 27, #OTHEREFR are trimmed and annotated, and some others #OTHEREFR are untrimmed with only audio-level annotations.

This weak supervision only requires weak annotation, i.e., if there is a cry in the audio file without frame-level annotation, therefore making the data annotation a lot easier.

Based on this weakly supervised anomaly detection and a well-known pretrained VGGish network #OTHEREFR for audio feature extraction, we design a data mining technique to obtain frame-level datasets for supervised CNN classification.

We use this dataset to train a simple, lightweight CNN, which runs very fast on embedded devices.

This CNN, as the feature extractor in an anomaly detection framework, gives better performance than the CNN alone.

The contribution of this paper is three-fold,

Weakly Supervised Detection Of Baby Cry

Abstract: Detection of baby cries is an important part of baby monitoring and health care. Almost all existing methods use supervised SVM, CNN, or their varieties. In this work, we propose to use weakly supervised anomaly detection to detect a baby cry. In this weak supervision, we only need weak annotation if there is a cry in an audio file. We design a data mining technique using the pretrained VGGish feature extractor and an anomaly detection network on long untrimmed audio files. The obtained datasets are used to trained a simple CNN feature network for cry/non-cry classification. This CNN is then used as a feature extractor in an anomaly detection framework to achieve better cry detection performance.

Real-World Anomaly Detection In Surveillance Videos

Abstract: Surveillance videos are able to capture a variety of realistic anomalies. In this paper, we propose to learn anomalies by exploiting both normal and anomalous videos. To avoid annotating the anomalous segments or clips in training videos, which is very time consuming, we propose to learn anomaly through the deep multiple instance ranking framework by leveraging weakly labeled training videos, i.e. the training labels (anomalous or normal) are at video-level instead of clip-level. In our approach, we consider normal and anomalous videos as bags and video segments as instances in multiple instance learning (MIL), and automatically learn a deep anomaly ranking model that predicts high anomaly scores for anomalous video segments. Furthermore, we introduce sparsity and temporal smoothness constraints in the ranking loss function to better localize anomaly during training. We also introduce a new large-scale first of its kind dataset of 128 hours of videos. It consists of 1900 long and untrimmed real-world surveillance videos, with 13 realistic anomalies such as fighting, road accident, burglary, robbery, etc. as well as normal activities. This dataset can be used for two tasks. First, general anomaly detection considering all anomalies in one group and all normal activities in another group. Second, for recognizing each of 13 anomalous activities. Our experimental results show that our MIL method for anomaly detection achieves significant improvement on anomaly detection performance as compared to the state-of-the-art approaches. We provide the results of several recent deep learning baselines on anomalous activity recognition. The low recognition performance of these baselines reveals that our dataset is very challenging and opens more opportunities for future work.


The UCF-Crime #REFR dataset is a large-scale video anomaly detection dataset, which contains 13 anomaly classes.

This dataset consists of not only abnormal videos but also normal videos where no abnormal events occur.

UCF-Crime only has video-level abnormal-or-not labels, i.e., whether the current video has abnormal events or not, without the specific class of the abnormal events.

When we use UCF-Crime for TAL, we only care about abnormal events. The normal events act as the background.

Therefore, to better match the temporal action localization task, we remove videos which only contain normal events to construct our UCF-Crime-TAL dataset.

Then we annotate the UCF-Crime-TAL dataset with dense labels at the frame level. Given a video, we first locate the abnormal events.

Co-Occurrence Matters: Learning Action Relation For Temporal Action Localization

Abstract: Temporal action localization (TAL) is a prevailing task due to its great application potential. Existing works in this field mainly suffer from two weaknesses: (1) They often neglect the multi-label case and only focus on temporal modeling. (2) They ignore the semantic information in class labels and only use the visual information. To solve these problems, we propose a novel Co-Occurrence Relation Module (CORM) that explicitly models the co-occurrence relationship between actions. Besides the visual information, it further utilizes the semantic embeddings of class labels to model the co-occurrence relationship. The CORM works in a plug-and-play manner and can be easily incorporated with the existing sequence models. By considering both visual and semantic co-occurrence, our method achieves high multi-label relationship modeling capacity. Meanwhile, existing datasets in TAL always focus on low-semantic atomic actions. Thus we construct a challenging multi-label dataset UCF-Crime-TAL that focuses on high-semantic actions by annotating the UCF-Crime dataset at frame level and considering the semantic overlap of different events. Extensive experiments on two commonly used TAL datasets, i.e., MultiTHUMOS and TSU, and our newly proposed UCF-Crime-TAL demenstrate the effectiveness of the proposed CORM, which achieves state-of-theart performance on these datasets.


NPs are stochastic processes using neural networks to capture the ground truth stochastic processes that generate the given data #REFR .

In a regression problem, y C i is a continuous vector, whereas in a classification problem, y C i can be a one-hot encoded vector. A target set is defined as

where the number of target points is N T .

We let a mapping function f : X → Y be sampled from a probability distribution over functions and a stochastic process defined as a probability measure of such sampled functions.

We assume that there exists a finite set of all possible functions P(X ) where f ∈ P(X ).

NPs learn the distribution of functions and provide uncertainty of target samples, preserving the property of GPs.

At the same time, NPs exploit function approximation of neural networks in a more efficient manner than GPs.

The learning objective of NPs is to maximise the likelihood of p(f (T )|C, T ) (Garnelo et al., 2018b; a; #OTHEREFR .

Multimodal Neural Processes For Uncertainty Estimation

Abstract: Neural processes (NPs) have brought the representation power of parametric deep neural networks and the reliable uncertainty estimation of non-parametric Gaussian processes together. Although recent development of NPs has shown success in both regression and classification, how to adapt NPs to multimodal data has not be carefully studied. For the first time, we propose a new model of NP family for multimodal uncertainty estimation, namely Multimodal Neural Processes. In a holistic and principled way, we develop a dynamic context memory updated by the classification error, a multimodal Bayesian aggregation mechanism to aggregate multimodal representations, and a new attention mechanism for calibrated predictions. In extensive empirical evaluation, our method achieves the stateof-the-art multimodal uncertainty estimation performance, showing its appealing ability of being robust against noisy samples and reliable in outof-domain detection.

Conditional Neural Processes

Abstract: Deep neural networks excel at function approximation, yet they are typically trained from scratch for each new function. On the other hand, Bayesian methods, such as Gaussian Processes (GPs), exploit prior knowledge to quickly infer the shape of a new function at test time. Yet GPs are computationally expensive, and it can be hard to design appropriate priors. In this paper we propose a family of neural models, Conditional Neural Processes (CNPs), that combine the benefits of both. CNPs are inspired by the flexibility of stochastic processes such as GPs, but are structured as neural networks and trained via gradient descent. CNPs make accurate predictions after observing only a handful of training data points, yet scale to complex functions and large datasets. We demonstrate the performance and versatility of the approach on a range of canonical machine learning tasks, including regression, classification and image completion.


On the other hand, VI-DSO #REFR ) employs a combined energy functional to jointly estimate camera poses and sparse scene geometry by minimizing the visual and inertial measurement errors.

Nevertheless, Transformers are not adequate for integrating absolute and relative poses with their low-dimensional information.

Also, the bidirectional properties #OTHEREFR of LSTMs are unfeasible for pose modeling.

#OTHEREFR fuse an absolute pose (from GNSS data) with a relative pose (from vehicle odometry).

The following two works integrate relative poses from inertial data with absolute poses from visual data.

VINet #OTHEREFR ) utilizes a concatenation approach to merge relative features extracted by an inertial LSTM encoder with absolute features extracted by a visual encoder.

Fusing Structure From Motion And Simulation-Augmented Pose Regression From Optical Flow For Challenging Indoor Environments ⋆

Abstract: The localization of objects is a crucial task in various applications such as robotics, virtual and augmented reality, and the transportation of goods in warehouses. Recent advances in deep learning have enabled the localization using monocular visual cameras. While structure from motion (SfM) predicts the absolute pose from a point cloud, absolute pose regression (APR) methods learn a semantic understanding of the environment through neural networks. However, both fields face challenges caused by the environment such as motion blur, lighting changes, repetitive patterns, and featureless structures. This study aims to address these challenges by incorporating additional information and regularizing the absolute pose using relative pose regression (RPR) methods. The optical flow between consecutive images is computed using the Lucas-Kanade algorithm, and the relative pose is predicted using an auxiliary small recurrent convolutional network. The fusion of absolute and relative poses is a complex task due to the mismatch between the global and local coordinate systems. State-of-the-art methods fusing absolute and relative poses use pose graph optimization (PGO) to regularize the absolute pose predictions using relative poses. In this work, we propose recurrent fusion networks to optimally align absolute and relative pose predictions to improve the absolute pose prediction. We evaluate eight different recurrent units and construct a simulation environment to pre-train the APR and RPR networks for better generalized training. Additionally, we record a large database of different scenarios in a challenging large-scale indoor environment that mimics a warehouse with transportation robots. We conduct hyperparameter searches and experiments to show the effectiveness of our recurrent fusion method compared to PGO. ⋆ Evaluating the performance of absolute pose regression methods, i.e., SfM and PoseNet, in challenging scenarios such as repetitive patterns, motion blur, structure-less surfaces, and difficult lighting conditions. Fusing absolute and relative poses using either PGO or recurrent networks.

Direct Sparse Visual-Inertial Odometry Using Dynamic Marginalization

Abstract: We present VI-DSO, a novel approach for visual-inertial odometry, which jointly estimates camera poses and sparse scene geometry by minimizing photometric and IMU measurement errors in a combined energy functional. The visual part of the system performs a bundle-adjustment like optimization on a sparse set of points, but unlike key-point based systems it directly minimizes a photometric error. This makes it possible for the system to track not only corners, but any pixels with large enough intensity gradients. IMU information is accumulated between several frames using measurement preintegration, and is inserted into the optimization as an additional constraint between keyframes. We explicitly include scale and gravity direction into our model and jointly optimize them together with other variables such as poses. As the scale is often not immediately observable using IMU data this allows us to initialize our visual-inertial system with an arbitrary scale instead of having to delay the initialization until everything is observable. We perform partial marginalization of old variables so that updates can be computed in a reasonable time. In order to keep the system consistent we propose a novel strategy which we call"dynamic marginalization". This technique allows us to use partial marginalization even in cases where the initial scale estimate is far from the optimum. We evaluate our method on the challenging EuRoC dataset, showing that VI-DSO outperforms the state of the art.

camera poses
inertial measurement errors


The ample evidence of compression even when orthographic word lengths are replaced by word durations in human languages provides stronger support for compression as a universal principle of the organization of languages with respect to previous research relying on word length in characters #REFR .

We have found that word lengths are shorter than expected by chance ( < ) in all languages in every collection (Figure 2 ).

Such a systematic finding is unlikely to be accidental and strongly indicates that compression is acting in all languages in our sample.

Crucially, the finding holds independently of how word length is measured.

It could be argued that these findings constitute evidence of compression in ensembles of language but not in individual languages.

The reason is that < does not imply that the difference between the actual word length and the random baseline is statistically significant for a single language.

However, we have shown that the Pearson correlation is indeed a linear function of and (Appendix A) and thus is significantly small in every language where the law of abbreviation has been confirmed using a Pearson correlation test.

Finally, the direct correspondence we have established between the average length of types ( ) and the random baseline sheds new light on previous research.

For instance, it has been shown that < in Chinese characters in six time periods spanning two millennia (Chen et al., 2015, Fig.

Direct And Indirect Evidence Of Compression Of Word Lengths. Zipf'S Law Of Abbreviation Revisited

Abstract: Direct and indirect evidence of compression of word lengths. Zipf's law of abbreviation revisited. Sonia Petrini 1 (0000-0002-0514-6223), Antoni Casas-i-Muñoz 2 (0000-0001-5690-316X), Jordi Cluet-i-Martinell 2 (0000-0003-4188-6728), Mengxue Wang 2 (0000-0002-8262-9333), Christian Bentz 3 (0000-0001-6570-9326), Ramon Ferrer-i-Cancho 1* (0000-0002-7820-923X)

Compression As A Universal Principle Of Animal Behavior

Abstract: A key aim in biology and psychology is to identify fundamental principles underpinning the behavior of animals, including humans. Analyses of human language and the behavior of a range of non-human animal species have provided evidence for a common pattern underlying diverse behavioral phenomena: Words follow Zipf’s law of brevity (the tendency of more frequently used words to be shorter), and conformity to this general pattern has been seen in the behavior of a number of other animals. It has been argued that the presence of this law is a sign of efficient coding in the information theoretic sense. However, no strong direct connection has been demonstrated between the law and compression, the information theoretic principle of minimizing the expected length of a code. Here, we show that minimizing the expected code length implies that the length of a word cannot increase as its frequency increases. Furthermore, we show that the mean code length or duration is significantly small in human language, and also in the behavior of other species in all cases where agreement with the law of brevity has been found. We argue that compression is a general principle of animal behavior that reflects selection for efficiency of coding.


This differs from our previous work #REFR where we assign stationary sensors to localize targets.

where z j i,t denotes the measurement of target j by robot i's sensor at time t.

v j i,t (x i,t , y j,t ) denotes the Gaussian measurement noise with zero mean and covariance

R(x i,t , y j,t ), i.e., v j i,t (x i,t , y j,t ) ∼ N (0, R(x i,t , y j,t )).

Here, both the sensing model h j i (x i,t , y j,t ) and measurement noise v j i,t (x i,t , y j,t ) depend on the state of robot i and target j.

d) Objective: The objective is to maximize the tracking quality of the targets by assigning robots and their selected actions to the targets.

Here, considering that robots are moving by executing actions, we assign robot-actions (per robot an action) to track the targets.

That way, we denote φ(j) as the set of robotactions assigned to target j and φ −1 (i k ) as the set of targets assigned to robot-action i k (i.e., robot i with action k).

We use φ l (j) to denote the l th robot-action assigned to target j. We order the assigned robot-actions such that

Denote the corresponding robot for each robot-action φ l (j) as i(φ l (j)).

Assignment Algorithms For Multi-Robot Multi-Target Tracking With Sufficient And Limited Sensing Capability

Abstract: We study the problem of assigning robots with actions to track targets. The objective is to optimize the robot team's tracking quality which can be defined as the reduction in the uncertainty of the targets' states. Specifically, we consider two assignment problems given the different sensing capabilities of the robots. In the first assignment problem, a single robot is sufficient to track a target. To this end, we present a greedy algorithm (Algorithm 1) that assigns a robot with its action to each target. We prove that the greedy algorithm has a 1/2approximation bound and runs in polynomial time. Then, we study the second assignment problem where two robots are necessary to track a target. We design another greedy algorithm (Algorithm 2) that assigns a pair of robots with their actions to each target. We prove that the greedy algorithm achieves a 1/3-approximation bound and has a polynomial running time. Moreover, we illustrate the performance of the two greedy algorithms in the ROS-Gazebo environment where the tracking patterns of one robot following one target using Algorithm 1 and two robots following one target using Algorithm 2 are clearly observed. Further, we conduct extensive comparisons to demonstrate that the two greedy algorithms perform close to their optimal counterparts and much better than their respective (1/2 and 1/3) approximation bounds.

Sensor Assignment Algorithms To Improve Observability While Tracking Targets

Abstract: In this paper, we study two sensor assignment problems for multitarget tracking with the goal of improving the observability of the underlying estimator. We consider various measures of the observability matrix as the assignment value function. We first study the general version where the sensors must form teams to track individual targets. If the value function is monotonically increasing and submodular, then a greedy algorithm yields a $1/2$ –approximation. We then study a restricted version where exactly two sensors must be assigned to each target. We present a $1/3$ –approximation algorithm for this problem, which holds for arbitrary value functions (not necessarily submodular or monotone). In addition to approximation algorithms, we also present various properties of observability measures. We show that the inverse of the condition number of the observability matrix is neither monotone nor submodular, but present other measures that are. Specifically, we show that the trace and rank of the symmetric observability matrix are monotone and submodular and the log determinant of the symmetric observability matrix is monotone and submodular when the matrix is nonsingular. If the target's motion model is not known, the inverse cannot be computed exactly. Instead, we present a lower bound for distance sensors. In addition to theoretical results, we evaluate our results empirically through simulations.


The proposed network architecture is largely similar to the object detector YOLOv3 #REFR with changes in the output.

We summarize Locanet, a framework that estimates the 3D positions of neighboring robots using a deep neural network #OTHEREFR .

In particular, rather than predicting bounding boxes for each robot, the network outputs for each robot the pixel coordinates of the robot's center and its distance to the camera directly.

In order to support multi-robot detection, the neural network outputs 28 × 40 grid cells with two channels each. The first channel is the confidence map.

Grid cell locations refer to the center of the robot in the image plane, where a confidence score above a threshold indicates that a robot was detected.

A score close to zero signals that no robot was detected in this part of the image.

The second channel is the depth map containing predicted distances between the neighboring robot and the camera. This channel is only valid for high confidence scores.

A Dataset And Comparative Study For Vision-Based Relative Position Estimation Of Multirotor Teams Flying In Close Proximity

Abstract: Multirotor teams are useful for inspection, delivery, and construction tasks, in which they might be required to fly very close to each other. In such close-proximity cases, nonlinear aerodynamic effects can cause catastrophic crashes, necessitating each robots' awareness of the surroundings. Existing approaches rely on expensive or heavy perception sensors. Instead, we propose to use the often ignored yaw degree-offreedom of multirotors to spin a single, cheap and lightweight monocular camera at a high angular rate for omnidirectional awareness. We provide a dataset collected with real-world physical flights as well as with 3D-rendered scenes and compare two existing learning-based methods in different settings with respect to success rate, relative position estimation, and downwash prediction accuracy. As application we demonstrate that our proposed spinning camera is capable of predicting the presence of aerodynamic downwash in a challenging swapping task.

Yolov3: An Incremental Improvement

Abstract: We present some updates to YOLO! We made a bunch of little design changes to make it better. We also trained this new network that's pretty swell. It's a little bigger than last time but more accurate. It's still fast though, don't worry. At 320x320 YOLOv3 runs in 22 ms at 28.2 mAP, as accurate as SSD but three times faster. When we look at the old .5 IOU mAP detection metric YOLOv3 is quite good. It achieves 57.9 mAP@50 in 51 ms on a Titan X, compared to 57.5 mAP@50 in 198 ms by RetinaNet, similar performance but 3.8x faster. As always, all the code is online at this https URL


For example, for the backdoored models produced via patch attack #REFR , the inverted trigger share the same location (i.e., right upper corner) and the color (i.e., yellow) with the injected trigger.

These results indicate UNICORN is effective for inverting backdoor triggers injected in different spaces.

We also show the visualization of the inverted triggers as well as the ground-truth triggers in Fig. 3 .

The first row shows the original image and the samples pasted with different injected triggers.

The dataset and the model used here are ImageNet and VGG16.

As can be observed, UNICORN can effectively invert the trigger that is similar to injected triggers.

For the attack in filter space, feature space and numerical space, our method also successfully inverted the global perturbations.

The inverted triggers might be not equal to the injected triggers.

This is because existing backdoor attacks are inaccurate #OTHEREFR .

In other words, a trigger that is similar but not equal to the injected one is also able to activate the malicious backdoor behaviors of the model.

We observed that pixel space attacks are more accurate than the attacks in other spaces.

Unicorn: A Unified Backdoor Trigger Inver-Sion Framework

Abstract: The backdoor attack, where the adversary uses inputs stamped with triggers (e.g., a patch) to activate pre-planted malicious behaviors, is a severe threat to Deep Neural Network (DNN) models. Trigger inversion is an effective way of identifying backdoor models and understanding embedded adversarial behaviors. A challenge of trigger inversion is that there are many ways of constructing the trigger. Existing methods cannot generalize to various types of triggers by making certain assumptions or attack-specific constraints. The fundamental reason is that existing work does not consider the trigger's design space in their formulation of the inversion problem. This work formally defines and analyzes the triggers injected in different spaces and the inversion problem. Then, it proposes a unified framework to invert backdoor triggers based on the formalization of triggers and the identified inner behaviors of backdoor models from our analysis. Our prototype UNICORN is general and effective in inverting backdoor triggers in DNNs. The code can be found at https://github.com/ RU-System-Software-and-Security/UNICORN.

Badnets: Identifying Vulnerabilities In The Machine Learning Model Supply Chain

Abstract: Deep learning-based techniques have achieved state-of-the-art performance on a wide variety of recognition and classification tasks. However, these networks are typically computationally expensive to train, requiring weeks of computation on many GPUs; as a result, many users outsource the training procedure to the cloud or rely on pre-trained models that are then fine-tuned for a specific task. In this paper we show that outsourced training introduces new security risks: an adversary can create a maliciously trained network (a backdoored neural network, or a \emph{BadNet}) that has state-of-the-art performance on the user's training and validation samples, but behaves badly on specific attacker-chosen inputs. We first explore the properties of BadNets in a toy example, by creating a backdoored handwritten digit classifier. Next, we demonstrate backdoors in a more realistic scenario by creating a U.S. street sign classifier that identifies stop signs as speed limits when a special sticker is added to the stop sign; we then show in addition that the backdoor in our US street sign detector can persist even if the network is later retrained for another task and cause a drop in accuracy of {25}\% on average when the backdoor trigger is present. These results demonstrate that backdoors in neural networks are both powerful and---because the behavior of neural networks is difficult to explicate---stealthy. This work provides motivation for further research into techniques for verifying and inspecting neural networks, just as we have developed tools for verifying and debugging software.


Quantitative Evaluation Metrics & User Study

We use the learned perceptual image patch similarity (LPIPS) metric #REFR to measure the diversity of generated images.

The lower the FID, the better the quality of the generated images. NDB and JSD.

We use the number of statistically different bins (NDB) and the Jensen-Shannon divergence (JSD) #OTHEREFR to evaluate the authenticity of the generated images.

Compared with FID, the NDB and JSD metrics are directly applied to image pixels and do not rely on the learned representation.

This makes the metric more sensitive to pixel-level differences in images.

We set the number of bins for NDB to 50. LPIPS.

We generate 1,000 samples for each class and compute the LPIPS distance between pairs of samples.

The average LPIPS distance for all the classes is used as the final LPIPS value. User Study. We conduct a user study based on pairwise comparisons.

Given the POC image groups generated by AGTGAN and other models, each subject needs to answer the question "Which POC image group is more realistic?" with a real POC image as a reference.

Agtgan: Unpaired Image Translation For Photographic Ancient Character Generation

Abstract: The study of ancient writings has great value for archaeology and philology. Essential forms of material are photographic characters, but manual photographic character recognition is extremely timeconsuming and expertise-dependent. Automatic classification is therefore greatly desired. However, the current performance is limited due to the lack of annotated data. Data generation is an inexpensive but useful solution to data scarcity. Nevertheless, the diverse glyph shapes and complex background textures of photographic ancient characters make the generation task difficult, leading to unsatisfactory results of existing methods. To this end, we propose an unsupervised generative adversarial network called AGTGAN in this paper. By explicitly modeling global and local glyph shape style, followed by a stroke-aware texture transfer and an associate adversarial learning mechanism, our method can generate characters with diverse glyphs and realistic textures. We evaluate our method on photographic ancient character datasets, e.g., OBC306 and CSDD. Our method outperforms other state-of-the-art methods in terms of various metrics and performs much better in terms of the diversity and authenticity of generated samples. With our generated images, experiments on the largest photographic oracle bone character dataset show that our method can achieve a significant increase in classification accuracy, up to 16.34%. The source code is

The Unreasonable Effectiveness Of Deep Features As A Perceptual Metric

Abstract: While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on the ImageNet classification task has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called"perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new Full Reference Image Quality Assessment (FR-IQA) dataset of perceptual human judgments, orders of magnitude larger than previous datasets. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by huge margins. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.

generated images
learned perceptual image


The elements in the output sequence z are calculated as the weighted sum of linearly transformed input elements #REFR .

Attention is the technique that has the capability of selecting and focusing on only the most pertinent information.

In self-attention essentially the inputs are able to interact with each other to identify what parts to pay more attention to.

This mechanism employs similar pixels and disregards the ones having no correlation to the other pixels in a feature map #OTHEREFR .

This works in a way that each attention head presides over an input sequence, x = (x1, x2, …, xn) where xi ∈ ℝ dx , and outputs another sequence z = (z1, z2, . . ., zn) where zi ∈ ℝ dz .

Here dx and dz refer to the embedding size of input and output sequence respectively.

The weight coefficient αij is calculated with the help of a softmax function as follows:

Here eij is calculated by implementing a scaled dot product to compare two input values: In multi-headed self-attention, individual attention heads are generally concatenated and run parallelly with the application of a linear transformation.

This permits the model to together pay attention to information from multiple representation subspaces simultaneously (Vaswani et al., 2017) .

Early Detection Of Alzheimer'S Disease Using Bottleneck Transformers

Abstract: Early detection of Alzheimer's Disease (AD) and its prodromal state, Mild Cognitive Impairment (MCI), is crucial for providing suitable treatment and preventing the disease from progressing. It can also aid researchers and clinicians to identify early biomarkers and minister new treatments that have been a subject of extensive research. The application of deep learning techniques on structural Magnetic Resonance Imaging (MRI) has shown promising results in diagnosing the disease. In this research, we intend to introduce a novel approach of using an ensemble of the self-attention-based Bottleneck Transformers with a sharpness aware minimizer for early detection of Alzheimer's Disease. The proposed approach has been tested on the widely accepted ADNI dataset and evaluated using accuracy, precision, recall, F1 score, and ROC-AUC score as the performance metrics.

Self-Attention With Relative Position Representations

Abstract: Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast to recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements. On the WMT 2014 English-to-German and English-to-French translation tasks, this approach yields improvements of 1.3 BLEU and 0.3 BLEU over absolute position representations, respectively. Notably, we observe that combining relative and absolute position representations yields no further improvement in translation quality. We describe an efficient implementation of our method and cast it as an instance of relation-aware self-attention mechanisms that can generalize to arbitrary graph-labeled inputs.


The model produces superior results in comparison to Cudeiro et al. #REFR and Richard et al.

#OTHEREFR aims for audio-driven animation that can capture variations in multiple speakers including a much larger dataset of 250 subjects.

They address the problem of lack of upper face motions using a categorical latent space that disentangles audio-correlated and audio-uncorrelated information based on a cross-modality loss. Fan et al.

#OTHEREFR proposes an audio and text-driven facial animation method that incorporates the large language model GPT-2 #OTHEREFR to encode the textual information.

The authors found that combined audio and text input yielded better results than audio-only or text-only model. However, the results still have problems regarding accurate lip-sync.

Most closely related to our work is Face-Former #OTHEREFR which uses a self-supervised pretrained speech model that addresses the scarcity of available data in existing audio-visual datasets.

#OTHEREFR using a modified version of transformers to handle longer sequences of data.

However, none of these methods can handle arbitrary variations in speech input while producing accurate lower and upper facial animation for multiple identities and emotions.

Facexhubert: Text-Less Speech-Driven E(X)Pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learning

Abstract: This paper presents FaceXHuBERT, a text-less speechdriven 3D facial animation generation method that allows to capture personalized and subtle cues in speech (e.g. identity, emotion and hesitation). It is also very robust to background noise and can handle audio recorded in a variety of situations (e.g. multiple people speaking). Recent approaches employ end-to-end deep learning taking into account both audio and text as input to generate facial animation for the whole face. However, scarcity of publicly available expressive audio-3D facial animation datasets poses a major bottleneck. The resulting animations still have issues regarding accurate lip-synching, expressivity, person-specific information and generalizability. We effectively employ self-supervised pretrained HuBERT model in the training process that allows us to incorporate both lexical and non-lexical information in the audio without using a large lexicon. Additionally, guiding the training with a binary emotion condition and speaker identity distinguishes the tiniest subtle facial motion. We carried out extensive objective and subjective evaluation in comparison to ground-truth and state-of-the-art work. A perceptual user study demonstrates that our approach produces superior results with respect to the realism of the animation 78% of the time in comparison to the state-of-the-art. In addition, our method is 4 times faster eliminating the use of complex sequential models such as transformers. We strongly recommend watching the supplementary video before reading the paper. We also provide the implementation and evaluation codes with a GitHub repository link.

Capture, Learning, And Synthesis Of 3D Speaking Styles

Abstract: Audio-driven 3D facial animation has been widely explored, but achieving realistic, human-like performance is still unsolved. This is due to the lack of available 3D datasets, models, and standard evaluation metrics. To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers. We then train a neural network on our dataset that factors identity from facial motion. The learned model, VOCA (Voice Operated Character Animation) takes any speech signal as input—even speech in languages other than English—and realistically animates a wide range of adult faces. Conditioning on subject labels during training allows the model to learn a variety of realistic speaking styles. VOCA also provides animator controls to alter speaking style, identity-dependent facial shape, and pose (i.e. head, jaw, and eyeball rotations) during animation. To our knowledge, VOCA is the only realistic 3D facial animation model that is readily applicable to unseen subjects without retargeting. This makes VOCA suitable for tasks like in-game video, virtual reality avatars, or any scenario in which the speaker, speech, or language is not known in advance. We make the dataset and model available for research purposes at http://voca.is.tue.mpg.de.


An extension to the quasi-static Rayleigh fading channel has been addressed in #REFR , where a scheme based on low-density parity check (LDPC) codes using a belief propagation decoder has been proposed.

Since its initial conceptualization in #OTHEREFR , the URA paradigm has attracted considerable interest from the research community, giving rise to many candidate solutions tailored to various communication scenarios.

Earlier works such as #OTHEREFR focus on information-theoretic limits and propose coding schemes for the additive white Gaussian noise (AWGN) channel setting.

Later on, several practical schemes based on compressed sensing have been proposed for the same setting #OTHEREFR , #OTHEREFR .

More recently the focus has shifted towards coding schemes for the massive MIMO (mMIMO) regime.

Initially, the URA problem on a Rayleigh block-fading channel in a mMIMO setting was formulated in #OTHEREFR .

The authors combined the coded compressed-sensig (CCS) paradigm with an outer tree code #OTHEREFR , and a covariance-based activity detection (AD) algorithm, showing the capability of achieving sum spectral efficiencies that scale with the coherence blocklength.

In the operating regime where the coherence blocklength exceeds the number of active users, the state-of-the-art approaches for URA leverage the large blocklength to device two-step approaches, in which communication resources are split between user activity detection and channel estimation on one hand, and coherent detection on the other hand.

Specifically, #OTHEREFR proposed the first pilot-aided communication scheme that employs multi-measurement-vector approximate message passing (MMV-AMP) for user activity detection and channel estimation, in combination with maximum ratio combining (MRC) outer Polar channel coding.

Bisparcs For Unsourced Random Access In Massive Mimo

Abstract: This paper considers the massive MIMO unsourced random access problem in a quasi-static Rayleigh fading setting. The proposed coding scheme is based on a concatenation of a "conventional" channel code (such as, e.g., LDPC) serving as an outer code, and a sparse regression code (SPARC) serving as an inner code. The scheme combines channel estimation, singleuser decoding, and successive interference cancellation in a novel way. The receiver performs joint channel estimation and SPARC decoding via an instance of a bilinear generalized approximate message passing (BiGAMP) based algorithm, which leverages the intrinsic bilinear structure that arises in the considered communication regime. The detection step is followed by a per-user soft-input-soft-output (SISO) decoding of the outer channel code in combination with a successive interference cancellation (SIC) step. We show via numerical simulation that the resulting scheme achieves stat-of-the-art performance in the massive connectivity setting, while attaining comparatively low implementation complexity. Index Terms-Internet of Things (IoT), Unsourced Random Access, bilinear generalized approximate message passing (BiGAMP), massive MIMO.

Energy Efficient Coded Random Access For The Wireless Uplink

Abstract: We discuss the problem of designing channel access architectures for enabling fast, low-latency, grant-free and uncoordinated uplink for densely packed wireless nodes. Specifically, we study random-access codes, previously introduced for the AWGN multiple-access channel (MAC) by Polyanskiy'2017, in the practically more relevant case of users subject to Rayleigh fading, when channel gains are unknown to the decoder. We propose a random coding achievability bound, which we analyze both non-asymptotically (at finite blocklength) and asymptotically. As a candidate practical solution, we propose an explicit sparse-graph based coding scheme together with an alternating belief-propagation decoder. The latter's performance is found to be surprisingly close to the finite-blocklength bounds. Our main findings are twofold. First, just like in the AWGN MAC we see that jointly decoding large number of users leads to a surprising phase transition effect, where at spectral efficiencies below a critical threshold (5-15 bps/Hz depending on reliability) a perfect multi-user interference cancellation is possible. Second, while the presence of Rayleigh fading significantly increases the minimal required energy-per-bit $E_b/N_0$ (from about 0-2 dB to about 8-11 dB), the inherent randomization introduced by the channel makes it much easier to attain the optimal performance via iterative schemes. ::: In all, it is hoped that a principled definition of the random-access model together with our information-theoretic analysis will open the road towards unified benchmarking and comparison performance of various random-access solutions, such as the currently discussed candidates (MUSA, SCMA, RSMA) for the 5G/6G.


Continuing the previous work and with the same belief of developing the meta-learner without forgetting base classes, #REFR proposed a denoising auto-encoder network (DAE) in the form of graph neural network (GNN), which is responsible for regularizing the weight generating meta-model.

[82] proposed a dynamic algorithm by introducing an attention-based classification weight generator and a cosinesimilarity based ConvNet recognition model.

The former is responsible for producing weights vectors for unseen classes by seeing a few examples from those classes.

The authors argued that it is not feasible for a ConvNet model with a typical linear layer classifier to handle classification weights of both base and novel classes.

They address this issue by proposing the second component that provides prediction by computing the cosine similarity between the generated weights and samples' representations.

The combination of these two components allowed the authors to achieve fast adaptation for novel classes without forgetting the base classes.

Basically, DAE takes contaminated classification weights with Gaussian noise and generates classification weights based on the conditional distribution of seen data from both novel and base classes.

Results showed that this model outperforms their previous model in #OTHEREFR .

#OTHEREFR argued that embedding vectors are comparable with the weights of the last linear layer of the ConvNet classifier.

Accordingly, instead of using two different components of meta-learner and base learner, they proposed a new process called weight imprinting that basically sets final layer weights based on normalized embeddings of the novel class's training samples.

Figure 19 shows the overall flow of their model, in which an embedding function first trains on base classes then tunes on novel classes where imprint weights are used in the final layers.

Meta-Learning Approaches For Few-Shot Learning: A Survey Of Recent Advances

Abstract: Despite its astounding success in learning deeper multi-dimensional data, the performance of deep learning declines on new unseen tasks mainly due to its focus on same-distribution prediction. Moreover, deep learning is notorious for poor generalization from few samples. Meta-learning is a promising approach that addresses these issues by adapting to new tasks with few-shot datasets. This survey first briefly introduces meta-learning and then investigates state-of-the-art meta-learning methods and recent advances in: (i) metric-based, (ii) memory-based, (iii), and learning-based methods. Finally, current challenges and insights for future researches are discussed.

Generating Classification Weights With Gnn Denoising Autoencoders For Few-Shot Learning

Abstract: Given an initial recognition model already trained on a set of base classes, the goal of this work is to develop a meta-model for few-shot learning. The meta-model, given as input some novel classes with few training examples per class, must properly adapt the existing recognition model into a new model that can correctly classify in a unified way both the novel and the base classes. To accomplish this goal it must learn to output the appropriate classification weight vectors for those two types of classes. To build our meta-model we make use of two main innovations: we propose the use of a Denoising Autoencoder network (DAE) that (during training) takes as input a set of classification weights corrupted with Gaussian noise and learns to reconstruct the target-discriminative classification weights. In this case, the injected noise on the classification weights serves the role of regularizing the weight generating meta-model. Furthermore, in order to capture the co-dependencies between different classes in a given task instance of our meta-model, we propose to implement the DAE model as a Graph Neural Network (GNN). In order to verify the efficacy of our approach, we extensively evaluate it on ImageNet based few-shot benchmarks and we report strong results that surpass prior approaches. The code and models of our paper will be published on: this https URL


We first adopted one of the most representative benchmark datasets DBP15K(EN-FR) #REFR , which is widely adopted in previous works.

To measure the real effect of the existing EA methods on HHKGs, we also conducted extensive experiments on our new proposed datasets: ICEWS-WIKI and ICEWS-YAGO.

The statistics of these selected datasets are summarized in Table 1 .

Rethinking Gnn-Based Entity Alignment On Heterogeneous Knowledge Graphs: New Datasets And A New Method

Abstract: The development of knowledge graph (KG) applications has led to a rising need for entity alignment (EA) between heterogeneous KGs that are extracted from various sources. Recently, graph neural networks (GNNs) have been widely adopted in EA tasks due to GNNs' impressive ability to capture structure information. However, we have observed that the oversimplified settings of the existing common EA datasets are distant from real-world scenarios, which obstructs a full understanding of the advancements achieved by recent methods. This phenomenon makes us ponder: Do existing GNN-based EA methods really make great progress? In this paper, to study the performance of EA methods in realistic settings, we focus on the alignment of highly heterogeneous KGs (HHKGs) (e.g., event KGs and general KGs) which are different with regard to the scale and structure, and share fewer overlapping entities. First, we sweep the unreasonable settings, and propose two new HHKG datasets that closely mimic real-world EA scenarios. Then, based on the proposed datasets, we conduct extensive experiments to evaluate previous representative EA methods, and reveal interesting findings about the progress of GNN-based EA methods. We find that the structural information becomes difficult to exploit but still valuable in aligning HHKGs. This phenomenon leads to inferior performance of existing EA methods, especially GNNbased methods. Our findings shed light on the potential problems resulting from an impulsive application of GNN-based methods as a panacea for all EA datasets. Finally, we introduce a simple but effective method: Simple-HHEA, which comprehensively utilizes entity name, structure, and temporal information. Experiment results show Simple-HHEA outperforms previous models on HHKG datasets.

A Benchmarking Study Of Embedding-Based Entity Alignment For Knowledge Graphs

Abstract: Entity alignment seeks to find entities in different knowledge graphs (KGs) that refer to the same real-world object. Recent advancement in KG embedding impels the advent of embedding-based entity alignment, which encodes entities in a continuous embedding space and measures entity similarities based on the learned embeddings. In this paper, we conduct a comprehensive experimental study of this emerging field. This study surveys 23 recent embedding-based entity alignment approaches and categorizes them based on their techniques and characteristics. We further observe that current approaches use different datasets in evaluation, and the degree distributions of entities in these datasets are inconsistent with real KGs. Hence, we propose a new KG sampling algorithm, with which we generate a set of dedicated benchmark datasets with various heterogeneity and distributions for a realistic evaluation. This study also produces an open-source library, which includes 12 representative embedding-based entity alignment approaches. We extensively evaluate these approaches on the generated datasets, to understand their strengths and limitations. Additionally, for several directions that have not been explored in current approaches, we perform exploratory experiments and report our preliminary findings for future studies. The benchmark datasets, open-source library and experimental results are all accessible online and will be duly maintained.


Bootstrapping #REFR combines the noisy label and the DNN prediction as a soft pseudo-label to replace the original label.

Supervised training of deep learning models requires precisely labeled datasets and noisy labels can significantly degrade the generalization of models.

Therefore, various methods were proposed to improve robustness against noisy labels.

The typical algorithms for combating noisy labels including adding regularization #OTHEREFR , designing robust loss functions #OTHEREFR , selecting possible clean samples #OTHEREFR and correcting the labels #OTHEREFR .

We mainly introduce the label correction approaches which are most related to this work.

Specifically, label correction algorithms aim to identify and correct suspicious labels in an iterative framework.

Furthermore, Dynamic bootstrapping #OTHEREFR use a Beta-Mixture-Model to combine label dynamically for every sample.

Differently, Joint Optimization #OTHEREFR trained their model with a large learning rate for several epochs firstly, and then generated pseudo-labels by averaging the predictions of that epochs.

Recently, SELFIE #OTHEREFR and Ada-Corr #OTHEREFR selectively combine the labels of noisy data for corresponding to true labels with a high probability.

However, almost all existing noisy label studies, focusing on the classification task, cannot be adapted to cross-modal matching directly in which the noisy label refers to the alignment errors between data pairs rather than the categorical labeling errors.

The label of cross-modal matching can be consider as y i ∈ {0, 1}, where data pair is positively correlated (y i = 1) or not (y i = 0).

Bicro: Noisy Correspondence Rectification For Multi-Modality Data Via Bi-Directional Cross-Modal Similarity Consistency

Abstract: As one of the most fundamental techniques in multimodal learning, cross-modal matching aims to project various sensory modalities into a shared feature space. To achieve this, massive and correctly aligned data pairs are required for model training. However, unlike unimodal datasets, multimodal datasets are extremely harder to collect and annotate precisely. As an alternative, the co-occurred data pairs (e.g., image-text pairs) collected from the Internet have been widely exploited in the area. Unfortunately, the cheaply collected dataset unavoidably contains many mismatched data pairs, which have been proven to be harmful to the model's performance. To address this, we propose a general framework called BiCro (Bidirectional Cross-modal similarity consistency), which can be easily integrated into existing cross-modal matching models and improve their robustness against noisy data. Specifically, BiCro aims to estimate soft labels for noisy data pairs to reflect their true correspondence degree. The basic idea of BiCro is motivated by that-taking image-text matching as an example-similar images should have similar textual descriptions and vice versa. Then the consistency of these two similarities can be recast as the estimated soft labels to train the matching model. The experiments on three popular cross-modal matching datasets demonstrate that our method significantly improves the noise-robustness of various matching models, and surpass the state-of-the-art by a clear margin.The code is available at https://github.com/xu5zhao/BiCro.

Training Deep Neural Networks On Noisy Labels With Bootstrapping

Abstract: Current state-of-the-art deep learning systems for visual object recognition and detection use purely supervised training with regularization such as dropout to avoid overfitting. The performance depends critically on the amount of labeled examples, and in current practice the labels are assumed to be unambiguous and accurate. However, this assumption often does not hold; e.g. in recognition, class labels may be missing; in detection, objects in the image may not be localized; and in general, the labeling may be subjective. In this work we propose a generic way to handle noisy and incomplete labeling by augmenting the prediction objective with a notion of consistency. We consider a prediction consistent if the same prediction is made given similar percepts, where the notion of similarity is between deep network features computed from the input data. In experiments we demonstrate that our approach yields substantial robustness to label noise on several datasets. On MNIST handwritten digits, we show that our model is robust to label corruption. On the Toronto Face Database, we show that our model handles well the case of subjective labels in emotion recognition, achieving state-of-the- art results, and can also benefit from unlabeled face images with no modification to our method. On the ILSVRC2014 detection challenge data, we show that our approach extends to very deep networks, high resolution images and structured outputs, and results in improved scalable detection.


With increasing demands and diverse services, several other projects like SAT5G and SATis5 are involving in NTN technologies, especially in LEOs #REFR .

Low Earth orbit (LEO) satellite constellation has been developed rapidly, bringing a large number of prospects in sixthgeneration (6G) non-terrestrial network (NTN) communications #OTHEREFR - #OTHEREFR .

Lots of global organization and industry, such as SpaceX, Amazon, OneWeb, and TeleSAT have announced investment for extensive services and advanced researches to provide globalcoverage applications #OTHEREFR - #OTHEREFR .

The NTN specification is specified by 3rd Generation Partnership Project (3GPP) since Release 15, aiming for global-coverage and throughput-aware services #OTHEREFR .

LEO satellites possess several benefits compared to the conventional satellite communications of medium Earth orbit (MEO), high Earth orbit (HEO) and geostationary orbit (GEO) #OTHEREFR .

Comparatively, lower LEO height allows lower latency, alleviated pathloss and lower deployment cost.

To conquer the huge pathloss compared to terrestrial channels, LEOs have to conduct multiple narrow beamforming to concentrate power with high beam-gain to the desired users in several directions #OTHEREFR .

Moreover, massive LEOs are deployed and controlled by the ground gateway to support seamless coverage not covered by terrestrial infrastructure #OTHEREFR , #OTHEREFR .

Nevertheless, under such dense LEO networks, limited resources should be appropriately assigned to prevent severe backdrops of inter-LEO, inter-beam and Doppler-induced interferences from high-mobility #OTHEREFR - #OTHEREFR , which becomes compellingly imperative.

Hierarchical Multi-Agent Multi-Armed Bandit For Resource Allocation In Multi-Leo Satellite Constellation Networks

Abstract: Low Earth orbit (LEO) satellite constellation is capable of providing global coverage area with high-rate services in the next sixth-generation (6G) non-terrestrial network (NTN). Due to limited onboard resources of operating power, beams, and channels, resilient and efficient resource management has become compellingly imperative under complex interference cases. However, different from conventional terrestrial base stations, LEO is deployed at considerable height and under high mobility, inducing substantially long delay and interference during transmission. As a result, acquiring the accurate channel state information between LEOs and ground users is challenging. Therefore, we construct a framework with a twoway transmission under unknown channel information and no data collected at long-delay ground gateway. In this paper, we propose hierarchical multi-agent multi-armed bandit resource allocation for LEO constellation (mmRAL) by appropriately assigning available radio resources. LEOs are considered as collaborative multiple macro-agents attempting unknown trials of various actions of microagents of respective resources, asymptotically achieving suitable allocation with only throughput information. In simulations, we evaluate mmRAL in various cases of LEO deployment, serving numbers of users and LEOs, hardware cost and outage probability. Benefited by efficient and resilient allocation, the proposed mmRAL system is capable of operating in homogeneous or heterogeneous orbital planes or constellations, achieving the highest throughput performance compared to the existing benchmarks in open literature.

Satellite Communications In The New Space Era: A Survey And Future Challenges

Abstract: Satellite communications have recently entered a period of renewed interest motivated by technological advances and nurtured through private investment and ventures. The present survey aims at capturing the state of the art in SatComs, while highlighting the most promising open research topics. Firstly, the main innovation drivers are motivated, such as new constellation types, on-board processing capabilities, nonterrestrial networks and space-based data collection/processing. Secondly, the most promising applications are described i.e. 5G integration, space communications, Earth observation, aeronautical and maritime tracking and communication. Subsequently, an in-depth literature review is provided across five axes: i) system aspects, ii) air interface, iii) medium access, iv) networking, v) testbeds & prototyping. Finally, a number of future challenges and the respective open research topics are described.


Our approach to solve the above problem is based on an end-to-end trainable framework that includes a differentiable optimization layer #REFR formulated by the proposed att-CLF based QP.

Given (i) a sensor measurement z z z ∈ R d (e.g., front-view RGB images of the vehicle in this work), where d ∈ N is its dimension, (ii) vehicle dynamics (1), (iii) a nominal controller h (x x x, z z z) = u u u (such as a model predictive controller), and (iv) a learning-based end-to-end perception-to-control model h(z z z|θ ) = u u u, parametrized by θ , our goal is to find optimal parameters:

where l is a similarity measure (or a loss function).

We also wish to equip the learning-based endto-end controller h with stability properties which we will describe in the following. Approach.

The proposed att-CLF is observation dependent, and thus, it is adaptive to changing environments.

The model takes front-view RGB images z z z as inputs whose features are extracted by CNN.

In order to enforce temporal coherency and prevent the model from being susceptible to sudden input change (e.g., affected by sunlight), we employ LSTM after the CNN.

The outputs of the LSTM are used to construct the att-CLF, as well as to determine other optimization parameters in the att-CLF based QP.

The outputs of the QP are the controls u u u of the vehicle.

Learning Stability Attention In Vision-Based End-To-End Driving Policies

Abstract: Modern end-to-end learning systems can learn to explicitly infer control from perception. However, it is difficult to guarantee stability and robustness for these systems since they are often exposed to unstructured, high-dimensional, and complex observation spaces (e.g., autonomous driving from a stream of pixel inputs). We propose to leverage control Lyapunov functions (CLFs) to equip end-to-end vision-based policies with stability properties and introduce stability attention in CLFs (att-CLFs) to tackle environmental changes and improve learning flexibility. We also present an uncertainty propagation technique that is tightly integrated into att-CLFs. We demonstrate the effectiveness of att-CLFs via comparison with classical CLFs, model predictive control, and vanilla end-to-end learning in a photo-realistic simulator and on a real full-scale autonomous vehicle.

Optnet: Differentiable Optimization As A Layer In Neural Networks

Abstract: This paper presents OptNet, a network architecture that integrates optimization problems (here, specifically in the form of quadratic programs) as individual layers in larger end-to-end trainable deep networks. These layers allow complex dependencies between the hidden states to be captured that traditional convolutional and fully-connected layers are not able to capture. In this paper, we develop the foundations for such an architecture: we derive the equations to perform exact differentiation through these layers and with respect to layer parameters; we develop a highly efficient solver for these layers that exploits fast GPU-based batch solves within a primal-dual interior point method, and which provides backpropagation gradients with virtually no additional cost on top of the solve; and we highlight the application of these approaches in several problems. In one particularly standout example, we show that the method is capable of learning to play Sudoku given just input and output games, with no a priori information about the rules of the game; this task is virtually impossible for other neural network architectures that we have experimented with, and highlights the representation capabilities of our approach.


While C-OVD is not our primary concern, we include the corresponding methods for reference and report their performance when only COCO Captions #REFR is available.

For completeness, we include two related benchmarks: Zero-Shot Detection (ZSD) #OTHEREFR and Weakly Supervised Detection (WSD) #OTHEREFR .

Our method mainly focuses on the V-OVD benchmark and the G-OVD benchmark. As shown in tab.

2, our OADP framework achieves 30.0 mAP N 50 on the V-OVD benchmark.

RegionCLIP* #OTHEREFR uses CLIP #OTHEREFR as the pretrained weight, thus adhering to the V-OVD benchmark.

Some V-OVD methods #OTHEREFR are not included because they rely on large-scale detection and image-text datasets and cannot be compared fairly.

Under such constraint, the performance of the caption-based methods is relatively lower than the V-OVD methods, even if additional caption data is used.

Moreover, our OADP framework is perpendicular to the caption-based methods and has the potential to achieve higher performance using captions.

For the G-OVD benchmark, we generate pseudo labels for novel categories as described in sec.

#OTHEREFR Most of the methods under the OV-LVIS setting adhere to the V-OVD benchmark, so we conduct experiments on the V-OVD benchmark only.

For the object detection task, our OADP framework achieves 21.9 AP r , surpassing DetPro #OTHEREFR by 1.1 AP r .

Object-Aware Distillation Pyramid For Open-Vocabulary Object Detection

Abstract: Open-vocabulary object detection aims to provide object detectors trained on a fixed set of object categories with the generalizability to detect objects described by arbitrary text queries. Previous methods adopt knowledge distillation to extract knowledge from Pretrained Vision-and-Language Models (PVLMs) and transfer it to detectors. However, due to the non-adaptive proposal cropping and single-level feature mimicking processes, they suffer from information destruction during knowledge extraction and inefficient knowledge transfer. To remedy these limitations, we propose an Object-Aware Distillation Pyramid (OADP) framework, including an Object-Aware Knowledge Extraction (OAKE) module and a Distillation Pyramid (DP) mechanism. When extracting object knowledge from PVLMs, the former adaptively transforms object proposals and adopts object-aware mask attention to obtain precise and complete knowledge of objects. The latter introduces global and block distillation for more comprehensive knowledge transfer to compensate for the missing relation information in object distillation. Extensive experiments show that our method achieves significant improvement compared to current methods. Especially on the MS-COCO dataset, our OADP framework reaches 35.6 mAP N 50 , surpassing the current state-of-the-art method by 3.3 mAP N 50. Code is released at https://github.com/LutingWang/OADP.

Microsoft Coco Captions: Data Collection And Evaluation Server

Abstract: In this paper we describe the Microsoft COCO Caption dataset and evaluation server. When completed, the dataset will contain over one and a half million captions describing over 330,000 images. For the training and validation images, five independent human generated captions will be provided. To ensure consistency in evaluation of automatic caption generation algorithms, an evaluation server is used. The evaluation server receives candidate captions and scores them using several popular metrics, including BLEU, METEOR, ROUGE and CIDEr. Instructions for using the evaluation server are provided.


They first divided an image into 64 × 64 patches, predicted fake patches by their model, and calculated the threshold by dividing the number of fake patches to the total number of patches #REFR . In 2020, Jane et al.

For analyzing image textures and recording long-term information about image textures, introduced the Gram-Net network, derived by adding 6 Gram blocks to the ResNet18 architecture.

Each Gram block in the ResNet18 architecture contained one convolution layer, one Gram matrix #OTHEREFR layer to extract image texture properties, two convolution layers, batch normalization, Relu, and one pooling layer #OTHEREFR .

Some works #OTHEREFR used facial patches for diagnosis. In 2018, Jain et al.

introduced an architecture consisting of 5 convolutional layers and 2 fully connected layers, and a wide residual connection in order to add the output of the second layer to the fifth layer.

A thresholdbased approach and prediction based on threshold were used to classify images as real or fake.

#OTHEREFR introduced a three-level patch-based approach for fake detection.In the first level, they divided an image into 64 × 64 blocks, and obtained a vector for each image.

They also trained a SVM with RBF kernel with these vectors to determine whether an image was real or altered.

On the second level, weights of the previous layer were frozen, and if an image has been altered, the network detected that images have been retouched or generated by a GAN.

Finally, in the third level, when an image was produced by a GAN, weights of the previous level were frozen, and at this level, the type of GANs was detected.

Bharati et al, in 2016, proposed considering four facial patches, including the right periocular, left periocular, nose, and mouth. They trained a Deep Boltzmann machine for each patch.

Deepfake Detection Of Occluded Images Using A Patch-Based Approach

Abstract: DeepFake involves the use of deep learning and artificial intelligence techniques to produce or change video and image contents typically generated by GANs. Moreover, it can be misused and leads to fictitious news, ethical and financial crimes, and also affects the performance of facial recognition systems. Thus, detection of real or fake images is significant specially to authenticate originality of people's images or videos. One of the most important challenges in this topic is obstruction that decreases the system precision. In this study, we present a deep learning approach using the entire face and face patches to distinguish real/fake images in the presence of obstruction with a three-path decision: first entire-face reasoning, second a decision based on the concatenation of feature vectors of face patches, and third a majority vote decision based on these features. To test our approach, new datasets including real and fake images are created. For producing fake images, StyleGAN and StyleGAN2 are trained by FFHQ images and also StarGAN and PGGAN are trained by CelebA images. The CelebA and FFHQ datasets are used as real images. The proposed approach reaches higher results in early epochs than other methods and increases the SoTA results by 0.4%-7.9% in the different built data-sets. Also, we have shown in experimental results that weighing the patches may improve accuracy.

On Detecting Gans And Retouching Based Synthetic Alterations

Abstract: Digitally retouching images has become a popular trend, with people posting altered images on social media and even magazines posting flawless facial images of celebrities. Further, with advancements in Generative Adversarial Networks (GANs), now changing attributes and retouching have become very easy. Such synthetic alterations have adverse effect on face recognition algorithms. While researchers have proposed to detect image tampering, detecting GANs generated images has still not been explored. This paper proposes a supervised deep learning algorithm using Convolutional Neural Networks (CNNs) to detect synthetically altered images. The algorithm yields an accuracy of 99.65% on detecting retouching on the ND-IIITD dataset. It outperforms the previous state of the art which reported an accuracy of 87% on the database. For distinguishing between real images and images generated using GANs, the proposed algorithm yields an accuracy of 99.83%.


To address this issue, we use an Out-of-Distribution (OoD) detector #REFR to identify a subset of D P that maps very close to the private training set D i T . Figure 3 .

In traditional distillation methods, the labels generated by the teacher model on the public dataset are used to train the student model #OTHEREFR .

However, all the public dataset samples are used irrespective of their similarity to the local dataset.

The OoD detector used in our IDKD framework for decentralized training.

After local convergence, a calibration dataset (can be the public dataset or a separate dataset) and the private dataset are used as OoD data and ID data respectively. This is used to identify the optimal threshold.

OoD detector identifies the samples in the public set D P that look like In-Distribution (ID) data to the local model on each node.

Among different OoD detectors proposed in literature #OTHEREFR , we choose the maximum softmax probability (MSP) detector #OTHEREFR because of its simplicity and low computational overhead. Figure 3 visualizes the MSP detector.

The samples with confidence (i.e., softmax probability) greater than a set threshold t are classified as ID by the MSP detector The hyperparameter t i.e, threshold is tuned based on the True Positive Rate (TPR) of the OoD detector.

Homogenizing Non-Iid Datasets Via In-Distribution Knowledge Distillation For Decentralized Learning

Abstract: Decentralized learning enables serverless training of deep neural networks (DNNs) in a distributed manner on multiple nodes. This allows for the use of large datasets, as well as the ability to train with a wide variety of data sources. However, one of the key challenges with decentralized learning is heterogeneity in the data distribution across the nodes. In this paper, we propose In-Distribution Knowledge Distillation (IDKD) to address the challenge of heterogeneous data distribution. The goal of IDKD is to homogenize the data distribution across the nodes. While such data homogenization can be achieved by exchanging data among the nodes sacrificing privacy, IDKD achieves the same objective using a common public dataset across nodes without breaking the privacy constraint. This public dataset is different from the training dataset and is used to distill the knowledge from each node and communicate it to its neighbors through the generated labels. With traditional knowledge distillation, the generalization of the distilled model is reduced because all the public dataset samples are used irrespective of their similarity to the local dataset. Thus, we introduce an Out-of-Distribution (OoD) detector at each node to label a subset of the public dataset that maps close to the local training data distribution. Finally, only labels corresponding to these subsets are exchanged among the nodes and with appropriate label averaging each node is finetuned on these data subsets along with its local data. Our experiments on multiple image classification datasets and graph topologies show that the proposed IDKD scheme is more effective than traditional knowledge distillation and achieves state-of-the-art generalization performance on heterogeneously distributed data with minimal communication overhead. Figure 1. Overview of decentralized training. Here networks are connected in a ring topology. Each node has its own private dataset. This dataset is not shared between the nodes, rather model weights/gradients are shared between the neighbors during training. However, such a setup struggles to converge when private datasets are heterogeneous i.e. non-identical and independently distributed (non-IID).

A Baseline For Detecting Misclassified And Out-Of-Distribution Examples In Neural Networks

Abstract: We consider the two related problems of detecting if an example is misclassified or out-of-distribution. We present a simple baseline that utilizes probabilities from softmax distributions. Correctly classified examples tend to have greater maximum softmax probabilities than erroneously classified and out-of-distribution examples, allowing for their detection. We assess performance by defining several tasks in computer vision, natural language processing, and automatic speech recognition, showing the effectiveness of this baseline across all. We then show the baseline can sometimes be surpassed, demonstrating the room for future research on these underexplored detection tasks.


But since the delay is bounded and assumptions on the loss function hold, the sequence produced by our Distributed Gradient update rule converges to the unique minimizer. (Theorem 1 #REFR ).

Each of them will participate in the update of the global parameters, by adding his computed loss, we can conclude that the final loss of our algorithm will be :

, with L i (w) the total loss created by the DO i .

We know that the time taken by the DO to compute the gradient, depend on a finite block of interaction, so we know that there exist D < ∞, such that for any DO i , the delay d i k < D.

Assume that each L i is differentiable and that ∇L i is 1 L − cococercive.

Moreover, in our experiments, we will use the same loss function as SAROS, we can conclude that both algorithms should converge to the same minimizer.

Drift: A Federated Recommender System With Implicit Feedback On The Items

Abstract: Nowadays there are more and more items available online, this makes it hard for users to find items that they like. Recommender systems aim to find the item who best suits the user, using his historical interactions. Depending on the context, these interactions may be more or less sensitive and collecting them brings an important problem concerning the users' privacy. Federated systems have shown that it is possible to make accurate and efficient recommendations without storing users' personal information. However, these systems use instantaneous feedback from the user. In this report, we propose DRIFT, a federated architecture for recommender systems, using implicit feedback. Our learning model is based on a recent algorithm for recommendation with implicit feedbacks SAROS [4]. We aim to make recommendations as precise as SAROS, without compromising the users' privacy. In this report we show that thanks to our experiments, but also thanks to a theoretical analysis on the convergence. We have shown also that the computation time has a linear complexity with respect to the number of interactions made. Finally, we have shown that our algorithm is secure, and participants in our federated system cannot guess the interactions made by the user, except DOs that have the item involved in the interaction. Acknowledgement I would like to express my sincere gratitude to Marta Soare, Radu Ciucanu, and Massih-Reza Amini for their invaluable assistance throughout the internship. I would also like to express my sincere thanks to Aleksandra Burashnikova, for all the help she gave me in understanding her algorithm.

Asynchronous Distributed Learning With Sparse Communications And Identification

Abstract: In this paper, we present an asynchronous optimization algorithm for distributed learning, that efficiently reduces the communications between a master and working machines by randomly sparsifying the local updates. This sparsification allows to lift the communication bottleneck often present in distributed learning setups where computations are performed by workers on local data while a master machine coordinates their updates to optimize a global loss. We prove that despite its sparse asynchronous communications, our algorithm allows for a fixed stepsize and benefits from a linear convergence rate in the strongly convex case. Moreover, for $\ell_1$-regularized problems, this algorithm identifies near-optimal sparsity patterns, so that all communications eventually become sparse. We furthermore leverage on this identification to improve our sparsification technique. We illustrate on real and synthetic data that this algorithm converges faster in terms of data exchanges.


They suggest that adaptive ODE solvers, used to solve the underlying initial value problems, cause gradient obfuscation #REFR , which, in turn, causes weaker adversarial attacks to fail.

As standard neural networks suffer from severely reduced accuracies when trained for robustness, novel architectures with inherent robustness properties have recently received increasing attention #OTHEREFR .

Neural Ordinary Differential Equations One particularly interesting such architecture are neural ODEs (NODEs) .

Built around solving initial value problems with learned dynamics, they are uniquely suited to time-series-based problems #OTHEREFR but have also been successfully applied to image classification .

More importantly, NODEs have been observed to exhibit inherent robustness properties against adversarial attacks #OTHEREFR Kang et al., 2021; #OTHEREFR .

However, recently found that this robustness is greatly diminished against stronger attacks.

This highlights the need for formal robustness guarantees to rigorously evaluate the true robustness of a model or architecture.

Robustness Verification For standard neural networks, many robustness verification methods have been proposed #OTHEREFR .

One particularly successful class of such methods #OTHEREFR propagates convex shapes through the neural network that capture the reachable sets of every neuron's values and uses them to check

Efficient Certified Training And Robustness Verification Of Neural Odes

Abstract: Neural Ordinary Differential Equations (NODEs) are a novel neural architecture, built around initial value problems with learned dynamics which are solved during inference. Thought to be inherently more robust against adversarial perturbations, they were recently shown to be vulnerable to strong adversarial attacks, highlighting the need for formal guarantees. However, despite significant progress in robustness verification for standard feed-forward architectures, the verification of high dimensional NODEs remains an open problem. In this work, we address this challenge and propose GAINS, an analysis framework for NODEs combining three key ideas: (i) a novel class of ODE solvers, based on variable but discrete time steps, (ii) an efficient graph representation of solver trajectories, and (iii) a novel abstraction algorithm operating on this graph representation. Together, these advances enable the efficient analysis and certified training of high-dimensional NODEs, by reducing the runtime from an intractable O(exp(d) + exp(T)) to O(d + T 2 log 2 T) in the dimensionality d and integration time T. In an extensive evaluation on computer vision (MNIST and FMNIST) and time-series forecasting (PHYSIO-NET) problems, we demonstrate the effectiveness of both our certified training and verification methods.

Obfuscated Gradients Give A False Sense Of Security: Circumventing Defenses To Adversarial Examples

Abstract: We identify obfuscated gradients as a phenomenon that leads to a false sense of security in defenses against adversarial examples. While defenses that cause obfuscated gradients appear to defeat optimization-based attacks, we find defenses relying on this effect can be circumvented. ::: For each of the three types of obfuscated gradients we discover, we describe indicators of defenses exhibiting this effect and develop attack techniques to overcome it. In a case study, examining all defenses accepted to ICLR 2018, we find obfuscated gradients are a common occurrence, with 7 of 8 defenses relying on obfuscated gradients. Using our new attack techniques, we successfully circumvent all 7 of them.

adaptive ODE solvers
gradient obfuscation


We demonstrate the applicability of ordinal motifs on real-world data using a medium sized formal context: the spices planner data set #REFR .

This context contains 56 meals (objects) and 37 spices and food categories (attributes).

The incidence encodes that a spice is recommended to be used to cook a meal or a meal belongs to a food category. The context has 531 formal concepts.

We conduct our experiment on the dual context, i.e., K d := (M, G, I d ), to derive ordinal motifs within the spices and food categories.

For our application we employ the standard scales #OTHEREFR , as they are the most commonly used.

We recall their definitions, where [n] := {1,...,n} and K 1 | K 2 is the context apposition operator.

Ordinal Motifs In Lattices

Abstract: Lattices are a commonly used structure for the representation and analysis of relational and ontological knowledge. In particular, the analysis of these requires a decomposition of a large and high-dimensional lattice into a set of understandably large parts. With the present work we propose /ordinal motifs/ as analytical units of meaning. We study these ordinal substructures (or standard scales) through order-embeddings and (full) scale-measures of formal contexts from the field of formal concept analysis. We show that the underlying decision problems are NP-complete and provide results on how one can incrementally identify ordinal motifs to save computational effort. Accompanying our theoretical results, we demonstrate how ordinal motifs can be leveraged to achieve textual explanations based on principles from human computer interaction.

Knowledge Cores In Large Formal Contexts

Abstract: Knowledge computation tasks are often infeasible for large data sets. This is in particular true when deriving knowledge bases in formal concept analysis (FCA). Hence, it is essential to come up with techniques to cope with this problem. Many successful methods are based on random processes to reduce the size of the investigated data set. This, however, makes them hardly interpretable with respect to the discovered knowledge. Other approaches restrict themselves to highly supported subsets and omit rare and interesting patterns. An essentially different approach is used in network science, called $k$-cores. These are able to reflect rare patterns if they are well connected in the data set. In this work, we study $k$-cores in the realm of FCA by exploiting the natural correspondence to bi-partite graphs. This structurally motivated approach leads to a comprehensible extraction of knowledge cores from large formal contexts data sets.


Metric Regularity On Riemannian Manifolds

Then #REFR implies that g(p ′′ ) − g(p ′ ) e ≤ ǫd(p ′ , p ′′ ) for all p ′ , p ′′ ∈ B δ (p).

f (p ′ ) − f (p ′′ ) − Df (p)[exp −1 p p ′ − exp −1 p p ′′ ] e ≤ 1 0 Df (γ p ′′ p ′ (t))[γ p ′′ p ′ (t)] − Df (p)[exp −1 p p ′ − exp −1 p p ′′ ] e dt, p ′ , p ′′ ∈ B δ (p). (15)

On the other hand, from Proposition 4 with p = p ′′ and q = p ′ we havė

γ p ′′ p ′ (t) = exp −1 γ p ′′ p ′ (t) p ′ − exp −1 γ p ′′ p ′ (t) p ′′ , t ∈ [0, 1].

Taking into account this equality and #OTHEREFR

with p = γ p ′′ p ′ (t), (15) yields f (p ′ ) − f (p ′′ ) − Df (p)(exp −1 p p ′ − exp −1 p p ′′ ) e ≤ ǫd(p ′ , p ′′ ), p ′ , p ′′ ∈ B δ (p).

Since ǫ > 0 was chosen arbitrarily we can conclude that lip(g;p) = 0.

Finally, to obtain the desired equality just use Lemma 1 with g as in #OTHEREFR and Φ = f + F .

Inexact Newton Methods For Solving Generalized Equations On Riemannian Manifolds

Abstract: The local convergence of an inexact Newton method is studied for solving generalized equations on Riemannian manifolds by using the metric regularity property which is explored as well. Under suitable conditions and without any additional geometric assumptions, local convergence results with linear and quadratic rate and a semi-local convergence result are obtained for the proposed method. Finally, the theory can be applied to problems of finding a singularity of the sum of two vector fields.

Algorithms For The Split Variational Inequality Problem

Abstract: We propose a prototypical Split Inverse Problem (SIP) and a new variational problem, called the Split Variational Inequality Problem (SVIP), which is a SIP. It entails finding a solution of one inverse problem (e.g., a Variational Inequality Problem (VIP)), the image of which under a given bounded linear transformation is a solution of another inverse problem such as a VIP. We construct iterative algorithms that solve such problems, under reasonable conditions, in Hilbert space and then discuss special cases, some of which are new even in Euclidean space.


The input feature map is always implicitly padded on each side. We refer to Dumoulin and Visin #REFR for illustrations.

However, it remains essential to predetermine the structure of hyperbolic feature maps.

Extension to the transposed setting The transposed convolutional layer is usually used in encoderdecoder architectures for up-sampling.

It is straightforward to generalize to hyperbolic space, as it can be defined using a standard convolutional layer.

A convolutional layer carries out a transposed convolution when the correct local connectivity is established by inserting zeros at certain positions.

Specifically, when stride s > 1, then s − 1 zero vectors are inserted between the features.

Under this relationship, the Lorentz transposed convolutional layer is a Lorentz convolutional layer with changed connectivity through origin padding.

Hyperbolic Geometry In Computer Vision: A Novel Framework For Convolutional Neural Networks

Abstract: Real-world visual data exhibit intrinsic hierarchical structures that can be represented effectively in hyperbolic spaces. Hyperbolic neural networks (HNNs) are a promising approach for learning feature representations in such spaces. However, current methods in computer vision rely on Euclidean backbones and only project features to the hyperbolic space in the task heads, limiting their ability to fully leverage the benefits of hyperbolic geometry. To address this, we present HCNN, the first fully hyperbolic convolutional neural network (CNN) designed for computer vision tasks. Based on the Lorentz model, we generalize fundamental components of CNNs and propose novel formulations of the convolutional layer, batch normalization, and multinomial logistic regression (MLR). Experimentation on standard vision tasks demonstrates the effectiveness of our HCNN framework and the Lorentz model in both hybrid and fully hyperbolic settings. Overall, we aim to pave the way for future research in hyperbolic computer vision by offering a new paradigm for interpreting and analyzing visual data. Our code is publicly available at https://github.com/kschwethelm/HyperbolicCV. * denotes equal contribution Preprint. Under review.

A Guide To Convolution Arithmetic For Deep Learning

Abstract: We introduce a guide to help deep learning practitioners understand and manipulate convolutional neural network architectures. The guide clarifies the relationship between various properties (input shape, kernel shape, zero padding, strides and output shape) of convolutional, pooling and transposed convolutional layers, as well as the relationship between convolutional and transposed convolutional layers. Relationships are derived for various cases, and are illustrated in order to make them intuitive.