Like in Astec, this is achieved in O(log(L)) time per data point using an ANN data structure #REFR .

Specifically, the goal is now to train label encoder Z l to minimize the mean squared error between feature representations x i (obtained from Step 1) and label embeddingl j if the j th label is positive for the data point i.

By letting P i be the set of all positive labels for the i th data point, i.e., P = {j :

Step 3, LightDXML finds the top k = O(log(L)) most hard negative labels for each data point using the shortlist S k by finding the k nearest label pointsŷ j to the feature representation x i .

Using S k , we now have access to both positive and negative labels for a data point.

Step 4, making use of S k , LightDXML learns the onevs-all classifier Z by jointly learning data feature embeddingŝ Z x , and training on the union of set of positive labels and a set of shortlisted negative labels (instead of training on all labels), i.e., argmin Zx,ZL

(Ẑ x , Z) = N i=1 j∈P i ∪S k (x i ) (x i , y ij ;Ẑ x , Z).

(3) It should be noted that Problem (3) is more efficient to solve than the original problem (1), due to training on a much reduced set of labels.

Light-Weight Deep Extreme Multilabel Classification

Abstract: Extreme multi-label (XML) classification refers to the task of supervised multi-label learning that involves a large number of labels. Hence, scalability of the classifier with increasing label dimension is an important consideration. In this paper, we develop a method called LightDXML which modifies the recently developed deep learning based XML framework by using label embeddings instead of feature embedding for negative sampling and iterating cyclically through three major phases: (1) proxy training of label embeddings (2) shortlisting of labels for negative sampling and (3) final classifier training using the negative samples. Consequently, LightDXML also removes the requirement of a re-ranker module, thereby, leading to further savings on time and memory requirements. The proposed method achieves the best of both worlds: while the training time, model size and prediction times are on par or better compared to the tree-based methods, it attains much better prediction accuracy that is on par with the deep learning based methods. Moreover, the proposed approach achieves the best tail-label prediction accuracy over most state-of-the-art XML methods on some of the large datasets 1. Code: https://github.com/misterpawan/ LightDXML

Efficient And Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs

Abstract: We present a new approach for the approximate K-nearest neighbor search based on navigable small world graphs with controllable hierarchy (Hierarchical NSW, HNSW). The proposed solution is fully graph-based, without any need for additional search structures, which are typically used at the coarse search stage of the most proximity graph techniques. Hierarchical NSW incrementally builds a multi-layer structure consisting from hierarchical set of proximity graphs (layers) for nested subsets of the stored elements. The maximum layer in which an element is present is selected randomly with an exponentially decaying probability distribution. This allows producing graphs similar to the previously studied Navigable Small World (NSW) structures while additionally having the links separated by their characteristic distance scales. Starting search from the upper layer together with utilizing the scale separation boosts the performance compared to NSW and allows a logarithmic complexity scaling. Additional employment of a heuristic for selecting proximity graph neighbors significantly increases performance at high recall and in case of highly clustered data. Performance evaluation has demonstrated that the proposed general metric space search index is able to strongly outperform previous opensource state-of-the-art vector-only approaches. Similarity of the algorithm to the skip list structure allows straightforward balanced distributed implementation.


Discrete prompt tuning transforms the task into a "fill-in-the-blank" format and then utilizes a pre-trained language model to predict the answer, which operates similarly to the masked language model (MLM) #REFR .

Prompt-based methods can be classified into two categories: discrete prompt tuning (Radford et al., 2018; #OTHEREFR and continuous prompt tuning #OTHEREFR Liu et al., 2021b) .

In subsequent research, a recent study #OTHEREFR introduced soft prompts (i.e., prompt embeddings) to replace manual templates, which consist of special tokens with adjustable embeddings.

We refer to this method as "prompt tuning" for simplicity.

Vanilla prompt tuning, which concatenates prompt embeddings with input tokens in the first layer and updates only the parameters of the prompt embeddings during the training phase, has several limitations #OTHEREFR Su et al., 2021) .

Firstly, since the effectiveness of prompt embeddings is highly related to the length, it is necessary to use prompt embeddings with hundreds of tokens in length to achieve better downstream task performance, as suggested by #OTHEREFR and (Su et al., 2021) .

However, longer prompt embeddings also reduce the length space of normal input text.

Global Prompt Cell: A Portable Control Module For Effective Prompt Tuning

Abstract: As a novel approach to tuning pre-trained models, prompt tuning involves freezing the parameters in downstream tasks while inserting trainable embeddings into inputs in the first layer. However, previous methods have mainly focused on the initialization of prompt embeddings. The question of how to train and utilize prompt embeddings in a reasonable way has become a limiting factor in the effectiveness of prompt tuning. To address this issue, we introduce the Global Prompt Cell (GPC), a portable control module for prompt tuning that selectively preserves prompt information across all encoder layers. Our experimental results demonstrate a 5.8% improvement on SuperGLUE datasets compared to vanilla prompt tuning.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Discrete prompt tuning
pre-trained language model


Other improvements to deep clustering include using optimal transport methods in feature space to create more informative clusters #REFR .

One can learn rich features by grouping semantically similar images together.

K-means clustering is one of the most widely used methods from classical machine learning.

A number of studies have adapted k-means to perform SSL with neural models.

Deep clustering alternates between assigning labels to images by performing k-means in the feature space, and updating the model to respect these assigned class labels #OTHEREFR .

More recent treatments of this approach use mean-shift updates to push features towards their cluster center, and have been shown to complement BYOL, a method based on two networks with the objective to predict pseudo-labels for each sample (discussed in Section 2.3).

A Cookbook Of Self-Supervised Learning

Self-Labelling Via Simultaneous Clustering And Representation Learning

Abstract: Combining clustering and representation learning is one of the most promising approaches for unsupervised learning of deep neural networks. However, doing so naively leads to ill posed learning problems with degenerate solutions. In this paper, we propose a novel and principled learning formulation that addresses these issues. The method is obtained by maximizing the information between labels and input data indices. We show that this criterion extends standard cross-entropy minimization to an optimal transport problem, which we solve efficiently for millions of input images and thousands of labels using a fast variant of the Sinkhorn-Knopp algorithm. The resulting method is able to self-label visual data so as to train highly competitive image representations without manual labels. Compared to the best previous method in this class, namely DeepCluster, our formulation minimizes a single objective function for both representation learning and clustering; it also significantly outperforms DeepCluster in standard benchmarks.


Knowledge Graph based Methods: Amongst knowledge graph based methods, #REFR diagnoses and triages performance failure issues in an online fashion by continuously building a knowledge base out of rules extracted from a random forest constructed over log data using heuristics and domain knowledge.

#OTHEREFR , #OTHEREFR uses KNN or its supervised versions to identify loglines that led to a failure.

Knowledge Mining based Methods: #OTHEREFR , #OTHEREFR takes a different approach of summarizing log events into an entityrelation knowledge graph by extracting custom entities and relationships from log lines and mining temporal and procedural dependencies between them from the overall log dump.

While this gives a more structured representation of the log summary, it is also an intuitive way of aggregating knowledge from logs, it is also a way to bridge the knowledge gap developer community who creates the log data and the site reliability engineers who typically consume the log data when investigating incidents.

However, eventually the end goal of constructing this knowledge graph representation of logs is to facilitate RCA.

While these works do provide use-cases like case-studies on RCA for this vision, but they leave ample scope of research towards a more concrete usage of this kind of knowledge mining in RCA.

#OTHEREFR constructs a system graph from the combination of KPI metrics and log data.

Based on the detected anomalies from these data sources, it extracts anomalous subgraphs from it and compares them with the normal system graph to detect the root cause.

Other works mine normal log patterns #OTHEREFR or time-weighted control flow graphs #OTHEREFR from normal executions and on estimates divergences from them to executions during ongoing failures to suggest root causes.

#OTHEREFR , #OTHEREFR , #OTHEREFR mines execution sequences or user actions #OTHEREFR either from normal and manually injected failures or from good or bad performing systems, in a knowledge base and utilizes the assumption that similar faults generate similar failures to match and diagnose type of failure.

Most of these knowledge based approaches incrementally expand their knowledge or rules to cater to newer incident types over time.

Ai For It Operations (Aiops) On Cloud Platforms: Reviews, Opportunities And Challenges

Abstract: Intelligence for IT operations (AIOps) aims to combine the power of AI with the big data generated by IT Operations processes, particularly in cloud infrastructures, to provide actionable insights with the primary goal of maximizing availability. There are a wide variety of problems to address, and multiple use-cases, where AI capabilities can be leveraged to enhance operational efficiency. Here we provide a review of the AIOps vision, trends challenges and opportunities, specifically focusing on the underlying AI techniques. We discuss in depth the key types of data emitted by IT Operations activities, the scale and challenges in analyzing them, and where they can be helpful. We categorize the key AIOps tasks as-incident detection, failure prediction, root cause analysis and automated actions. We discuss the problem formulation for each task, and then present a taxonomy of techniques to solve these problems. We also identify relatively under explored topics, especially those that could significantly benefit from advances in AI literature. We also provide insights into the trends in this field, and what are the key investment opportunities.

Decaf: Diagnosing And Triaging Performance Issues In Large-Scale Cloud Services

Abstract: Large scale cloud services use Key Performance Indicators (KPIs) for tracking and monitoring performance. They usually have Service Level Objectives (SLOs) baked into the customer agreements which are tied to these KPIs. Dependency failures, code bugs, infrastructure failures, and other problems can cause performance regressions. It is critical to minimize the time and manual effort in diagnosing and triaging such issues to reduce customer impact. Large volumes of logs and mixed type of attributes (categorical, continuous) make any automated or manual diagnosing non-trivial. ::: In this paper, we present the design, implementation and experience from building and deploying DeCaf, a system for automated diagnosis and triaging of KPI issues using service logs. It uses machine learning along with pattern mining to help service owners automatically root cause and triage performance issues. We present the learnings and results from case studies on two large scale cloud services in Microsoft where DeCaf successfully diagnosed 10 known and 31 unknown issues. DeCaf also automatically triages the identified issues by leveraging historical data. Our key insights are that for any such diagnosis tool to be effective in practice, it should a) scale to large volumes of service logs and attributes, b) support different types of KPIs and ranking functions, c) be integrated into the DevOps processes.


The attention mechanism #REFR has been used extensively in DL, especially NLP-related tasks, to learn what target tokens the source tokens should "look at".

We refer readers to recent surveys #OTHEREFR , #OTHEREFR for a thorough review of these works.

Lately, deep transformers demonstrate superior performance than other DL models on 1D sequential data, and multiple visualization works have been introduced for their interpretations #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR .

The success of transformers has also been extended to 2D images with the seminal work of vision transformers (ViTs) #OTHEREFR .

However, to the best of our knowledge, no comprehensive visual analyses have been conducted to demystify this type of powerful yet complex models, especially how attention works in the 2D image context. Our work tries to fill this gap.

Essentially, attention is a matrix where each cell denotes the attention magnitude that the source token (row) pays to the target (column).

Popular attention visualization techniques include flow maps #OTHEREFR , #OTHEREFR , parallel coordinates plots (PCPs) #OTHEREFR , and heatmaps #OTHEREFR , #OTHEREFR , #OTHEREFR .

For example, the flow maps used by Dong et al.

#OTHEREFR connect the source and target tokens with curves, the widths of which denote the attention strengths.

Vig #OTHEREFR arranges the source and target tokens along two parallel axes (i.e., a simplified PCP) and connects them with line segments in between to show the attention patterns.

How Does Attention Work In Vision Transformers? A Visual Analytics Attempt

Abstract: Vision transformer (ViT) expands the success of transformer models from sequential data to images. The model decomposes an image into many smaller patches and arranges them into a sequence. Multi-head self-attentions are then applied to the sequence to learn the attention between patches. Despite many successful interpretations of transformers on sequential data, little effort has been devoted to the interpretation of ViTs, and many questions remain unanswered. For example, among the numerous attention heads, which one is more important? How strong are individual patches attending to their spatial neighbors in different heads? What attention patterns have individual heads learned? In this work, we answer these questions through a visual analytics approach. Specifically, we first identify what heads are more important in ViTs by introducing multiple pruning-based metrics. Then, we profile the spatial distribution of attention strengths between patches inside individual heads, as well as the trend of attention strengths across attention layers. Third, using an autoencoder-based learning solution, we summarize all possible attention patterns that individual heads could learn. Examining the attention strengths and patterns of the important heads, we answer why they are important. Through concrete case studies with experienced deep learning experts on multiple ViTs, we validate the effectiveness of our solution that deepens the understanding of ViTs from head importance, head attention strength, and head attention pattern.

Neural Machine Translation By Jointly Learning To Align And Translate

Abstract: Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.

attention mechanism
DL, especially NLP-related


C. Semi-Distributed Multi-Agent Rl For Robust Handover Optimization

In contrast, the second learning rate, η 2 , is much smaller and is utilized to slow down the degradation of Q-values associated with previously positive experiences resulting from successful operations #REFR .

This is particularly crucial in providing uninterrupted and dependable service for XR users, whose needs may change over time, and guarantees a resilient user experience that can promptly recover from any disruptions.

Moreover, the semi-distributed learning framework used in this approach ensures guaranteed convergence due to the adoption of asynchronous updates, where agents update their policy and value function independently and asynchronously.

Although the optimality of the solution may not be guaranteed in highly dynamic and non-stationary environments like THz networks, the quick adaptation of deep hysteretic networks enables the system to maintain robust and resilient performance in response to unpredictable user behavior.

Hysteretic deep recurrent Q-networks leverage two distinct learning rates to handle the complex dynamics of the learning process.

The first learning rate, η 1 , is used when the temporal difference (TD)-error is nonnegative.

By implementing this approach, hysteresis is introduced, allowing subarrays to be more resilient against negative learning, exploration, and concurrent actions.

This approach significantly improves the stability and robustness of the learning process, leading to better overall performance and expedited convergence.

Similarly to #OTHEREFR , the deep hysteretic Q-network adopted is with one input layer, two fully connected hidden layers, one RNN hidden layer, a dueling layer, and an output layer.

The subarray's local observations and the estimated state-action value Q b.n , respectively define the input and output layer of the hysteretic deep recurrent Q-network.

Multi-agent RL frameworks are known to face the challenge of shadowed equilibria, a phenomenon where local observations and non-stationarity cause locally optimal actions to become suboptimal at the global level #OTHEREFR .

Joint Sensing, Communication, And Ai: A Trifecta For Resilient Thz User Experiences

Abstract: In this paper a novel joint sensing, communication, and artificial intelligence (AI) framework is proposed so as to optimize extended reality (XR) experiences over terahertz (THz) wireless systems. The proposed framework consists of three main components. First, a tensor decomposition framework is proposed to extract unique sensing parameters for XR users and their environment by exploiting then THz channel sparsity. Essentially, THz band's quasi-opticality is exploited and the sensing parameters are extracted from the uplink communication signal, thereby allowing for the use of the same waveform, spectrum, and hardware for both communication and sensing functionalities. Then, the Cramer-Rao lower bound is derived to assess the accuracy of the estimated sensing parameters. Second, a non-autoregressive multi-resolution generative artificial intelligence (AI) framework integrated with an adversarial transformer is proposed to predict missing and future sensing information. The proposed framework offers robust and comprehensive historical sensing information and anticipatory forecasts of future environmental changes, which are generalizable to fluctuations in both known and unforeseen user behaviors and environmental conditions. Third, a multi-agent deep recurrent hysteretic Q-neural network is developed to control the handover policy of reconfigurable intelligent surface (RIS) subarrays, leveraging the informative nature of sensing information to minimize handover cost, maximize the individual quality of personal experiences (QoPEs), and improve the robustness and resilience of THz links. Simulation results show a high generalizability of the proposed unsupervised generative AI framework to fluctuations in user behavior and velocity, leading to a 61 % improvement in instantaneous reliability compared to schemes with known channel state information.

Deep Decentralized Multi-Task Multi-Agent Reinforcement Learning Under Partial Observability

Abstract: Many real-world tasks involve multiple agents with partial observability and limited communication. Learning is challenging in these settings due to local viewpoints of agents, which perceive the world as non-stationary due to concurrently-exploring teammates. Approaches that learn specialized policies for individual tasks face problems when applied to the real world: not only do agents have to learn and store distinct policies for each task, but in practice identities of tasks are often non-observable, making these approaches inapplicable. This paper formalizes and addresses the problem of multi-task multi-agent reinforcement learning under partial observability. We introduce a decentralized single-task learning approach that is robust to concurrent interactions of teammates, and present an approach for distilling single-task policies into a unified policy that performs well across multiple related tasks, without explicit provision of task identity.


Note that DPP is a probabilistic model of repulsion, which has been widely adopted for solving subset sampling problems with diversity constraints in machine learning #REFR .

With the clients' data profiles, a DPP-based CS strategy can be further devised to avoid selecting similar clients in each round of training.

And the k-DPP is a variant of DPP, with which the size of sampled subsets is fixed at k #OTHEREFR .

Particularly, a DPP is a probabilistic model over subsets on a finite set, which can be derived from a positive semi-definite similarity kernel matrix #OTHEREFR .

For a finite set M with M elements, the similarity kernel matrix L can be expressed as L = {lm,n}M×M , with lm,n representing the similarity between the m-th and n-th elements in M.

Meanwhile, the DPP assigns a probability to sub-sampling any subset Y of M, which is proportional to the determinant of the sub-matrix LY regarding the subset Y, i.e.,

Dpp-Based Client Selection For Federated Learning With Non-Iid Data

Abstract: This paper proposes a client selection (CS) method to tackle the communication bottleneck of federated learning (FL) while concurrently coping with FL's data heterogeneity issue. Specifically, we first analyze the effect of CS in FL and show that FL training can be accelerated by adequately choosing participants to diversify the training dataset in each round of training. Based on this, we leverage data profiling and determinantal point process (DPP) sampling techniques to develop an algorithm termed Federated Learning with DPP-based Participant Selection (FL-DP 3 S). This algorithm effectively diversifies the participants' datasets in each round of training while preserving their data privacy. We conduct extensive experiments to examine the efficacy of our proposed method. The results show that our scheme attains a faster convergence rate, as well as a smaller communication overhead than several baselines.

Determinantal Point Processes For Machine Learning

Abstract: Determinantal point processes (DPPs) are elegant probabilistic models of repulsion that arise in quantum physics and random matrix theory. In contrast to traditional structured models like Markov random fields, which become intractable and hard to approximate in the presence of negative correlations, DPPs offer efficient and exact algorithms for sampling, marginalization, conditioning, and other inference tasks. While they have been studied extensively by mathematicians, giving rise to a deep and beautiful theory, DPPs are relatively new in machine learning. Determinantal Point Processes for Machine Learning provides a comprehensible introduction to DPPs, focusing on the intuitions, algorithms, and extensions that are most relevant to the machine learning community, and shows how DPPs can be applied to real-world applications like finding diverse sets of high-quality search results, building informative summaries by selecting diverse sentences from documents, modeling non-overlapping human poses in images or video, and automatically building timelines of important news stories. It presents the general mathematical background to DPPs along with a range of modeling extensions, efficient algorithms, and theoretical results that aim to enable practical modeling and learning.


The dataset comprises 33k QA pairs from 28k images sourced from the COCO2014 dataset #REFR .

For our experiments, we utilized the VQA-X dataset #OTHEREFR , an extension of the VQA-v2 dataset #OTHEREFR , which included explanations for each answer.

For the dataset division, we used all the images in the COCO2014 training set with 29k QA pairs as our training set.

We divided the COCO2014 validation set into our validation set and test set according to the proportion of 3:4, which contains 1.5k QA pairs and 2k QA pairs, respectively.

Compared to traditional vision models with specific tasks such as image classification and image segmentation, for the vision encoder, we only rely on their primary network function to output simple grid features rather than their time-consuming top-down features.

Thus, to better adapt to the Vision & Language task, we used the CLIP based on the structure of the vision transformer as the vision encoder.

The CLIP makes the fusion of vision and language features easier by encoding them in the same hidden space.

Interpretable Visual Question Answering Referring To Outside Knowledge

Abstract: We present a novel multimodal interpretable VQA model that can answer the question more accurately and generate diverse explanations. Although researchers have proposed several methods that can generate human-readable and fine-grained natural language sentences to explain a model's decision, these methods have focused solely on the information in the image. Ideally, the model should refer to various information inside and outside the image to correctly generate explanations, just as we use background knowledge daily. The proposed method incorporates information from outside knowledge and multiple image captions to increase the diversity of information available to the model. The contribution of this paper is to construct an interpretable visual question answering model using multimodal inputs to improve the rationality of generated results. Experimental results show that our model can outperform state-ofthe-art methods regarding answer accuracy and explanation rationality.

Microsoft Coco: Common Objects In Context

Abstract: We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.


Chou, Golovnev, Sudan, And Velusamy [Cgsv21B]

Next, in §5.2, we present results due to Chou, Golovnev, Sudan, Velingker, and Velusamy [CGS + 22] in the linear-space streaming setting, which generalize the result we've seen for Max-Cut (discussed in §3.4.3, due to #REFR ).

Our primary goal is to articulate technical statements of these results, since we'll use them later in Chapters 6 and 7.

We also give some broad-strokes discussions of the techniques involved, building on our work in the previous two chapters.

Specifically, in §5.1, we describe the results of Chou, Golovnev, Sudan, and Velusamy [CGSV21a; CGSV21b] on √ n-space streaming algorithms, which generalize the results we've already seen for Max-Cut (Theorem 3.1, due to #OTHEREFR ) and Max-DiCut (Theorem 4.1, due to #OTHEREFR ).

They include a so-called dichotomy theorem, which completely characterizes CSP approximability for √ n-space sketching algorithms (see §2.2) and builds on our "template distribution" analysis for Max-DiCut in Chapter 4.

This dichotomy will later be the basis for the [BHP + 22] analysis of Max-BCSP(f ) problems for symmetric f : Z k 2 → {0, 1}, described in Chapter 7 below.

We will use these in Chapter 6 to prove linear-space streaming approximation-resistance results for so-called "ordering constraint satisfaction problems" from our joint work #OTHEREFR .

On Streaming Approximation Algorithms For Constraint Satisfaction Problems

Abstract: In this thesis, we explore streaming algorithms for approximating constraint satisfaction problems (CSPs). The setup is roughly the following: A computer has limited memory space, sees a long "stream" of local constraints on a set of variables, and tries to estimate how many of the constraints may be simultaneously satisfied. The past ten years have seen a number of works in this area, and this thesis includes both expository material and novel contributions. Throughout, we emphasize connections to the broader theories of CSPs, approximability, and streaming models, and highlight interesting open problems. The first part of our thesis is expository: We present aspects of previous works that completely characterize the approximability of specific CSPs like Max-Cut and Max-DiCut with √ n-space streaming algorithm (on n-variable instances), while characterizing the approximability of all CSPs in √ n space in the special case of "composable" (i.e., sketching) algorithms, and of a particular subclass of CSPs with linear-space streaming algorithms. In the second part of the thesis, we present two of our own joint works. We begin with a work with Madhu Sudan and Santhoshini Velusamy in which we prove linear-space streaming approximation-resistance for all ordering CSPs (OCSPs), which "CSP-like" problems maximizing over sets of permutations. Previous works considered the maximum acyclic subgraph problem (MAS), the prototypical OCSP; even for MAS, we improve on both the inapproximability factor and the space bound. Next, we present joint work with Joanna Boyland, Michael Hwang, Tarun Prasad, and Santhoshini Velusamy in which we investigate the √ n-space streaming approximability of Boolean CSPs with negations. In particular, we give explicit √ n-space sketching approximability ratios for several families of CSPs, including Max-kAND; develop simpler optimal sketching approximation algorithms for threshold predicates; and show that previous iii Thesis advisor: Prof. Madhu Sudan Noah Singer streaming lower bounds are "incomplete" in that they fail to characterize the √ n-space streaming approximability of Max-3AND.

An Optimal Space Lower Bound For Approximating Max-Cut

Abstract: We consider the problem of estimating the value of MAX-CUT in a graph in the streaming model of computation. At one extreme, there is a trivial 2-approximation for this problem that uses only O(log n) space, namely, count the number of edges and output half of this value as the estimate for the size of the MAX-CUT. On the other extreme, for any fixed є > 0, if one allows O(n) space, a (1+є)-approximate solution to the MAX-CUT value can be obtained by storing an O(n)-size sparsifier that essentially preserves MAX-CUT value. Our main result is that any (randomized) single pass streaming algorithm that breaks the 2-approximation barrier requires Ω(n)-space, thus resolving the space complexity of any non-trivial approximations of the MAX-CUT value to within polylogarithmic factors in the single pass streaming model. We achieve the result by presenting a tight analysis of the Implicit Hidden Partition Problem introduced by Kapralov et al. [SODA’17] for an arbitrarily large number of players. In this problem a number of players receive random matchings of Ω(n) size together with random bits on the edges, and their task is to determine whether the bits correspond to parities of some hidden bipartition, or are just uniformly random. Unlike all previous Fourier analytic communication lower bounds, our analysis does not directly use bounds on the l2 norm of Fourier coefficients of a typical message at any given weight level that follow from hypercontractivity. Instead, we use the fact that graphs received by players are sparse (matchings) to obtain strong upper bounds on the l1 norm of the Fourier coefficients of the messages of individual players using their special structure, and then argue, using the convolution theorem, that similar strong bounds on the l1 norm are essentially preserved (up to an exponential loss in the number of players) once messages of different players are combined. We feel that our main technique is likely of independent interest.


In contrast with Shazeer and Stern #REFR , who only observe instabilities without warmup, we observe instabilities despite a long warmup period.

Various solutions have been proposed, including freezing the embedding layer #OTHEREFR , adding additional layer normalization #OTHEREFR , or reparametrizing the weights #OTHEREFR .

In our work we investigate instabilities which arise during CLIP training.

Unlike the instabilities observed in #OTHEREFR , we find these are not caused by attention entropy collapse.

Instead, our results indicate that spikes arise when the second moment estimator is out of date for the networks early layers.

While our analysis and methods build directly on Shazeer and Stern #OTHEREFR (AdaFactor), there are important differences.

Moreover, in contrast with Shazeer and Stern #OTHEREFR we find that an out-of-date second moment estimator is primarily an issue for the (patch) embedding layer, and measure how well loss spikes are predicted by this event.

Finally, we note that reserachers have moved away from AdaFactor in its original formulation for large-scale training #OTHEREFR , finding AdaFactor to under-perform AdamW #OTHEREFR .

We believe this is due to the factored second moment or absence of first moment.

This is why our focus is AdamW #OTHEREFR which is the de facto standard optimizer for transformers.

Stable And Low-Precision Training For Large-Scale Vision-Language Models

Abstract: We introduce new methods for 1) accelerating and 2) stabilizing training for large language-vision models. 1) Towards accelerating training, we introduce SwitchBack , a linear layer for int8 quantized training which provides a speed-up of 13-25% while matching the performance of bfloat16 training within 0.1 percentage points for the 1B parameter CLIP ViT-Huge-the largest int8 training to date. Our main focus is int8 as GPU support for float8 is rare, though we also analyze float8 training through simulation. While SwitchBack proves effective for float8, we show that standard techniques are also successful if the network is trained and initialized so that large feature magnitudes are discouraged, which we accomplish via layer-scale initialized with zeros. 2) Towards stable training, we analyze loss spikes and find they consistently occur 1-8 iterations after the squared gradients become underestimated by their AdamW second moment estimator. As a result, we recommend an AdamW-Adafactor hybrid, which we refer to as StableAdamW because it avoids loss spikes when training a CLIP ViT-Huge model and outperforms gradient clipping.

Adafactor: Adaptive Learning Rates With Sublinear Memory Cost

Abstract: In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-row and per-column sums of these moving averages, and estimating the per-parameter second moments based on these sums. We demonstrate empirically that this method produces similar results to the baseline. Secondly, we show that adaptive methods can produce larger-than-desired updates when the decay rate of the second moment accumulator is too slow. We propose update clipping and a gradually increasing decay rate scheme as remedies. Combining these methods and dropping momentum, we achieve comparable results to the published Adam regime in training the Transformer model on the WMT 2014 English-German machine translation task, while using very little auxiliary storage in the optimizer. Finally, we propose scaling the parameter updates based on the scale of the parameters themselves.


For example, Poirot #REFR threat detection is modeled as an imprecise graph pattern matching problem.

Provenance graphs have been used extensively in APT detection in recent years due to the superiority of connecting nodes with causal relationships and representing data flow and control flow relationships between system objects.

Both academics and industry are paying increasing attention to this type of learning method #OTHEREFR .

APT detection based on provenance graph is mainly divided into three directions: graph matching-based detection, anomaly score-based detection, and tag propagationbased detection.

The first is graph matching-based detection.

Due to the substructure in the provenance graph can completely describe malicious behavior, it is a very popular method to use graph matching-based detection method to detect APT attacks.

A graph matching method is proposed to identify attacks in provenance graphs.

But it needs to construct the attack graph according to the prior knowledge, and cannot detect the unknown attack method.

UNICORN #OTHEREFR is the first APT intrusion detection system to analyze the local complete system operation.

The provenance graph is converted into a sequence of feature vectors and uses an automaton to model the clustering.

However, the effectiveness of detection is negatively impacted by a large number of automatic opportunities, and the ability to describe the feature sequence is weak.

Tbdetector:Transformer-Based Detector For Advanced Persistent Threats With Provenance Graph

Abstract: APT detection is difficult to detect due to the longterm latency, covert and slow multistage attack patterns of Advanced Persistent Threat (APT). To tackle these issues, we propose TBDetector, a transformer-based advanced persistent threat detection method for APT attack detection. Considering that provenance graphs provide rich historical information and have the powerful attacks historic correlation ability to identify anomalous activities, TBDetector employs provenance analysis for APT detection, which summarizes longrunning system execution with space efficiency and utilizes transformer with self-attention based encoder-decoder to extract long-term contextual features of system states to detect slow-acting attacks. Furthermore, we further introduce anomaly scores to investigate the anomaly of different system states, where each state is calculated with an anomaly score corresponding to its similarity score and isolation score. To evaluate the effectiveness of the proposed method, we have conducted experiments on five public datasets, i.e., streamspot, cadets, shellshock, clearscope, and wget baseline. Experimental results and comparisons with state-of-the-art methods have exhibited better performance of our proposed method.

Poirot: Aligning Attack Behavior With Kernel Audit Records For Cyber Threat Hunting

Abstract: Cyber threat intelligence (CTI) is being used to search for indicators of attacks that might have compromised an enterprise network for a long time without being discovered. To have a more effective analysis, CTI open standards have incorporated descriptive relationships showing how the indicators or observables are related to each other. However, these relationships are either completely overlooked in information gathering or not used for threat hunting. In this paper, we propose a system, called POIROT, which uses these correlations to uncover the steps of a successful attack campaign. We use kernel audits as a reliable source that covers all causal relations and information flows among system entities and model threat hunting as an inexact graph pattern matching problem. Our technical approach is based on a novel similarity metric which assesses an alignment between a query graph constructed out of CTI correlations and a provenance graph constructed out of kernel audit log records. We evaluate POIROT on publicly released real-world incident reports as well as reports of an adversarial engagement designed by DARPA, including ten distinct attack campaigns against different OS platforms such as Linux, FreeBSD, and Windows. Our evaluation results show that POIROT is capable of searching inside graphs containing millions of nodes and pinpoint the attacks in a few minutes, and the results serve to illustrate that CTI correlations could be used as robust and reliable artifacts for threat hunting.


At each x ∈ D n c , there is a tangent space T x D n c , which can be viewed as the first-order approximation of the hyperbolic manifold at x #REFR .

A hyperbolic space is a non-Euclidean space with constant negative curvature.

There are different but equivalent models to describe the same hyperbolic geometry.

In this paper, we work with the Poincaré ball model, in which all points are inside a ball.

The hyperbolic space with constant negative curvature c is denoted by (D n c , g c x ).

It consists of the n-dimensional hyperbolic manifold D n c = {x ∈ R n : c x < 1} with the Riemannian metric g c x = (λ c x ) 2 g E , where λ c x = 2/(1 − c x 2 ) and g E = I n is the Euclidean metric.

The tangent space is then useful to perform Euclidean operations that we are familiar with but are undefined in hyperbolic spaces.

A hyperbolic space and the tangent space at a point are connected through the exponential map exp c

Node-Specific Space Selection Via Localized Geometric Hyperbolicity In Graph Neural Networks

Abstract: Many graph neural networks have been developed to learn graph representations in either Euclidean or hyperbolic space, with all nodes' representations embedded in a single space. However, a graph can have hyperbolic and Euclidean geometries at different regions of the graph. Thus, it is suboptimal to indifferently embed an entire graph into a single space. In this paper, we explore and analyze two notions of local hyperbolicity, describing the underlying local geometry: geometric (Gromov) and model-based, to determine the preferred space of embedding for each node. The two hyperbolicities' distributions are aligned using the Wasserstein metric such that the calculated geometric hyperbolicity guides the choice of the learned model hyperbolicity. As such our model Joint Space Graph Neural Network (JSGNN) can leverage both Euclidean and hyperbolic spaces during learning by allowing node-specific geometry space selection. We evaluate our model on both node classification and link prediction tasks and observe promising performance compared to baseline models.

Constant Curvature Graph Convolutional Networks

Abstract: Interest has been rising lately towards methods representing data in non-Euclidean spaces, e.g. hyperbolic or spherical, that provide specific inductive biases useful for certain real-world data properties, e.g. scale-free, hierarchical or cyclical. However, the popular graph neural networks are currently limited in modeling data only via Euclidean geometry and associated vector space operations. Here, we bridge this gap by proposing mathematically grounded generalizations of graph convolutional networks (GCN) to (products of) constant curvature spaces. We do this by i) introducing a unified formalism that can interpolate smoothly between all geometries of constant curvature, ii) leveraging gyro-barycentric coordinates that generalize the classic Euclidean concept of the center of mass. Our class of models smoothly recover their Euclidean counterparts when the curvature goes to zero from either side. Empirically, we outperform Euclidean GCNs in the tasks of node classification and distortion minimization for symbolic data exhibiting non-Euclidean behavior, according to their discrete curvature.


An understanding of affine, global (Euclidean), or local (Euclidean) rigidity #REFR of a realization Θ(S) has importance in several areas such as molecular dynamics [18; 4; 8] or sensor network localization [34; 32] .

#OTHEREFR Given an alignment S ∈ O(d) m of the local views, a consensus representation Θ(S) of the points can be obtained by averaging the local coordinates of the points due to the (rigidly transformed) views containing them.

In the noiseless setting where the local views are clean measurements of the data (obtained by applying an unknown rigid transformation to a subset of data points), a perfect alignment of views is possible.

Equivalently, when the views are noiseless, a value of zero for F is attainable, and an S that achieves it is called a "perfect alignment".

Clearly, a perfect alignment is an optimal one, while the converse may not hold.

To be consistent with previous works [12; 13] , the consensus representation of the points Θ(S) due to a perfect alignment S of the views is called a realization of the framework.

Under a mild assumption on the structure of the local views, [3; 31; 12] characterized the affine rigidity of a realization by the rank of a certain matrix derived from the framework Θ.

It was shown in #OTHEREFR that deriving a similar characterization of global rigidity is NP-Hard.

Nevertheless a characterization of the local rigidity is useful from an algorithmic standpoint as we show in this work.

Furthermore, necessary and sufficient conditions on the overlapping structure of the local views for affine rigidity were derived in #OTHEREFR .

Similar results in the context of local and global rigidity form our second set of contributions.

Non-Degenerate Rigid Alignment In A Patch Framework

Abstract: Given a set of overlapping local views (patches) of a dataset, we consider the problem of finding a rigid alignment of the views that minimizes a 2-norm based alignment error. In general, the views are noisy and a perfect alignment may not exist. In this work, we characterize the non-degeneracy of an alignment in the noisy setting based on the kernel and positivity of a certain matrix. This leads to a polynomial time algorithm for testing the non-degeneracy of a given alignment. Consequently, we focus on Riemannian gradient descent for minimization of the error and obtain a sufficient condition on an alignment for the algorithm to converge (locally) linearly to it. In the case of noiseless views, a perfect alignment exists, resulting in a realization of the points that respects the geometry of the views. Under a mild condition on the views, we show that the non-degeneracy of a perfect alignment is equivalent to the local rigidity of the resulting realization. By specializing the characterization of a non-degenerate alignment to the noiseless setting, we obtain necessary and sufficient conditions on the overlapping structure of the views for a locally rigid realization. Similar results are also obtained in the context of global rigidity.

On Affine Rigidity

Abstract: We define the notion of affine rigidity of a hypergraph and prove a variety of fundamental results for this notion. First, we show that affine rigidity can be determined by the rank of a specific matrix which implies that affine rigidity is a generic property of the hypergraph.Then we prove that if a graph is is $(d+1)$-vertex-connected, then it must be "generically neighborhood affinely rigid" in $d$-dimensional space. This implies that if a graph is $(d+1)$-vertex-connected then any generic framework of its squared graph must be universally rigid. ::: Our results, and affine rigidity more generally, have natural applications in point registration and localization, as well as connections to manifold learning.


It is found that data sampling may generate local models that over-fit (under-fit) for the minority (majority) classes #REFR .

In general, sampling based techniques are easier to operate compared with other groups of methods (e.g., algorithm-centered techniques) and do not require extensive expertise in ML.

Thus, these methods have become popular when the models are operated by out-of-domain experts.

Another strength of sampling based techniques is that they are agnostic to the core classification models #OTHEREFR .

As such, these techniques can be incorporated into any classification model, and event an ensemble of multiple classification models.

However, straightforward data sampling is not guaranteed to boost the classification accuracy, and sometimes may even worsen the model performance on one client #OTHEREFR .

A possible explanation could be that naive data resampling may prevent clients from learning the "Special Knowledge" from local dataset in FL, which eventually leads to low accuracy #OTHEREFR .

Therefore, sampling strategies need to be carefully designed for FL in order to handle class imbalance effectively.

Therefore, many attempts have been made to propose sampling strategies that address data imbalance systematically in FL.

In general, sampling can be done on either the data instance level (i.e., determining if one data instance should be involved in training) or the client level (i.e., determining if the data instances owned by one client should be involved in training).

In this section, we categorize sampling techniques in FL into three groups and introduce them respectively: (1) data sampling; (2) client sampling; and (3) hybrid data and client sampling.

A Survey On Class Imbalance In Federated Learning

Abstract: Federated learning, which allows multiple client devices in a network to jointly train a machine learning model without direct exposure of clients' data, is an emerging distributed learning technique due to its nature of privacy preservation. However, it has been found that models trained with federated learning usually have worse performance than their counterparts trained in the standard centralized learning mode, especially when the training data is imbalanced. In the context of federated learning, data imbalance may occur either locally one one client device, or globally across many devices. The complexity of different types of data imbalance has posed challenges to the development of federated learning technique, especially considering the need of relieving data imbalance issue and preserving data privacy at the same time. Therefore, in the literature, many attempts have been made to handle class imbalance in federated learning. In this paper, we present a detailed review of recent advancements along this line. We first introduce various types of class imbalance in federated learning, after which we review existing methods for estimating the extent of class imbalance without the need of knowing the actual data to preserve data privacy. After that, we discuss existing methods for handling class imbalance in FL, where the advantages and disadvantages of the these approaches are discussed. We also summarize common evaluation metrics for class imbalanced tasks, and point out potential future directions.

Learning Imbalanced Datasets With Label-Distribution-Aware Margin Loss

Abstract: Deep learning algorithms can fare poorly when the training dataset suffers from heavy class-imbalance but the testing criterion requires good generalization on less frequent classes. We design two novel methods to improve performance in such scenarios. First, we propose a theoretically-principled label-distribution-aware margin (LDAM) loss motivated by minimizing a margin-based generalization bound. This loss replaces the standard cross-entropy objective during training and can be applied with prior strategies for training with class-imbalance such as re-weighting or re-sampling. Second, we propose a simple, yet effective, training schedule that defers re-weighting until after the initial stage, allowing the model to learn an initial representation while avoiding some of the complications associated with re-weighting or re-sampling. We test our methods on several benchmark vision tasks including the real-world imbalanced dataset iNaturalist 2018. Our experiments show that either of these methods alone can already improve over existing techniques and their combination achieves even better performance gains.


In yet some other cases, scientific results about bot activity were later found to largely match with independent platform removals of malicious accounts #REFR Tardelli et al., 2022) , which supports the correctness of the scientific results.

The previous misconceptions might induce readers to think that social bots research has led to flawed-if not outright useless-results, as claimed by some scholars #OTHEREFR , even in sensationalist terms #OTHEREFR .

However, we see at least two strong arguments against this thesis.

Firstly, in spite of the limitations of bot detectors, there have been multiple glaring examples of bot studies that were able to bring to light demonstrably harmful campaigns.

For example, the detectors developed as part of some scientific endeavors were later deployed on online platforms and used to remove large numbers of malicious accounts #OTHEREFR .

Similarly, results of some studies on bot activity led platforms to remove the accounts identified as malicious bots (Ferrara, 2022) .

These cases represent but some of the success stories of social bots research.

If researchers had not developed bot detection techniques, we would not have been able to identify pockets of anomalous accounts engaged in well-engineered malicious activities.

Therefore, even if no universal bot detector still exists and in spite of the many caveats to consider in bot detection tasks, being able to detect some malicious bots puts us in an advantageous position than being able to detect none.

Secondly, the benefits of social bots research extend beyond the detec-tion of malicious bots.

For example, research and experimentation on social bots led to the development of neutral bots used for assessing the degree of political polarization on a platform #OTHEREFR , "news bots" used for journalistic purposes to curate, aggregate, and disseminate content collected from multiple sources #OTHEREFR , or even bots used for content moderation #OTHEREFR .

Demystifying Misconceptions In Social Bots Research

Abstract: The science of social bots seeks knowledge and solutions to one of the most debated forms of online misinformation. Yet, social bots research is plagued by widespread biases, hyped results, and misconceptions that set the stage for ambiguities, unrealistic expectations, and seemingly irreconcilable findings. Overcoming such issues is instrumental towards ensuring reliable solutions and reaffirming the validity of the scientific method. In this contribution we revise some recent results in social bots research, highlighting and correcting factual errors as well as methodological and conceptual issues. More importantly, we demystify common misconceptions, addressing fundamental points on how social bots research is discussed. Our analysis surfaces the need to discuss misinformation research in a rigorous, unbiased, and responsible way. This article bolsters such effort by identifying and refuting common fallacious arguments used by both proponents and opponents of social bots research as well as providing indications on the correct methodologies and sound directions for future research in the field.

Charting The Landscape Of Online Cryptocurrency Manipulation

Abstract: Cryptocurrencies represent one of the most attractive markets for financial speculation. As a consequence, they have attracted unprecedented attention on social media. Besides genuine discussions and legitimate investment initiatives, several deceptive activities have flourished. In this work, we chart the online cryptocurrency landscape across multiple platforms. To reach our goal, we collected a large dataset, composed of more than 50M messages published by almost 7M users on Twitter, Telegram and Discord, over three months. We performed bot detection on Twitter accounts sharing invite links to Telegram and Discord channels, and we discovered that more than 56% of them were bots or suspended accounts. Then, we applied topic modeling techniques to Telegram and Discord messages, unveiling two different deception schemes - "pump-and-dump" and "Ponzi" - and identifying the channels involved in these frauds. Whereas on Discord we found a negligible level of deception, on Telegram we retrieved 296 channels involved in pump-and-dump and 432 involved in Ponzi schemes, accounting for a striking 20% of the total. Moreover, we observed that 93% of the invite links shared by Twitter bots point to Telegram pump-and-dump channels, shedding light on a little-known social bot activity. Charting the landscape of online cryptocurrency manipulation can inform actionable policies to fight such abuse.


C. Power Control For Secure Transmission 1) Background:

Considering a two-link interference channel with one eavesdropper, the authors of #REFR devise a power control algorithm based on an altruistic and egoistic setting.

The aim of secure transmission is to send messages to legitimate receivers without any information leakage to eavesdroppers.

Aside from the information-theoretical study #OTHEREFR - #OTHEREFR , the optimization aspect has attracted extensive research interest as well.

For a wiretap channel with one legitimate receiver, one eavesdropper, and multiple jammers, #OTHEREFR characterizes the power control as a Srtacjelberg game and thereby proposes an iterative optimization that achieves a Srtacjelberg equilibrium.

Moreover, #OTHEREFR suggest a second-order-cone approach to power control for secure transmission in a cell-free MIMO network.

For the UAV network case, #OTHEREFR proposes optimizing powers via block coordinate descent and successive convex approximation.

The use of machine learning has become popular in this area, e.g., the deep reinforcement learning approach in #OTHEREFR and the Q-learning approach in #OTHEREFR .

2) Problem Formulation: Consider L base-stations (BSs) each serving a legitimate downlink user terminal.

Mixed Max-And-Min Fractional Programming For Wireless Networks

Abstract: Fractional programming (FP) plays a crucial role in wireless network design because many relevant problems involve maximizing or minimizing ratio terms. Notice that the maximization case and the minimization case of FP cannot be converted to each other in general, so they have to be dealt with separately in most of the previous studies. Thus, an existing FP method for maximizing ratios typically does not work for the minimization case, and vice versa. However, the FP objective can be mixed max-and-min, e.g., one may wish to maximize the signal-tointerference-plus-noise ratio (SINR) of the legitimate receiver while minimizing that of the eavesdropper. We aim to fill the gap between max-FP and min-FP by devising a unified optimization framework. The main results are threefold. First, we extend the existing max-FP technique called quadratic transform to the min-FP, and further develop a full generalization for the mixed case. Second. we provide a minorization-maximization (MM) interpretation of the proposed unified approach, thereby establishing its convergence and also obtaining a matrix extension; another result we obtain is a generalized Lagrangian dual transform which facilitates the solving of the logarithmic FP. Finally, we present three typical applications: the age-of-information (AoI) minimization, the Cramér-Rao bound minimization for sensing, and the secure data rate maximization, none of which can be efficiently addressed by the previous FP methods. Index Terms-Multi-ratio mixed max-and-min fractional programming (FP); matrix FP; unified approach; age-of-information (AoI); Cramér-Rao bound; sensing; secure transmission.

Joint Power Control In Wiretap Interference Channels

Abstract: Interference in wireless networks degrades the signal quality at the terminals. However, it can potentially enhance the secrecy rate. This paper investigates the secrecy rate in a two-user interference network where one of the users, namely user 1, requires to establish a confidential connection. User 1 wants to prevent an unintended user of the network to decode its transmission. User 1 has to transmit such that its secrecy rate is maximized while the quality of service at the destination of the other user, user 2, is satisfied, and both user's power limits are taken into account. We consider two scenarios: 1) user 2 changes its power in favor of user 1, an altruistic scenario, 2) user 2 is selfish and only aims to maintain the minimum quality of service at its destination, an egoistic scenario. It is shown that there is a threshold for user 2's transmission power that only below or above which, depending on the channel qualities, user 1 can achieve a positive secrecy rate. Closed-form solutions are obtained in order to perform joint optimal power control. Further, a new metric called secrecy energy efficiency is introduced. We show that in general, the secrecy energy efficiency of user 1 in an interference channel scenario is higher than that of an interference-free channel.


As shown in Table 1 , our self-supervised model achieves comparable performance with supervised HPLFlowNet #REFR on FT3D s dataset.

From the results, it can be found that our model can outperform all compared self-supervised methods in terms of the four metrics on the FT3D s test data. Especially, our model brings 8.72% gains for metric AS.

For the KITTI s dataset, our model brings substantial improvements on all metrics.

To be specific, our model outperforms the second best method RCP #OTHEREFR by 8.68% and 6.58% on metrics AS and AR, respectively.

Besides, it is worth noting that our model can even achieve an EPE metric of 3.62cm, which is much lower than the EPE (6.19cm ) of recent RigidFlow.

We also compare our model with some supervised methods, such as FlowNet3D #OTHEREFR and FLOT #OTHEREFR , etc.

Without any fine-tuning on KITTI s dataset, our model can even outperform the supervised methods listed in Table 1 , which proves that our model has better generalization ability.

For real scenes, most local regions are with similar flow patterns.

Thanks to dynamically clustering mechanism, our model clusters points with similar flow pattern into the same clusters and encodes the superpointlevel flow into the GRU for flow refinement, thereby leading to satisfactory performance on real scenes.

Performance on point clouds with occlusions.

Following the experimental settings used in Self-Point-Flow #OTHEREFR and RigidFlow #OTHEREFR , we train our model on KITTI r dataset and evaluate our model on both KITTI o and KITTI t datasets.

Self-Supervised 3D Scene Flow Estimation Guided By Superpoints

Abstract: 3D scene flow estimation aims to estimate point-wise motions between two consecutive frames of point clouds. Superpoints, i.e., points with similar geometric features, are usually employed to capture similar motions of local regions in 3D scenes for scene flow estimation. However, in existing methods, superpoints are generated with the offline clustering methods, which cannot characterize local regions with similar motions for complex 3D scenes well, leading to inaccurate scene flow estimation. To this end, we propose an iterative end-to-end superpoint based scene flow estimation framework, where the superpoints can be dynamically updated to guide the point-level flow prediction. Specifically, our framework consists of a flow guided superpoint generation module and a superpoint guided flow refinement module. In our superpoint generation module, we utilize the bidirectional flow information at the previous iteration to obtain the matching points of points and superpoint centers for soft point-to-superpoint association construction, in which the superpoints are generated for pairwise point clouds. With the generated superpoints, we first reconstruct the flow for each point by adaptively aggregating the superpoint-level flow, and then encode the consistency between the reconstructed flow of pairwise point clouds. Finally, we feed the consistency encoding along with the reconstructed flow into GRU to refine point-level flow. Extensive experiments on several different datasets show that our method can achieve promising performance. Code is available at https://github. com/supersyq/SPFlowNet.

Hplflownet: Hierarchical Permutohedral Lattice Flownet For Scene Flow Estimation On Large-Scale Point Clouds

Abstract: We present a novel deep neural network architecture for end-to-end scene flow estimation that directly operates on large-scale 3D point clouds. Inspired by Bilateral Convolutional Layers (BCL), we propose novel DownBCL, UpBCL, and CorrBCL operations that restore structural information from unstructured point clouds, and fuse information from two consecutive point clouds. Operating on discrete and sparse permutohedral lattice points, our architectural design is parsimonious in computational cost. Our model can efficiently process a pair of point cloud frames at once with a maximum of 86K points per frame. Our approach achieves state-of-the-art performance on the FlyingThings3D and KITTI Scene Flow 2015 datasets. Moreover, trained on synthetic data, our approach shows great generalization ability on real-world data and on different point densities without fine-tuning.

self-supervised model
supervised HPLFlowNet


The graph compressor is inspired by #REFR , and it works by computing the attention scores for all sentence nodes, filtering out nodes with the lowest scores, and then masking the rest using their attention scores.

Given G D and Q D from the graph encoder, the graph compressor aims to "compress" the graph by selecting a subset of salient nodes and edges.

Here we focus on filtering the sentence nodes, because we want to identify key sentences that help generate the summary.

After the compression, all selected sentence nodes and their linked word and document nodes represent the compressed graph and their embeddings will be used by the text decoder for summary generation.

Firstly, attention scores of the sentence nodes are calculated based on the updated node embeddings from our proposed graph encoder MGAT(Q D , G D ):

where r is the only trainable parameter of the graph compressor which transforms the updated node embedding into a scalar.

Then, based on these scores, we select sentence nodes with the highest scores:

Compressed Heterogeneous Graph For Abstractive Multi-Document Summarization

Abstract: Multi-document summarization (MDS) aims to generate a summary for a number of related documents. We propose HGSUM-an MDS model that extends an encoder-decoder architecture to incorporate a heterogeneous graph to represent different semantic units (e.g., words and sentences) of the documents. This contrasts with existing MDS models which do not consider different edge types of graphs and as such do not capture the diversity of relationships in the documents. To preserve only key information and relationships of the documents in the heterogeneous graph, HGSUM uses graph pooling to compress the input graph. And to guide HGSUM to learn the compression, we introduce an additional objective that maximizes the similarity between the compressed graph and the graph constructed from the ground-truth summary during training. HGSUM is trained end-to-end with the graph similarity and standard cross-entropy objectives. Experimental results over MULTI-NEWS, WCEP-100, and ARXIV show that HGSUM outperforms state-of-the-art MDS models. The code for our model and experiments is available at: https://github.com/oaimli/HGSum.

Self-Attention Graph Pooling

Abstract: Advanced methods of applying deep learning to structured data such as graphs have been proposed in recent years. In particular, studies have focused on generalizing convolutional neural networks to graph data, which includes redefining the convolution and the downsampling (pooling) operations for graphs. The method of generalizing the convolution operation to graphs has been proven to improve performance and is widely used. However, the method of applying downsampling to graphs is still difficult to perform and has room for improvement. In this paper, we propose a graph pooling method based on self-attention. Self-attention using graph convolution allows our pooling method to consider both node features and graph topology. To ensure a fair comparison, the same training procedures and model architectures were used for the existing pooling methods and our method. The experimental results demonstrate that our method achieves superior graph classification performance on the benchmark datasets using a reasonable number of parameters.


While text-to-table, for example, also uses pre-trained language models as a basis, it is directly initialized with the pre-trained weights of a language model (i.e., BART #REFR ), which has been pre-trained on plain text only.

While there exist already approaches that transform textual data to tables like text-to-table #OTHEREFR or STable #OTHEREFR , we think that these approaches are not suitable for implementing multi-modal database operations in an MMDB.

One key difference of our approach compared to text-to-table and STable is that the underlying models have to be trained from scratch for every new text collection.

Different from that, we carefully design a new pre-training procedure that allows our MMDB-Model to provide more meaningful representations for table extraction and thus realize MMOps on unseen texts with just a few examples.

Moreover, another key difference is that text-to-table or STable produce the data for an output table using a transformer-based generative decoder #OTHEREFR .

The first disadvantage here is that the model can "make up" values, that are not actually present in the input text, but that the model picked up during (pre-)training. This phenomenon is called hallucination #OTHEREFR .

On top of that, transformer-based decoders output tables token-by-token in an autoregressive manner, which requires a pass through the decoder for every token in the output table.

This results in a computationally expensive decoding process as we show in our evaluation.

Towards Multi-Modal Dbmss For Seamless Querying Of Texts And Tables

Abstract: In this paper, we propose Multi-Modal Databases (MMDBs), which is a new class of database systems that can seamlessly query text and tables using SQL. To enable seamless querying of textual data using SQL in an MMDB, we propose to extend relational databases with so-called multi-modal operators (MMOps) which are based on the advances of recent large language models such as GPT-3. The main idea of MMOps is that they allow text collections to be treated as tables without the need to manually transform the data. As we show in our evaluation, our MMDB prototype can not only outperform state-of-the-art approaches such as text-totable in terms of accuracy and performance but it also requires significantly less training data to fine-tune the model for an unseen text collection.

Bart: Denoising Sequence-To-Sequence Pre-Training For Natural Language Generation, Translation, And Comprehension

Abstract: We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and many other more recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of the original sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa with comparable training resources on GLUE and SQuAD, achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 6 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also report ablation experiments that replicate other pretraining schemes within the BART framework, to better measure which factors most influence end-task performance.

language model
pre-trained language models


Existing FL implementations generally consider static strategies to allocate spectrum (i.e., PRBs) to FLUs for an entire LTR #REFR , leading to underutilization of spectrum, called spectrum fragmentation.

Due to temporal variation of channels, static resource allocation (e.g., licensed/unlicensed spectrum and transmit power) of current FL implementations #OTHEREFR can lead to two obscure problems.

(i) Wireless resource over-provisioning (WROP), referring to surpassing FLUs' QoS requirements upon over-provisioning of wireless resources, causing an increase in expenditure, energy consumption, and interference.

(ii) Wireless resources under provisioning (WRUP), referring to the wireless resource deficiency of FLUs at some time instants, which can cause SLA violations for FLSs.

WRUP can reduce FLSOs satisfaction, preventing them from running their FLSs in the future.

To harness WROP/WRUP, we propose two mechanisms in EV-FL to dynamically match resources to FLSOs' de- 3) Spectrum fragmentation: In FL, a local training round (LTR) of FLUs consists of (i) GM downloading, (ii) LM training, and (iii) LM uploading.

To our knowledge, we are among the first to identify spectrum fragmentation in FL, meaning that FLUs only utilize wireless resources for GM downloading and LM uploading, while these resources are idle during LM training.

Spectrum fragmentation under DCC. Spectrum fragmentation becomes more severe upon considering DCC.

In DCC, we have two types of LTRs (see Fig.

2 ): (i) DPU LTR, consisting of three steps: (i-a) GM downloading, (i-b) LM training, and (i-c) dispersing the trained LM to neighboring CHUs.

(ii) CHU LTR, consisting of five steps: (ii-a) GM downloading, (ii-b) LM training, (ii-c) waiting for DPUs connected to the CHU to perform DPU LTR, (ii-d) performing local aggregation, and (ii-e) aggregated LM uploading to the BS.

Synergies Between Federated Learning And O-Ran: Towards An Elastic Virtualized Architecture For Multiple Distributed Machine Learning Services

Abstract: Federated learning (FL) is the most popular distributed machine learning technique. However, implementation of FL over modern wireless networks faces key challenges caused by (i) dynamics of the network conditions, (ii) coexistence of multiple FL services/tasks in the system, and (iii) concurrent execution of FL services with other network services, which are not jointly considered in prior works. Motivated by these challenges, we introduce a generic FL paradigm over next-generation (NextG) networks, called dynamic multi-service FL (DMS-FL). We identify three unexplored design considerations in DMS-FL: (i) FL service operator accumulation, (ii) wireless resource fragmentation, and (iii) signal strength fluctuations. We take the first steps towards addressing these design considerations through proposing a novel distributed ML architecture called elastic virtualized FL (EV-FL). EV-FL unleashes the full potential of Open RAN (O-RAN) systems and introduces an elastic resource provisioning methodology to execute FL services. It further constitutes a multi-timescale FL management system that introduces three dimensions into existing FL architectures: (i) virtualization, (ii) scalability, and (iii) elasticity. Through investigating EV-FL, we reveal a series of open research directions for future work. We finally simulate EV-FL to demonstrate its potential to save wireless resources and increase fairness among FL services.

Federated Learning In Mobile Edge Networks: A Comprehensive Survey

Abstract: In recent years, mobile devices are equipped with increasingly advanced sensing and computing capabilities. Coupled with advancements in Deep Learning (DL), this opens up countless possibilities for meaningful applications. Traditional Machine Learning (ML) approaches require the data to be centralized in a cloud server. However, this results in critical issues related to unacceptable latency and communication inefficiency. To this end, Mobile Edge Computing (MEC) has been proposed to bring intelligence closer to the edge, where data is produced. However, conventional enabling technologies for ML at mobile edge networks still require personal data to be shared with external parties, e.g., edge servers. Recently, in light of increasingly stringent data privacy legislation and growing privacy concerns, the concept of Federated Learning (FL) has been introduced. In FL, end devices use their local data to train an ML model required by the server. The end devices then send the model updates rather than raw data to the server for aggregation. FL can serve as an enabling technology in mobile edge networks since it enables the collaborative training of an ML model and also enables DL for mobile edge network optimization. However, in a large-scale and complex mobile edge network, heterogeneous devices with varying constraints are involved. This raises challenges of communication costs, resource allocation, and privacy and security in the implementation of FL at scale. In this survey, we begin with an introduction to the background and fundamentals of FL. Then, we highlight the aforementioned challenges of FL implementation and review existing solutions. Furthermore, we present the applications of FL for mobile edge network optimization. Finally, we discuss the important challenges, open issues and future research directions in FL.

Existing FL implementations
underutilization


Minimum Spanning Tree Games And Approximation

From the more theoretical perspective, one can easily see that for MST games the cost function c( • ) is subadditive, yet it is not submodular in general; see #REFR for a characterization when it is.

In this section we address a well known special class of games known as minimum cost spanning tree (MST) games #OTHEREFR , where the cost of the outside option for a set of agents is determined by the cost of a minimum cost spanning tree in the subgraph induced by these agents. MST games are known to have a non-empty core.

Moreover, it is known that finding some element in the core is computationally easy and can be done by computing a minimum spanning tree #OTHEREFR .

The optimization problem that we address asks for the maximal amount that can be charged to the agents while no proper subset of agents would prefer the outside option.

Recalling that the computation of maximum shareable costs can be done in polynomial time when c( • ) is submodular, it is a natural question to ask if this still holds for subadditive cost functions.

In that respect, note that the weighted graph games as studied in #OTHEREFR have polynomial time algorithms to decide non-emptiness of the core whenever c( • ) is subadditive, hence Theorem 4 applied to weighted graphs games does not give an answer to this question.

We will see that for MST games, maximizing shareable costs cannot be done efficiently unless P=NP.

In fact, one could define an even more general class of problems in the spirit of cooperative games with restricted coalition formation, by defining a (downward-closed) set system that describes all those subsets of agents that are able to cooperate and hence have access to an outside option, while all other subsets do not have that option.

The almost core as studied in this paper arises as the special case where the set system is the (n − 1)-uniform matroid.

Algorithmic Solutions For Maximizing Shareable Costs

Abstract: This paper addresses the optimization problem to maximize the total costs that can be shared among a group of agents, while maintaining stability in the sense of the core constraints of a cooperative transferable utility game, or TU game. This means that all subsets of agents have an outside option at a certain cost, and stability requires that the cost shares are defined so that none of the outside options is preferable. When maximizing total shareable costs, the cost shares must satisfy all constraints that define the core of a TU game, except for being budget balanced. The paper gives a fairly complete picture of the computational complexity of this optimization problem, in relation to classical computational problems on the core. We also show that, for games with an empty core, the problem is equivalent to computing minimal core relaxations for several relaxations that have been proposed earlier. As an example for a class of cost sharing games with non-empty core, we address minimum cost spanning tree games. While it is known that cost shares in the core can be found efficiently, we show that the computation of maximal cost shares is NP-hard for minimum cost spanning tree games. We also derive a 2-approximation algorithm. Our work opens several directions for future work.

An Efficient Characterization Of Submodular Spanning Tree Games

Abstract: Cooperative games are an important class of problems in game theory, where the goal is to distribute a value among a set of players who are allowed to cooperate by forming coalitions. An outcome of the game is given by an allocation vector that assigns a value share to each player. A crucial aspect of such games is submodularity (or convexity). Indeed, convex instances of cooperative games exhibit several nice properties, e.g. regarding the existence and computation of allocations realizing some of the most important solution concepts proposed in the literature. For this reason, a relevant question is whether one can give a polynomial time characterization of submodular instances, for prominent cooperative games that are in general non-convex.


The aforementioned audio-visual distillation methods #REFR highly relate the cross-modal synchronization to semantic shows that we first erase the irrelevant modality noise.

Knowledge distillation has become an effective approach to ensemble different data sources to enrich the representation ability of target modality #OTHEREFR . Compared with visual-related modalities (e.g.

RGB and optical flow), sound naturally contains vivid supervisory information from an independent auditory source, which deserves thorough exploration due to its synchronized nature with vision.

Inspired by this, crossmodal knowledge distillation has been extensively investigated in audio-visual learning #OTHEREFR , leveraging the synchronization between auditory and visual modalities.

(c) illustrates that we further capture the audio-visual correspondence and rectify the semantic correlation. consistency, and directly transfer knowledge across the audiovisual modalities.

However, such semantic consistency is hard to guarantee in unconstrained videos, thus, directly transferring knowledge across audio-visual modalities assuming highly-related semantic consistency could harm the distillation performance.

The similar phenomenon of semantic misalignment is also found in self-supervised multi-modal learning #OTHEREFR but got little attention in cross-modal distillation.

In this work, we reconsider the conventional semantic consistency assumption and point out that the irrelevant modality noise and differentiated semantic correlation are blamed for the failure of such assumption and harm the crossmodal distillation performance, especially in unconstrained audio-visual scenarios.

Concretely, in cross-modal distillation, irrelevant modality noise is considered as the signals in one modality that are unrelated to its accompanying target modality.

Robust Cross-Modal Knowledge Distillation For Unconstrained Videos

Abstract: Cross-modal distillation has been widely used to transfer knowledge across different modalities, enriching the representation of the target unimodal one. Recent studies highly relate the temporal synchronization between vision and sound to the semantic consistency for cross-modal distillation. However, such semantic consistency from the synchronization is hard to guarantee in unconstrained videos, due to the irrelevant modality noise and differentiated semantic correlation. To this end, we first propose a Modality Noise Filter (MNF) module to erase the irrelevant noise in teacher modality with cross-modal context. After this purification, we then design a Contrastive Semantic Calibration (CSC) module to adaptively distill useful knowledge for target modality, by referring to the differentiated sample-wise semantic correlation in a contrastive fashion. Extensive experiments show that our method could bring a performance boost compared with other distillation methods in both visual action recognition and video retrieval task. We also extend to the audio tagging task to prove the generalization of our method.

Soundnet: Learning Sound Representations From Unlabeled Video

Abstract: We learn rich natural sound representations by capitalizing on large amounts of unlabeled sound data collected in the wild. We leverage the natural synchronization between vision and sound to learn an acoustic representation using two-million unlabeled videos. Unlabeled video has the advantage that it can be economically acquired at massive scales, yet contains useful signals about natural sound. We propose a student-teacher training procedure which transfers discriminative visual knowledge from well established visual recognition models into the sound modality using unlabeled video as a bridge. Our sound representation yields significant performance improvements over the state-of-the-art results on standard benchmarks for acoustic scene/object classification. Visualizations suggest some high-level semantics automatically emerge in the sound network, even though it is trained without ground truth labels.


BERTScore #REFR ) is a neural metric that relies on pre-trained models to compute the semantic similarity with the reference.

We utilize the accuracy of pairwise system-ranking (Kocmi et al., 2021) of three types of Kendall correlation.

Specifically, these values are computed by flattening the scores into a single vector and calculating the average correlations over systems, or over segments.

Baseline We compare LLMs with several commonly used baseline metrics for MT evaluation.

BLEU #OTHEREFR is the most popular metric that compares the n-gram overlap of the translation with human reference, but it has been criticized for not capturing the full semantic meaning of the translation #OTHEREFR .

BLEURT #OTHEREFR and COMET #OTHEREFR are supervised neural metrics that leverage human judgments to train. They have shown a high correlation with human judgments.

Error Analysis Prompting Enables Human-Like Translation Evaluation In Large Language Models: A Case Study On Chatgpt

Abstract: Generative large language models (LLMs), e.g., ChatGPT, have demonstrated remarkable proficiency across several NLP tasks such as machine translation, question answering, text summarization, and natural language understanding. Recent research (Kocmi and Federmann, 2023) has shown that utilizing Chat-GPT for assessing the quality of machine translation (MT) achieves state-of-the-art performance at the system level but performs poorly at the segment level. To further improve the performance of LLMs on MT quality assessment, we conducted an investigation into several prompting methods. Our results indicate that by combining Chain-of-Thoughts (Wei et al., 2022) and Error Analysis (Lu et al., 2022), a new prompting method called Error Analysis Prompting, LLMs like ChatGPT can generate human-like MT evaluations at both the system and segment level. Additionally, we discovered some limitations of Chat-GPT as an MT evaluator, such as unstable scoring and biases when provided with multiple translations in a single query. Our findings aim to provide a preliminary experience for appropriately evaluating translation quality on Chat-GPT while offering a variety of tricks in designing prompts for in-context learning. We anticipate that this report will shed new light on advancing the field of translation evaluation with LLMs by enhancing both the accuracy and reliability of metrics.

Bertscore: Evaluating Text Generation With Bert

Abstract: We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task and show that BERTScore is more robust to challenging examples compared to existing metrics.


B. Multi-Modal Fusion For Object Detection 1) Multi-Modal Object Detection:

PointPainting #REFR projects LiDAR points into the output of an image-only semantic segmentation network, and appends the class scores to each point.

To date, several studies have investigated multi-modal fusion for 2D and 3D object detection.

Frustum PointNets #OTHEREFR extract the 3D bounding frustum of an object by extruding 2D bounding boxes from image detectors.

PointFusion #OTHEREFR combines a CNN and a PointNet #OTHEREFR architecture respectively to process images and raw point clouds then predict 3D boxes.

All these fusion methods of RGB and LiDAR achieve high average precision on the benchmarks, however, the coupling or interrelation of two modalities will cause the whole system to fail easily once part of the sensors break down.

Besides, the methods above only provide a deterministic predict result, making it risky to carry out in the real application.

2) Adaptive fusion: Several new studies have proposed self-adaptive techniques in computer vision.

Therefore, the robustness of those tasks can be improved to some extent.

Adaptnet #OTHEREFR uses a convoluted mixture of deep experts(CMoDE) fusion techniques to learn features from complementary modalities and spectra.

Informative Data Selection With Uncertainty For Multi-Modal Object Detection

Abstract: Noise has always been nonnegligible trouble in object detection by creating confusion in model reasoning, thereby reducing the informativeness of the data. It can lead to inaccurate recognition due to the shift in the observed pattern, that requires a robust generalization of the models. To implement a general vision model, we need to develop deep learning models that can adaptively select valid information from multi-modal data. This is mainly based on two reasons. Multi-modal learning can break through the inherent defects of single-modal data, and adaptive information selection can reduce chaos in multi-modal data. To tackle this problem, we propose a universal uncertainty-aware multi-modal fusion model. It adopts a multi-pipeline loosely coupled architecture to combine the features and results from point clouds and images. To quantify the correlation in multimodal information, we model the uncertainty, as the inverse of data information, in different modalities and embed it in the bounding box generation. In this way, our model reduces the randomness in fusion and generates reliable output. Moreover, we conducted a completed investigation on the KITTI 2D object detection dataset and its derived dirty data. Our fusion model is proven to resist severe noise interference like Gaussian, motion blur, and frost, with only slight degradation. The experiment results demonstrate the benefits of our adaptive fusion. Our analysis on the robustness of multi-modal fusion will provide further insights for future research.

Pointpainting: Sequential Fusion For 3D Object Detection

Abstract: Camera and lidar are important sensor modalities for robotics in general and self-driving cars in particular. The sensors provide complementary information offering an opportunity for tight sensor-fusion. Surprisingly, lidar-only methods outperform fusion methods on the main benchmark datasets, suggesting a gap in the literature. In this work, we propose PointPainting: a sequential fusion method to fill this gap. PointPainting works by projecting lidar points into the output of an image-only semantic segmentation network and appending the class scores to each point. The appended (painted) point cloud can then be fed to any lidar-only method. Experiments show large improvements on three different state-of-the art methods, Point-RCNN, VoxelNet and PointPillars on the KITTI and nuScenes datasets. The painted version of PointRCNN represents a new state of the art on the KITTI leaderboard for the bird's-eye view detection task. In ablation, we study how the effects of Painting depends on the quality and format of the semantic segmentation output, and demonstrate how latency can be minimized through pipelining.

image-only semantic segmentation
LiDAR points


For each training subject, we sample (a) 5,000 surface points as G s , which are evenly distributed on the mesh surface; (b) 5,000 occupancy-target points as G o in the 3D space, for which we follow the sampling strategy in #REFR .

In particular, 15/16 points in G o are evenly sampled on the mesh surface.

Gaussian perturbation is then applied to them along the surface normal direction.

The rest 1/16 points are randomly sampled within the predefined 3D space where the mesh lies in.

We employ Rembg [1] to segment out the background for real-world images and use Kaolin [2] to calculate the SDF from the SMPL-X model.

We apply PIXIE #OTHEREFR to estimate the SMPL-X model for the segmented human subject.

Sesdf: Self-Evolved Signed Distance Field For Implicit 3D Clothed Human Reconstruction

Abstract: Figure 1. Self-evolved Signed Distance Field (SeSDF): We propose Self-evolved Signed Distance Field (SeSDF), which can flexibly reconstruct 3D clothed human models from a single-view image (top) or uncalibrated multi-view images (bottom). SeSDF can robustly recover fine geometry details from any type of poses, which allows us to generate clothed human avatars.

Pifu: Pixel-Aligned Implicit Function For High-Resolution Clothed Human Digitization

Abstract: We introduce Pixel-aligned Implicit Function (PIFu), a highly effective implicit representation that locally aligns pixels of 2D images with the global context of their corresponding 3D object. Using PIFu, we propose an end-to-end deep learning method for digitizing highly detailed clothed humans that can infer both 3D surface and texture from a single image, and optionally, multiple input images. Highly intricate shapes, such as hairstyles, clothing, as well as their variations and deformations can be digitized in a unified way. Compared to existing representations used for 3D deep learning, PIFu can produce high-resolution surfaces including largely unseen regions such as the back of a person. In particular, it is memory efficient unlike the voxel representation, can handle arbitrary topology, and the resulting surface is spatially aligned with the input image. Furthermore, while previous techniques are designed to process either a single image or multiple views, PIFu extends naturally to arbitrary number of views. We demonstrate high-resolution and robust reconstructions on real world images from the DeepFashion dataset, which contains a variety of challenging clothing types. Our method achieves state-of-the-art performance on a public benchmark and outperforms the prior work for clothed human digitization from a single image.