Visualization Tools For Supporting Error Discovery.

For example, FairVis #REFR uses visualizations to help ML developers discover model bias by investigating known subgroups and exploring similar groups in the tabular data.

Interactive visualization is a powerful method for helping ML developers explore and interpret their models #OTHEREFR .

While many visualizations have been built to help practitioners evaluate models over time, one area of recent work has focused on designing and developing analytic tools for ML error discovery [e.g. #OTHEREFR .

Similarly, Visual Auditor #OTHEREFR automatically surfaces underperforming subgroups and leverages graph visualizations to help practitioners interpret the relationships between subgroups and discover new errors.

For image data, explAIner #OTHEREFR combines interactive visualization and post-hoc ML explanation techniques [e.g., #OTHEREFR to help practitioners diagnose problems with image classifiers.

For text data, Seq2seq-Vis #OTHEREFR helps practitioners debug sequence-to-sequence models by visualizing the model's internal mechanisms throughout each of its inference stages.

The success of these recent visual ML diagnosis systems highlights the outstanding potential of applying visualization techniques to help ML developers detect errors.

Instead of visualizing a model's internals [e.g., #OTHEREFR , we treat ML models as black-box systems and focus on probing their behaviors on different data subsets.

Angler: Helping Machine Translation Practitioners Prioritize Model Improvements

Abstract: Angler enables ML developers to easily explore and curate challenge sets for machine translation. (A) The Table View lists all challenge sets, allowing users to compare them by metrics such as sample count, model performance, and familiarity score. After selecting a set, (B) the Detail View allows users to further explore samples in this set across various dimensions. (B1) The Timeline enables users to query data samples by time. (B2) The Spotlight presents visualizations with linking and brushing to help users characterize the set from different angles. (B3) The Sentence List shows all selected data samples and allows users to further fine-tune before exporting this challenge set for downstream tasks.

Fairvis: Visual Analytics For Discovering Intersectional Bias In Machine Learning

Abstract: The growing capability and accessibility of machine learning has led to its application to many real-world domains and data about people. Despite the benefits algorithmic systems may bring, models can reflect, inject, or exacerbate implicit and explicit societal biases into their outputs, disadvantaging certain demographic subgroups. Discovering which biases a machine learning model has introduced is a great challenge, due to the numerous definitions of fairness and the large number of potentially impacted subgroups. We present FAIRVIS, a mixed-initiative visual analytics system that integrates a novel subgroup discovery technique for users to audit the fairness of machine learning models. Through FAIRVIS, users can apply domain knowledge to generate and investigate known subgroups, and explore suggested and similar subgroups. FAIRVIS's coordinated views enable users to explore a high-level overview of subgroup performance and subsequently drill down into detailed investigation of specific subgroups. We show how FAIRVIS helps to discover biases in two real datasets used in predicting income and recidivism. As a visual analytics system devoted to discovering bias in machine learning, FAIRVIS demonstrates how interactive visualization may help data scientists and the general public understand and create more equitable algorithmic systems.


In this work, we focus on language models with a Transformer encoder architecture such as BERT #REFR , that extract relevant outputs from a domain-specific evidence-based text corpus.

Large Language Models (LLMs), such as Generative Pre-trained Transformer 3 (GPT-3) #OTHEREFR , generate text that is grammatically correct, fluent, and informative.

However, there is little to no control over the data that were used for model training.

Consequently, LLMs are prone to hallucinating and providing untruthful outputs .

Ironically, this reflects LLMs' ability to be better at learning the training distribution and consequently follow inverse scaling law #OTHEREFR .

And while some of the recent research efforts are focused on providing explanations of where the LLM's outputs came from #OTHEREFR , such research is in its infancy.

Deep neural networks trained on domain-specific datasets, including those used in Natural Language Processing (NLP), are most heavily dependent on the quality of the training dataset, which is usually small in size #OTHEREFR as it is costly to engage a large number of domain experts for annotation.

It is thus important to create high-quality training data for language models to perform better.

In this paper, we propose a data-centric framework for Machine Reading Comprehension (MRC) datasets that increases the original dataset quality by both (i) keeping the size of the original dataset fixed, and (ii) augmenting the original dataset by adding new training samples.

MRC is a Natural Language Understanding (NLU) task.

Its goal is to answer questions based on the information provided in a passage .

A Data-Centric Framework For Improving Domain-Specific Machine Reading Comprehension Datasets

Abstract: Low-quality data can cause downstream problems in high-stakes applications. Datacentric approach emphasizes on improving dataset quality to enhance model performance. High-quality datasets are needed for generalpurpose Large Language Models (LLMs) training, as well as for domain-specific models, which are usually small in size as it is costly to engage a large number of domain experts for their creation. Thus, it is vital to ensure high-quality domain-specific training data. In this paper, we propose a framework for enhancing the data quality of original datasets 1. We applied the proposed framework to four biomedical datasets and showed relative improvement of up to 33%/40% for fine-tuning of retrieval/reader models on the BioASQ dataset when using back translation to enhance the original dataset quality.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


Discretizations with finite lower bandwidths for the differential operators studied in this paper are provided by the ultraspherical spectral method #REFR .

Combining Theorems 2.2 and 2.3, we find that for any integer n we have (2.12) z ∈ Ω γ n (z, T ) < ⊂ Λ (T ).

Moreover, since the convergence of γ n (z, T ) to T (z) −1 −1 is locally uniform, these approximations converge to Λ (T ) as n → ∞ without spectral pollution or spectral invisibility.

4 There are generally two ways to make this a practical computation:

• If we have discretizations of the finite-rank operators T (z)P * n and T (z) * P * n that have finite lower bandwidths (or are well approximated by such matrices), we take rectangular truncations capturing the full range #OTHEREFR .

With respect to the appropriate norms, which can differ in the domain and range space, the smallest singular values of the resulting matrices are the same as those of T (z)P * n and T (z) * P * n .

• If we have discretizations of P n T (z) * T (z)P * n and P n T (z)T (z) * P * n , then we can compute their smallest singular values and take square roots to compute γ n #OTHEREFR .

The first method should be preferred over the second wherever possible since it avoids the loss of precision owing to the square root.

There are situations where the second method seems unavoidable #OTHEREFR .

It can be shown that it is not always possible to compute Λ (T ) by discretizing with square, finite sections P n T P * n of T #OTHEREFR .

One must be careful if one wants to compute Λ (T ) by discretizing first.

Avoiding Discretization Issues For Nonlinear Eigenvalue Problems

Abstract: The first step when solving an infinite-dimensional eigenvalue problem is often to discretize it. We show that one must be extremely careful when discretizing nonlinear eigenvalue problems. Using examples, we show that discretization can: (1) introduce spurious eigenvalues, (2) entirely miss spectra, and (3) bring in severe ill-conditioning. While there are many eigensolvers for solving matrix nonlinear eigenvalue problems, we propose a solver for general holomorphic infinitedimensional nonlinear eigenvalue problems that avoids discretization issues, which we prove is stable and converges. Moreover, we provide an algorithm that computes the problem's pseudospectra with explicit error control, allowing verification of computed spectra. The algorithm and numerical examples are publicly available in infNEP, which is a software package written in MATLAB.

A Fast And Well-Conditioned Spectral Method

Abstract: A spectral method is developed for the direct solution of linear ordinary differential equations with variable coefficients. The method leads to matrices which are almost banded, and a numerical solver is presented that takes O(m^2n) operations, where m is the number of Chebyshev points needed to resolve the coefficients of the differential operator and n is the number of Chebyshev coefficients needed to resolve the solution to the differential equation. We prove stability of the method by relating it to a diagonally preconditioned system which has a bounded condition number, in a suitable norm. For Dirichlet boundary conditions, this implies stability in the standard 2-norm. An adaptive QR factorization is developed to efficiently solve the resulting linear system and automatically choose the optimal number of Chebyshev coefficients needed to represent the solution. The resulting algorithm can efficiently and reliably solve for solutions that require as many as a million unknowns.


It has also been suggested that a lack of truthfulness in a robot's statement can negatively affect a human's evaluation of a robot's abilities #REFR .

Works in human factors have attempted to measure perceived safety through physiological signals that can indicate stress, fear, or anxiety #OTHEREFR .

Some of this work has been applied to industrial manipulators #OTHEREFR , though none have been applied in a fast-paced HRC setting to the best of our knowledge.

Communication in HRC -Prior work displayed that verbal communication in HRC affects collaboration as well as users' perception of the robot #OTHEREFR .

Studies have shown that verbal communication between the robot and the user improves subjective and objective metrics on team performance and users prefer a speaking robot over a non-speaking one #OTHEREFR .

Furthermore, #OTHEREFR found that autonomous agents who exchanged information regarding their intention (i.e., deliberative communication #OTHEREFR ) collaborated with humans better than those sharing information in regard to the world state.

Although these studies produced interesting findings about robot communication, the corresponding tasks did not have time pressure, deviating from real-time decision-making.

In our experiment, the robot asserts its intention via a short verbal statement in a male/female voice to inform the participant that it will attempt to hit an incoming table tennis ball.

The verbal communication is done in real-time by an agile robot conducting a high-speed striking maneuver.

Deviating from prior literature, we additionally allow the human to communicate intent with the robot (bidirectional communication), creating a realistic, rapid, proximate HRC scenario where the human must reason about received communication, whether to communicate intent, which agent should hit the ball, and the ball trajectory, all within a short duration.

Agile Robot Manipulation -Agile robotic systems that work alongside human beings is a long-standing goal in manufacturing, where high-speed robotic maneuvers are safely conducted with humans collaborating in proximity with the robot.

The Effect Of Robot Skill Level And Communication In Rapid, Proximate Human-Robot Collaboration

Abstract: As high-speed, agile robots become more commonplace, these robots will have the potential to better aid and collaborate with humans. However, due to the increased agility and functionality of these robots, close collaboration with humans can create safety * Both authors contributed equally to this research.

Planning With Verbal Communication For Human-Robot Collaboration

Abstract: Human collaborators coordinate effectively their actions through both verbal and non-verbal communication. We believe that the the same should hold for human-robot teams. We propose a formalism that enables a robot to decide optimally between taking a physical action toward task completion and issuing an utterance to the human teammate. We focus on two types of utterances: verbal commands, where the robot asks the human to take a physical action, and state-conveying actions, where the robot informs the human about its internal state, which captures the information that the robot uses in its decision making. Human subject experiments show that enabling the robot to issue verbal commands is the most effective form of communicating objectives, while retaining user trust in the robot. Communicating information about the robot’s state should be done judiciously, since many participants questioned the truthfulness of the robot statements when the robot did not provide sufficient explanation about its actions.


Discussion: Example Of Data Interaction Matrices

In this section, we analyze different types of interaction. We select the Circle dataset #REFR .

Its input features are 2D points and it is a binary classification task.

The distribution of the two classes represents two concentric circles, See Figure 3 . The two classes are balanced. Each class contains 300 generated points.

In the matrix, the points are first sorted by their class (0 or 1).

Within each class, the points are sorted based on their input feature x 1 , and then by x 2 .

For example, the point at index 0 is a blue point (class 0) with the smallest x 1 value.

Optimizing Data Shapley Interaction Calculation From O(2 N ) To O(Tn 2 ) For Knn Models

Abstract: With the rapid growth of data availability and usage, quantifying the added value of each training data point has become a crucial process in the field of artificial intelligence. The Shapley values have been recognized as an effective method for data valuation, enabling efficient training set summarization, acquisition, and outlier removal. In this paper, we introduce "STI-KNN", an innovative algorithm that calculates the exact pair-interaction Shapley values for KNN models in O(tn 2) time, which is a significant improvement over the O(2 n) time complexity of baseline methods. By using STI-KNN, we can efficiently and accurately evaluate the value of individual data points, leading to improved training outcomes and ultimately enhancing the effectiveness of artificial intelligence applications.

Scikit-Learn: Machine Learning In Python

Abstract: Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing machine learning to non-specialists using a general-purpose high-level language. Emphasis is put on ease of use, performance, documentation, and API consistency. It has minimal dependencies and is distributed under the simplified BSD license, encouraging its use in both academic and commercial settings. Source code, binaries, and documentation can be downloaded from http://scikit-learn.sourceforge.net.


Moreover, in KD only the output of the teacher model is needed to transmit thus the transmission burden is much less than SL #REFR . Li et al.

As a promising technology, KD achieves success in many fields, since it can transfer knowledge in a big well-trained NN, named teacher model, to an arbitrary small NN, named student model, by providing the outputs of the big NN as soft labels for student model and using these soft labels as an additional gradient to train the student model.

Therefore, the performance of the student model can be improved.

Since in the KD method arbitrary NN can be used as a student model, thus different users can select the different models on demand in FL systems.

Therefore, by using the KD method, the straggle in the training procedure of FL can be avoided.

Because the NNs are not the same between different users, thus there is no need to directly aggregate the NNs of different users, even if the user with more computational resources performs several more training epochs, there is no gradient confusion among different user NNs.

combine migration learning and knowledge refinement for personalized federal learning, and each customer can use private data and a specific model for training. Zhou et al.

proposed distilled one-shot federated learning (DOSFL), thus significantly reducing communication costs while ensuring model accuracy #OTHEREFR .

Although the KD enables users to select different models based on their demands, however, in KD, a big teacher model is needed, and training such a model usually relay on a huge dataset.

Therefore, just using one user's limited dataset cannot train the teacher model well. However.

the teacher model is usually too big to train in users' CPU.

Digital Twin-Assisted Knowledge Distillation Framework For Heterogeneous Federated Learning

Abstract: In this paper, to deal with the heterogeneity in federated learning (FL) systems, a knowledge distillation (KD) driven training framework for FL is proposed, where each user can select its neural network model on demand and distill knowledge from a big teacher model using its own private dataset. To overcome the challenge of train the big teacher model in resource limited user devices, the digital twin (DT) is exploit in the way that the teacher model can be trained at DT located in the server with enough computing resources. Then, during model distillation, each user can update the parameters of its model at either the physical entity or the digital agent. The joint problem of model selection and training offloading and resource allocation for users is formulated as a mixed integer programming (MIP) problem. To solve the problem, Q-learning and optimization are jointly used, where Q-learning selects models for users and determines whether to train locally or on the server, and optimization is used to allocate resources for users based on the output of Q-learning. Simulation results show the proposed DT-assisted KD framework and joint optimization method can significantly improve the average accuracy of users while reducing the total delay.

Detailed Comparison Of Communication Efficiency Of Split Learning And Federated Learning

Abstract: We compare communication efficiencies of two compelling distributed machine learning approaches of split learning and federated learning. We show useful settings under which each method outperforms the other in terms of communication efficiency. We consider various practical scenarios of distributed learning setup and juxtapose the two methods under various real-life scenarios. We consider settings of small and large number of clients as well as small models (1M - 6M parameters), large models (10M - 200M parameters) and very large models (1 Billion-100 Billion parameters). We show that increasing number of clients or increasing model size favors split learning setup over the federated while increasing the number of data samples while keeping the number of clients or model size low makes federated learning more communication efficient.


Communication-Aware Activation Checkpointing (Cac)

We now turn our attention to a second source of redundant communication in large model training, namely activation checkpointing #REFR .

Intermediate activations in a neural network generated during the forward pass need to be stashed in memory as they are required during the backward pass for the gradient computation.

However, for large model training, storing all the activations can leads to tremendous memory overhead.

Activation checkpointing alleviates this issue by storing only a subset of the activations, which are essentially just the input activations of every layer.

During the backward pass of a layer, the remaining activations are re materialized from its stashed input activation by doing a local forward pass for that layer.

Thus, activation checkpointing saves activation memory at the expense of a duplicate forward pass for every layer, and is almost always used for training large neural networks.

A Novel Tensor-Expert Hybrid Parallelism Approach To Scale Mixture-Of-Experts Training

Abstract: A new neural network architecture called Mixture-of-Experts (MoE) has been proposed recently that increases the parameters of a neural network (the base model) by adding sparsely activated expert blocks, without changing the total number of floating point operations for training or inference. In theory, this architecture allows us to train arbitrarily large models while keeping the computational costs same as that of the base model. However, beyond 64 to 128 experts blocks, prior work has observed diminishing returns in the test accuracies of these MoE models. Thus, training high quality MoE models requires us to scale the size of the base models, along with the number of expert blocks. In this work, we propose a novel, three-dimensional, hybrid parallel algorithm that combines tensor, expert, and data parallelism to enable the training of MoE models with 4 − 8× larger base models than the current state-of-theart-DeepSpeed-MoE. We propose memory optimizations in the optimizer step, and communication optimizations that eliminate redundant movement of data. Removing these redundancies provides a speedup of nearly 21%. When training a 40 billion parameter MoE model (6.7 billion base model with 16 experts) on 128 V100 GPUs, our optimizations significantly improve the peak half precision flop/s from 20% to 27%.

Training Deep Nets With Sublinear Memory Cost

Abstract: We propose a systematic approach to reduce the memory consumption of deep neural network training. Specifically, we design an algorithm that costs O( √ n) memory to train a n layer network, with only the computational cost of an extra forward pass per mini-batch. As many of the state-of-the-art models hit the upper bound of the GPU memory, our algorithm allows deeper and more complex models to be explored, and helps advance the innovations in deep learning research. We focus on reducing the memory cost to store the intermediate feature maps and gradients during training. Computation graph analysis is used for automatic in-place operation and memory sharing optimizations. We show that it is possible to trade computation for memory giving a more memory efficient training algorithm with a little extra computation cost. In the extreme case, our analysis also shows that the memory consumption can be reduced to O(logn) with as little as O(n logn) extra cost for forward computation. Our experiments show that we can reduce the memory cost of a 1,000-layer deep residual network from 48G to 7G on ImageNet problems. Similarly, significant memory cost reduction is observed in training complex recurrent neural networks on very long sequences.


As shown in #REFR , in the underparametrized setting, there exist input data such that for all output data from a set of positive Lebesgue measure there does not exist minimizers in the optimization landscape.

However, for general measures µ not necessarily consisting of a finite number of Dirac measures, the literature on the existence of global minima is very limited.

There exist positive results for the approximation of functions in the space

) with shallow feedforward ANNs using heavyside activation #OTHEREFR , the approximation of one-dimensional Lipschitz continuous target functions with shallow feedforward ANNs using ReLU activation and the standard mean square error #OTHEREFR , and the approximation of multi-dimensional, continuous target functions with shallow residual ANNs using ReLU activation, see #OTHEREFR .

Conversely, for several common (smooth) activations such as the standard logistic activation, softplus, arctan, hyperbolic tangent and softsign there, generally, do not exist minimizers in the optimization landscape for smooth target functions (or even polynomials), see #OTHEREFR and #OTHEREFR .

This phenomenon can also be observed in empirical risk minimization for the hyperbolic tangent activation.

It remains an open problem whether for ReLU activation this phenomenon prevails.

In this article, we prove, for the first time, existence results for shallow feedforward ReLU ANNs with multi-dimensional input space.

Interestingly, minimizers exist under very mild assumptions on the optimization problem.

This existence property indicates the robustness of ReLU activation and may be a reason for its success in practical applications. For the proof we proceed as follows.

First, we show existence of minimizers in an extended target space that comprises of the representable responses of ANNs and additional discontinuous generalized responses.

On The Existence Of Optimal Shallow Feedforward Networks With Relu Activation

Abstract: We prove existence of global minima in the loss landscape for the approximation of continuous target functions using shallow feedforward artificial neural networks with ReLU activation. This property is one of the fundamental artifacts separating ReLU from other commonly used activation functions. We propose a kind of closure of the search space so that in the extended space minimizers exist. In a second step, we show under mild assumptions that the newly added functions in the extension perform worse than appropriate representable ReLU networks. This then implies that the optimal response in the extended target space is indeed the response of a ReLU network.

Best K-Layer Neural Network Approximations

Abstract: We investigate the geometry of the empirical risk minimization problem for $k$-layer neural networks. We will provide examples showing that for the classical activation functions $\sigma(x)= 1/\bigl(1 + \exp(-x)\bigr)$ and $\sigma(x)=\tanh(x)$, there exists a positive-measured subset of target functions that do not have best approximations by a fixed number of layers of neural networks. In addition, we study in detail the properties of shallow networks, classifying cases when a best $k$-layer neural network approximation always exists or does not exist for the ReLU activation $\sigma=\max(0,x)$. We also determine the dimensions of shallow ReLU-activated networks.


This example violates the consistency condition set out in #REFR . We elaborate in Appendix A.

Further, U → V ∈ E if U and V are micro-variables and U is a parent of V , and S ↔S ∈ C ifS is the aggregation of S.

The first consequence of aggregation is that the aggregated variables may no longer show a well-defined causal relation: Example 1 Consider the cause-effect relation X → Y , with X = (X 1 , X 2 ) and Y = (Y 1 , Y 2 ). For j = 1, 2, set

DefineX := π X (X 1 , X 2 ) = X 1 + X 2 , and likewiseȲ := π Y (Y 1 , Y 2 ) = Y 1 + Y 2 .

Then, for α 1 = α 2 , the effect of settingX tox onȲ is ill-defined without any further specification.

In reality, the case where α 1 = α 2 is rare, and the generic cases are when α 1 = α 2 . Since the operation 'settingX tō

x' is not well-defined a priori, its micro-realization needs to be specified whenever talking about it.

More precisely, we need to specify a distribution according to which we randomize the micro state (the 'joint manipulation' in #OTHEREFR ): Definition 5 (macro intervention) Given an aggregation map, π : X →X , a (stochastic) macro-intervention atx, denoted do(x), is a probability measure over X , denoted

P do π,x (X), such that π(x) =x ∀x ∈ supp(P do π,x (X))

Meaningful Causal Aggregation And Paradoxical Confounding

Abstract: In aggregated variables the impact of interventions is typically ill-defined because different micro-realizations of the same macro-intervention can result in different changes of downstream macro-variables. We show that this ill-definedness of causality on aggregated variables can turn unconfounded causal relations into confounded ones and vice versa, depending on the respective microrealization. We argue that it is practically infeasible to only use aggregated causal systems when we are free from this ill-definedness. Instead, we need to accept that macro causal relations are typically defined only with reference to the micro states. On the positive side, we show that causeeffect relations can be aggregated when the macro interventions are such that the distribution of micro states is the same as in the observational distribution and also discuss generalizations of this observation.

Causal Consistency Of Structural Equation Models

Abstract: Complex systems can be modelled at various levels of detail. Ideally, causal models of the same system should be consistent with one another in the sense that they agree in their predictions of the effects of interventions. We formalise this notion of consistency in the case of Structural Equation Models (SEMs) by introducing exact transformations between SEMs. This provides a general language to consider, for instance, the different levels of description in the following three scenarios: (a) models with large numbers of variables versus models in which the `irrelevant' or unobservable variables have been marginalised out; (b) micro-level models versus macro-level models in which the macro-variables are aggregate features of the micro-variables; (c) dynamical time series models versus models of their stationary behaviour. Our analysis stresses the importance of well specified interventions in the causal modelling process and sheds light on the interpretation of cyclic SEMs.


Currently, neuro-symbolic solutions rely either on costly pre-training methods or on additional supervision at the symbolic representation level provided by the neural network, in order to effectively utilize subsequent learning feedback from the logical component #REFR .

Recently, neuro-symbolic learning has received attention as a new approach for integrating symbolic-based and sub-symbolic methods based on neural networks.

This integration provides new capabilities in terms of perception and reasoning.

This traditional top-down learning paradigm is subject to the problem of representational collapse.

To gain a clearer understanding of the problem, imagine we have a tuple of three images, each of which contains a single digit (e.g., < 3, 5, 8 >).

Along with this, we have information about the logical relationships between these digits (e.g., the third digit is the sum of the first two).

Note that this task introduces less supervision compared to the digit addition experiment typically used in neuro-symbolic systems (i.e. the information about the sum is not provided).

Current neuro-symbolic solutions can easily solve the task by mapping all input data onto the same symbol 0 and clearly solve the constrained task.

Learning Symbolic Representations Through Joint Generative And Discriminative Training

Abstract: We introduce GEDI, a Bayesian framework that combines existing selfsupervised learning objectives with likelihood-based generative models. This framework leverages the benefits of both GEnerative and DIscriminative approaches, resulting in improved symbolic representations over standalone solutions. Additionally, GEDI can be easily integrated and trained jointly with existing neuro-symbolic frameworks without the need for additional supervision or costly pre-training steps. We demonstrate through experiments on real-world data, including SVHN, CIFAR10, and CIFAR100, that GEDI outperforms existing selfsupervised learning strategies in terms of clustering performance by a significant margin. The symbolic component further allows it to leverage knowledge in the form of logical constraints to improve performance in the small data regime.

Deepproblog: Neural Probabilistic Logic Programming

Abstract: We introduce DeepProbLog, a probabilistic logic programming language that incorporates deep learning by means of neural predicates. We show how existing inference and learning techniques can be adapted for the new language. Our experiments demonstrate that DeepProbLog supports (i) both symbolic and subsymbolic representations and inference, (ii) program induction, (iii) probabilistic (logic) programming, and (iv) (deep) learning from examples. To the best of our knowledge, this work is the first to propose a framework where general-purpose neural networks and expressive probabilistic-logical modeling and reasoning are integrated in a way that exploits the full expressiveness and strengths of both worlds and can be trained end-to-end based on examples.


Preliminaries: Slot Attention Autoencoder

Each slot is then decoded independently with a spatial broadcast decoder #REFR g producing slot reconstruc-tionsÎ i 's and segmentation masks m i 's.

, called slots, in the bottleneck, where n may change anytime during training or testing.

Slots are initially sampled from a Gaussian with learned mean and standard deviation, without conditioning additional variables, including the slot ID.

This affords changing the number of slots without re-training, and yields invariance to permutation of the ordering of the slots, which are updated iteratively using dot-product attention normalized over slots, which fosters competition among slots.

The result is passed through a Gated Recurrent Unit (GRU) with a multi-layer perceptron (MLP) to yield the update residual for slots.

All parameters are shared among slots to preserve permutation symmetry.

The final image reconstruction isÎ = n i=1Î i m i where denotes elementwise multiplication.

SAN is trained by minimizing a reconstruction loss (typically MSE) between I andÎ.

Divided Attention: Unsupervised Multi-Object Discovery With Contextually Separated Slots

Abstract: We introduce a method to segment the visual field into independently moving regions, trained with no ground truth or supervision. It consists of an adversarial conditional encoder-decoder architecture based on Slot Attention, modified to use the image as context to decode optical flow without attempting to reconstruct the image itself. In the resulting multi-modal representation, one modality (flow) feeds the encoder to produce separate latent codes (slots), whereas the other modality (image) conditions the decoder to generate the first (flow) from the slots. This design frees the representation from having to encode complex nuisance variability in the image due to, for instance, illumination and reflectance properties of the scene. Since customary autoencoding based on minimizing the reconstruction error does not preclude the entire flow from being encoded into a single slot, we modify the loss to an adversarial criterion based on Contextual Information Separation. The resulting min-max optimization fosters the separation of objects and their assignment to different attention slots, leading to Divided Attention, or DivA. DivA outperforms recent unsupervised multi-object motion segmentation methods while tripling run-time speed up to 104FPS and reducing the performance gap from supervised methods to 12% or less. DivA can handle different numbers of objects and different image sizes at training and test time, is invariant to permutation of object labels, and does not require explicit regularization.

Spatial Broadcast Decoder: A Simple Architecture For Learning Disentangled Representations In Vaes

Abstract: We present a simple neural rendering architecture that helps variational autoencoders (VAEs) learn disentangled representations. Instead of the deconvolutional network typically used in the decoder of VAEs, we tile (broadcast) the latent vector across space, concatenate fixed X- and Y-"coordinate" channels, and apply a fully convolutional network with 1x1 stride. This provides an architectural prior for dissociating positional from non-positional features in the latent distribution of VAEs, yet without providing any explicit supervision to this effect. We show that this architecture, which we term the Spatial Broadcast decoder, improves disentangling, reconstruction accuracy, and generalization to held-out regions in data space. It provides a particularly dramatic benefit when applied to datasets with small objects. We also emphasize a method for visualizing learned latent spaces that helped us diagnose our models and may prove useful for others aiming to assess data representations. Finally, we show the Spatial Broadcast Decoder is complementary to state-of-the-art (SOTA) disentangling techniques and when incorporated improves their performance.

segmentation masks
spatial broadcast decoder


For the ScanNet-v2 dataset, we conduct experiments on the "3D Semantic label with Limited Annotations" benchmark #REFR , where only 20 fixed points are labeled in each room scene.

We train the segmentation model on Area 1, 2, 3, 4, 6 and test it on Area 5, respectively.

The ScanNet-v2 dataset contains 1513 3D scans with 20 categories, which are divided into 1201, 312, and 100 scans for training, validation, and testing, respectively. Weak Annotation Scheme.

On the S3DIS dataset, we follow existing methods #OTHEREFR to annotate the training data under the "OTOC" setting.

For each object, we randomly select a point as the labeled one with the same probability.

Consequently, only 0.02% of points have been labeled in the whole dataset.

If there is no special declaration, we implemented our proposed RAC-Net with the KPConv rigid backbone, which is same as the latest DAT model #OTHEREFR .

We used the SGD to train the model with a batch size of 2 and a learning rate of 0.01.

We set the confidence threshold τ as 0.7 and the uncertainty threshold κ as 0.05.

All experiments in this paper were conducted in an identical environment (Hardware: single NVIDIA RTX 3090 GPU; Software: PyTorch 1.7.0 and CUDA 11.0).

Reliability-Adaptive Consistency Regularization For Weakly-Supervised Point Cloud Segmentation

Abstract: Weakly-supervised point cloud segmentation with extremely limited labels is highly desirable to alleviate the expensive costs of collecting densely annotated 3D points. This paper explores to apply the consistency regularization that is commonly used in weakly-supervised learning, for its point cloud counterpart with multiple data-specific augmentations, which has not been well studied. We observe that the straightforward way of applying consistency constraints to weakly-supervised point cloud segmentation has two major limitations: noisy pseudo labels due to the conventional confidence-based selection and insufficient consistency constraints due to discarding unreliable pseudo labels. Therefore, we propose a novel Reliability-Adaptive Consistency Network (RAC-Net) to use both prediction confidence and model uncertainty to measure the reliability of pseudo labels and apply consistency training on all unlabeled points while with different consistency constraints for different points based on the reliability of corresponding pseudo labels. Experimental results on the S3DIS and ScanNet-v2 benchmark datasets show that our model achieves superior performance in weakly-supervised point cloud segmentation. The code will be released.

Scannet: Richly-Annotated 3D Reconstructions Of Indoor Scenes

Abstract: A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets. Unfortunately, in the context of RGB-D scene understanding, very little data is available – current datasets cover a small range of scene views and have limited semantic annotations. To address this issue, we introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations. To collect this data, we designed an easy-to-use and scalable RGB-D capture system that includes automated surface reconstruction and crowdsourced semantic annotation. We show that using this data helps achieve state-of-the-art performance on several 3D scene understanding tasks, including 3D object classification, semantic voxel labeling, and CAD model retrieval.


The results in Table 2 show that ALCIR consistently outperforms all baselines in terms of the accuracy measures on all three datasets, usually with a large margin #REFR .

Table 2 reports the main results of our experiments, where the category is given to the recommenders.

We name our proposed model ALCIR, which stands for Adversarial Learning for Complementary Item Recommendation.

To analyze the contribution of the individual elements of our architecture, we report results for (a) ALCIR-Sup, which only consists of the supervised learning component from Section 4 and (b) ALCIR, which corresponds to the full model as described in Section 5.

We additionally observe that already the supervised component of our model (ALCIR-Supervised) performs better than the baselines.

The adversarial component is then successful in even further increasing these already strong results.

Considering the ranking of the other models, we notice that the popularity-based approach represents a baseline that can be difficult to beat.

The DCF model and even the improved DCF-Hard model never reach the accuracy levels of the popularity-based method.

P-Companion works better, but still does not reach the Hit Rate values of popular-item recommendations.

Semi-Supervised Adversarial Learning For Complementary Item Recommendation

Abstract: Complementary item recommendations are a ubiquitous feature of modern e-commerce sites. Such recommendations are highly effective when they are based on collaborative signals like co-purchase statistics. In certain online marketplaces, however, e.g., on online auction sites, constantly new items are added to the catalog. In such cases, complementary item recommendations are often based on item side-information due to a lack of interaction data. In this work, we propose a novel approach that can leverage both item side-information and labeled complementary item pairs to generate effective complementary recommendations for cold items, i.e., for items for which no co-purchase statistics yet exist. Given that complementary items typically have to be of a different category than the seed item, we technically maintain a latent space for each item category. Simultaneously, we learn to project distributed item representations into these category spaces to determine suitable recommendations. The main learning process in our architecture utilizes labeled pairs of complementary items. In addition, we adopt ideas from Cycle Generative Adversarial Networks (CycleGAN) to leverage available item information even in case no labeled data exists for a given item and category. Experiments on three e-commerce datasets show that our method is highly effective. CCS CONCEPTS • Information systems → Recommender systems.

Deep Item-Based Collaborative Filtering For Sparse Implicit Feedback

Abstract: Recommender systems are ubiquitous in the domain of e-commerce, used to improve the user experience and to market inventory, thereby increasing revenue for the site. Techniques such as item-based collaborative filtering are used to model users' behavioral interactions with items and make recommendations from items that have similar behavioral patterns. However, there are challenges when applying these techniques on extremely sparse and volatile datasets. On some e-commerce sites, such as eBay, the volatile inventory and minimal structured information about items make it very difficult to aggregate user interactions with an item. In this work, we describe a novel deep learning-based method to address the challenges. We propose an objective function that optimizes a similarity measure between binary implicit feedback vectors between two items. We demonstrate formally and empirically that a model trained to optimize this function estimates the log of the cosine similarity between the feedback vectors. We also propose a neural network architecture optimized on this objective. We present the results of experiments comparing the output of the neural network with traditional item-based collaborative filtering models on an implicit-feedback dataset, as well as results of experiments comparing different neural network architectures on user purchase behavior on eBay. Finally, we discuss the results of an A/B test that show marked improvement of the proposed technique over eBay's existing collaborative filtering recommender system.


Policy gradient approaches are a model-free algorithm suitable for offline RL, and are analyzed in #REFR .

Learning Controllers Offline Learning a controller from offline data is a familiar paradigm for control theorist and practitioners.

It typically consists of system identification, followed by robust #OTHEREFR or certainty-equivalent #OTHEREFR ) control design, see Figure 2 .

Recent work provides finite sample guarantees for such methods #OTHEREFR .

Upper and lower bounds on the sample complexity of stabilization from offline data are presented in #OTHEREFR .

The RL community has a similar paradigm, known as offline RL #OTHEREFR .

Lower bounds on the variance of the gradient estimates in policy gradient approaches are supplied in #OTHEREFR .

Lower bounds for offline linear control are also studied in #OTHEREFR with the objective of designing optimal experiments.

We instead focus on the LQR setting to understand the dependence of the excess cost on interpretable system-theoretic quantities.

Online LQR The problem of learning the optimal LQR controller online has a rich history beginning withÅström and #OTHEREFR .

Regret minimization was introduced in Lai (1986); #OTHEREFR .

The Fundamental Limitations Of Learning Linear-Quadratic Regulators

Abstract: We present a local minimax lower bound on the excess cost of designing a linear-quadratic controller from offline data. The bound is valid for any offline exploration policy that consists of a stabilizing controller and an energy bounded exploratory input. The derivation leverages a relaxation of the minimax estimation problem to Bayesian estimation, and an application of Van Trees' inequality. We show that the bound aligns with system-theoretic intuition. In particular, we demonstrate that the lower bound increases when the optimal control objective value increases. We also show that the lower bound increases when the system is poorly excitable, as characterized by the spectrum of the controllability gramian of the system mapping the noise to the state and the H ∞ norm of the system mapping the input to the state. We further show that for some classes of systems, the lower bound may be exponential in the state dimension, demonstrating exponential sample complexity for learning the linear-quadratic regulator offline.

Global Convergence Of Policy Gradient Methods For The Linear Quadratic Regulator

Abstract: Direct policy gradient methods for reinforcement learning and continuous control problems are a popular approach for a variety of reasons: 1) they are easy to implement without explicit knowledge of the underlying model 2) they are an "end-to-end" approach, directly optimizing the performance metric of interest 3) they inherently allow for richly parameterized policies. A notable drawback is that even in the most basic continuous control problem (that of linear quadratic regulators), these methods must solve a non-convex optimization problem, where little is understood about their efficiency from both computational and statistical perspectives. In contrast, system identification and model based planning in optimal control theory have a much more solid theoretical footing, where much is known with regards to their computational and statistical properties. This work bridges this gap showing that (model free) policy gradient methods globally converge to the optimal solution and are efficient (polynomially so in relevant problem dependent quantities) with regards to their sample and computational complexities.


Yang (2020) devised a systematic method to compute such a kernel corresponding to a large number of neural network architectures, and #REFR examined the case of convolutional residual networks.

#OTHEREFR introduced deep kernels by allegory with deep networks, and #OTHEREFR

(2018) showed that these kernels do indeed correspond to multi-layer neural networks with Gaussian process priors.

They also showed that in practical settings, the Gaussian process limit arises rather quickly, with networks of width of order 100 behaving as Gaussian processes.

Further work has demonstrated that Gaussian process behavior of deep networks remains when the network is trained using gradient descent. #OTHEREFR

(2018) introduced the neural tangent kernel (NTK), which describes how a trained neural network exhibits Gaussian process behavior in the large-width regime.

A common theme in these works is that whilst Gaussian-process equivalents to large networks exist, they are often expensive to compute.

For example in #OTHEREFR , constructing the kernel matrix for an image-recognition task took 10 hours on a 1000-core machine.

This work brings together the computational method from #OTHEREFR

(2020) with some of the advances in understanding the (kernels of) Gaussian processes connection to large DNNs.

We first review some connections between RKHS eigenstructures and deep model structures; we clarify how the corresponding RKHS gives rise to polynomial kernels with spherical harmonics as the orthogonal basis; we introduce kernels of continuous depth so that depth may be estimated as a kernel hyper-parameter; and we introduce variational learning of spherical-harmonic phases, which enables scaling to larger input dimensions than previously.

Sparse Gaussian Processes With Spherical Harmonic Features Revisited

Abstract: We revisit the Gaussian process model with spherical harmonic features and study connections between the associated RKHS, its eigenstructure, and deep models. Based on this, we introduce a new class of kernels which correspond to deep models of continuous depth. In our formulation, depth can be estimated as a kernel hyper-parameter by optimizing the evidence lower bound. Further, we introduce sparseness in the eigenbasis by variational learning of the spherical harmonic phases. This enables scaling to larger input dimensions than previously, while also allowing for learning of high frequency variations. We validate our approach on machine learning benchmark datasets.

Deep Convolutional Networks As Shallow Gaussian Processes

Abstract: We show that the output of a (residual) convolutional neural network (CNN) with an appropriate prior over the weights and biases is a Gaussian process (GP) in the limit of infinitely many convolutional filters, extending similar results for dense networks. For a CNN, the equivalent kernel can be computed exactly and, unlike"deep kernels", has very few parameters: only the hyperparameters of the original CNN. Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one filter per layer. The kernel equivalent to a 32-layer ResNet obtains 0.84% classification error on MNIST, a new record for GPs with a comparable number of parameters.


However, this loss tends to generate repetitive and generic content #REFR and does not take into account synonyms, various sentences structures or the semantic closeness.

Following this idea, the Automated Audio Captioning (AAC) task appeared in 2017 #OTHEREFR and aims to create systems that generate a sentence written in natural language that describes an audio file.

The audio can contain various sound events (human, natural, domestic, urban, music, effects...) of different lengths, recorded with different devices and in different scenes.

The description can contain any kind of detail in the audio, with temporal or spatial relations between them (followed by, in the background...) or different characterizations (high-pitched, short, repetitive...).

Since the descriptions are written by humans, we need to consider different words used to describe similar sounds (Birds are calling / chirping / singing / tweeting), different sentence structures (A door that needs to be oiled / A door with squeaky hinges), subjectivity (Man speaks in a foreign language), high-level descriptions (A vulgar man speaks / Unintelligible conversation), and vagueness (Someone speaks instead of A man gives a speech over a reverberating microphone).

In AAC, most approaches use deep learning models trained with the standard Cross-Entropy (CE) loss.

Several studies introduced another criterion, the Self-Critical Sequence Training #OTHEREFR (SCST) used in reinforcement learning to fine-tune the model directly on a metric instead of the loss.

This technique relies on sampling the next word to generate a new sentence.

If this sentence has a higher score than the original one, the model is rewarded and the outputs probabilities for this new sentence are encouraged.

However, this technique leads to degenerated sentences #OTHEREFR , with repetitive n-grams without syntactical correctness.

Motivated by the limitations of CE and SCST, in this work, we attempted to add a Sentence Embedding Regression (SER) loss used in #OTHEREFR to improve our model.

Multitask Learning In Audio Captioning: A Sentence Embedding Regression Loss Acts As A Regularizer

Abstract: In this work, we propose to study the performance of a model trained with a sentence embedding regression loss component for the Automated Audio Captioning task. This task aims to build systems that can describe audio content with a single sentence written in natural language. Most systems are trained with the standard Cross-Entropy loss, which does not take into account the semantic closeness of the sentence. We found that adding a sentence embedding loss term reduces overfitting, but also increased SPIDEr from 0.397 to 0.418 in our first setting on the AudioCaps corpus. When we increased the weight decay value, we found our model to be much closer to the current stateof-the-art methods, with a SPIDEr score up to 0.444 compared to a 0.475 score. Moreover, this model uses eight times less trainable parameters. In this training setting, the sentence embedding loss has no more impact on the model performance.

The Curious Case Of Neural Text Degeneration

Abstract: Despite considerable advances in neural language modeling, it remains an open question what the best decoding strategy is for text generation from a language model (e.g. to generate a story). The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration — output text that is bland, incoherent, or gets stuck in repetitive loops. To address this we propose Nucleus Sampling, a simple but effective method to draw considerably higher quality text out of neural language models. Our approach avoids text degeneration by truncating the unreliable tail of the probability distribution, sampling from the dynamic nucleus of tokens containing the vast majority of the probability mass. To properly examine current maximization-based and stochastic decoding methods, we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition. Our results show that (1) maximization is an inappropriate decoding objective for open-ended text generation, (2) the probability distributions of the best current language models have an unreliable tail which needs to be truncated during generation and (3) Nucleus Sampling is the best decoding strategy for generating long-form text that is both high-quality — as measured by human evaluation — and as diverse as human-written text.


The following result was proved in #REFR It permutes the positions of the m tape symbols according to π.

3.2. The oblivious Turing machines are finitely generated.

In this section, we show that OBpZ d , n, kq is finitely generated.

Our proof is based on the existence of strongly universal reversible gates for permutations of A m , which can be found in #OTHEREFR for the binary alphabet case, and generalized to other alphabets in #OTHEREFR .

We need a finite generating set for permutations of QˆΣ m , and hence the proof in #OTHEREFR has to be adjusted to account for non-homogeneous alphabet sizes (that is, due to possibly having n ‰ k).

Let us remark that the case n " 1 is trivial: The group LPpZ d , 1, kq is finite and SHIFTpZ d , 1, kq is generated by the single step moves. We hence assume that n ě 2.

Now we can conjugate the prefix applicationf using a rewiring to getf π " r´1 π˝f˝r π , we callf π an application of f in the coordinates Based on the decomposition in Figure 3 , we first conclude that any controlled 3-cycle f of QˆΣ m is a composition of four applications of controlled swaps of QˆΣ m´2 .

In the figure, the components of QˆΣ m have been ordered in parallel horizontal wires, the Q-component being among the topmost three wires.

Referring to the symbols in the illustration, the gate on the left is a generic 3-cycle ppszabcdw ptzabcdw qszabcdwq where one of the first three wires is the Q-component, a, b, c, d P Σ and w P Σ m´6 .

The proposed decomposition consists of two different controlled swaps p 1 and p 2 applied twice in the order f " p 1 p 2 p 1 p 2 .

Because p 1 and p 2 are involutions, the decomposition amounts to identity unless the input is of the form xyzabcdw where x P tp, qu and y P ts, tu.

The Group Of Reversible Turing Machines: Subgroups, Generators And Computability

Abstract: We study an abstract group of reversible Turing machines. In our model, each machine is interpreted as a homeomorphism over a space which represents a tape filled with symbols and a head carrying a state. These homeomorphisms can only modify the tape at a bounded distance around the head, change the state and move the head in a bounded way. We study three natural subgroups arising in this model: the group of finite-state automata, which generalizes the topological full groups studied in topological dynamics and the theory of orbit-equivalence; the group of oblivious Turing machines whose movement is independent of tape contents, which generalizes lamplighter groups and has connections to the study of universal reversible logical gates; and the group of elementary Turing machines, which are the machines which are obtained by composing finite-state automata and oblivious Turing machines. We show that both the group of oblivious Turing machines and that of elementary Turing machines are finitely generated, while the group of finite-state automata and the group of reversible Turing machines are not. We show that the group of elementary Turing machines has undecidable torsion problem. From this, we also obtain that the group of cellular automata (more generally, the automorphism group of any uncountable one-dimensional sofic subshift) contains a finitely-generated subgroup with undecidable torsion problem. We also show that the torsion problem is undecidable for the topological full group of a full Z d-shift on a non-trivial alphabet if and only if d ě 2.

Strongly Universal Reversible Gate Sets

Abstract: It is well-known that the Toffoli gate and the negation gate together yield a universal gate set, in the sense that every permutation of $\{0,1\}^n$ can be implemented as a composition of these gates. Since every bit operation that does not use all of the bits performs an even permutation, we need to use at least one auxiliary bit to perform every permutation, and it is known that one bit is indeed enough. Without auxiliary bits, all even permutations can be implemented. We generalize these results to non-binary logic: If $A$ is a finite set of odd cardinality then a finite gate set can generate all permutations of $A^n$ for all $n$, without any auxiliary symbols. If the cardinality of $A$ is even then, by the same argument as above, only even permutations of $A^n$ can be implemented for large $n$, and we show that indeed all even permutations can be obtained from a finite universal gate set. We also consider the conservative case, that is, those permutations of $A^n$ that preserve the weight of the input word. The weight is the vector that records how many times each symbol occurs in the word. It turns out that no finite conservative gate set can, for all $n$, implement all conservative even permutations of $A^n$ without auxiliary bits. But we provide a finite gate set that can implement all those conservative permutations that are even within each weight class of $A^n$.


4) Convergence Of The Inferred Network (Rq4):

When η = 0, our loss function in (8) boils down to the case in #REFR , which only takes into account the structure-community relationship without node metadata, leading to unsatisfactory performance.

• The AUC score between G (t) and G tends to increase with node queries.

This suggests that EC-SiamNet is indeed effective in accurately inferring edges, leading to a fast convergence of the inferred network towards the underlying true network, as noted in Remark 1. 5) Effect of hyperparameters (RQ5): In Figs.

6 and 7, we investigate the impact of two key hyperparameters, including η and λ, on the performance of META-CODE in terms of the NMI when the Facebook 348 and Engineering datasets are used.

When a hyperparameter varies so that its effect is clearly revealed, another parameter is set to the pivot value addressed in Section V-D. Our findings are as follows.

• The effect of η: From Fig 6, the maximum NMI is achieved at η = 1.5 and η = 1.0 on Facebook 348 and Engineering, respectively.

On the other hand, setting η to a high value leads to relatively low performance for both datasets since the metadata-community relationship is over-emphasized during training.

Hence, it is crucial to suitably determine the value of η in guaranteeing satisfactory performance.

• The effect of λ: From Fig 7, the maximum NMI is achieved at λ = 2.0 and λ = 3.0 on Facebook 348 and Engineering, respectively.

When λ = 0, our node query strategy in (9) focuses only on selecting nodes belonging to more communities.

In contrast, setting λ to a high value leads to a negative effect on the performance because it over-emphasizes the selection of nodes distributed over diverse communities.

Graph Neural Network-Aided Exploratory Learning For Community Detection With Unknown Topology

Abstract: In social networks, the discovery of community structures has received considerable attention as a fundamental problem in various network analysis tasks. However, due to privacy concerns or access restrictions, the network structure is often unknown, thereby rendering established community detection approaches ineffective without costly network topology acquisition. To tackle this challenge, we present META-CODE, a novel end-to-end solution for detecting overlapping communities in networks with unknown topology via exploratory learning aided by easy-to-collect node metadata. Specifically, META-CODE consists of three iterative steps in addition to the initial network inference step: 1) node-level community-affiliation embeddings based on graph neural networks (GNNs) trained by our new reconstruction loss, 2) network exploration via communityaffiliation-based node queries, and 3) network inference using an edge connectivity-based Siamese neural network model from the explored network. Through comprehensive evaluations using five real-world datasets, we demonstrate that META-CODE exhibits (a) its superiority over benchmark community detection methods, (b) empirical evaluations as well as theoretical findings to see the effectiveness of our node query, (c) the influence of each module, and (d) its computational efficiency.

Overlapping Community Detection With Graph Neural Networks

Abstract: Community detection is a fundamental problem in machine learning. While deep learning has shown great promise in many graphrelated tasks, developing neural models for community detection has received surprisingly little attention. The few existing approaches focus on detecting disjoint communities, even though communities in real graphs are well known to be overlapping. We address this shortcoming and propose a graph neural network (GNN) based model for overlapping community detection. Despite its simplicity, our model outperforms the existing baselines by a large margin in the task of community recovery. We establish through an extensive experimental evaluation that the proposed model is effective, scalable and robust to hyperparameter settings. We also perform an ablation study that confirms that GNN is the key ingredient to the power of the proposed model.


This is because it performed incredibly well among the algorithms tested for this purpose and was the second most computationally efficient algorithm tested #REFR .

Statistical techniques are the simplest of the three types of document similarity algorithms.

They compare text by first turning the sentences into vectors, and then comparing said vectors.

The most used and most effective way of comparison is through cosine similarity, but other methods such as Euclidean distance are occasionally used.

Of the possible preprocessors, Sentence-BERT was judged to be most ideal for the purposes of this paper.

Neural networks are another possible avenue of document similarity, and a very effective one.

The basics of neural network-based techniques is that they are first fed training documents of pairs of texts that are either similar to each other or different.

Gradually, the neural network learns which pairs of texts are similar and which pairs are different through understanding semantic information.

Generally, the text is run through a tokenizer first, which allows the semantic information to be more understandable to the network.

Some models, like BERT and XLNet, are pre-trained, which allows the network to have accurate predictions without needing long training times and massive corpora for each individual task.

A Comparison Of Document Similarity Algorithms

Abstract: Document similarity is an important part of Natural Language Processing and is most commonly used for plagiarism-detection and text summarization. Thus, finding the overall most effective document similarity algorithm could have a major positive impact on the field of Natural Language Processing. This report sets out to examine the numerous document similarity algorithms, and determine which ones are the most useful. It addresses the most effective document similarity algorithm by categorizing them into 3 types of document similarity algorithms: statistical algorithms, neural networks, and corpus/knowledge-based algorithms. The most effective algorithms in each category are also compared in our work using a series of benchmark datasets and evaluations that test every possible area that each algorithm could be used in.

Sentence-Bert: Sentence Embeddings Using Siamese Bert-Networks

Abstract: BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. ::: In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. ::: We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.

algorithms
computationally efficient algorithm


Compared to Soft-DTW #REFR , which produces soft alignments, DecDTW outputs the optimal, continuous time warp.

Our DecDTW layer encodes an implicit function φ = DecDTW(x, y, λ, s min , s max , b min , b max ), which yields the optimal warp given input signals x, y, regularisation λ and constraints

s min = {s min i } m i=1 , s max = {s max i } m i=1 , b min = {b min i } m i=1 , b max = {b max i } m i=1 .

The warp φ can be used in a downstream loss J(φ ), e.g., the error between predicted warp φ and a ground truth warp φ gt .

The GDTW discrepancy is recovered by setting J =f .

The DecDTW forward pass solves the GDTW problem given by Equation 4 given the input parameters.

We solve Equation 4 using a dynamic programming (DP) approach as proposed in #OTHEREFR instead of calling a general-purpose NLP solver; this is to minimise computation time.

The DP approach uses the fact that φ lies on a compact subset of R m defined by the global bounds, allowing for efficient discretisation.

Finding the globally optimal solution to the discretised version of Equation 4 can be done using DP.

The resultant solution φ is an approximation to the solution of the continuous NLP, with the approximation error dependent on the resolution of discretisation.

Deep Declarative Dynamic Time Warping For End-To-End Learning Of Alignment Paths

Abstract: This paper addresses learning end-to-end models for time series data that include a temporal alignment step via dynamic time warping (DTW). Existing approaches to differentiable DTW either differentiate through a fixed warping path or apply a differentiable relaxation to the min operator found in the recursive steps used to solve the DTW problem. We instead propose a DTW layer based around bilevel optimisation and deep declarative networks, which we name DecDTW. By formulating DTW as a continuous, inequality constrained optimisation problem, we can compute gradients for the solution of the optimal alignment (with respect to the underlying time series) using implicit differentiation. An interesting byproduct of this formulation is that DecDTW outputs the optimal warping path between two time series as opposed to a soft approximation, recoverable from Soft-DTW. We show that this property is particularly useful for applications where downstream loss functions are defined on the optimal alignment path itself. This naturally occurs, for instance, when learning to improve the accuracy of predicted alignments against ground truth alignments. We evaluate DecDTW on two such applications, namely the audio-to-score alignment task in music information retrieval and the visual place recognition task in robotics, demonstrating state-of-the-art results in both.

Soft-Dtw: A Differentiable Loss Function For Time-Series

Abstract: We propose in this paper a differentiable learning loss between time series, building upon the celebrated dynamic time warping (DTW) discrepancy. Unlike the Euclidean distance, DTW can compare time series of variable size and is robust to shifts or dilatations across the time dimension. To compute DTW, one typically solves a minimal-cost alignment problem between two time series using dynamic programming. Our work takes advantage of a smoothed formulation of DTW, called soft-DTW, that computes the soft-minimum of all alignment costs. We show in this paper that soft-DTW is a differentiable loss function, and that both its value and gradient can be computed with quadratic time/space complexity (DTW has quadratic time but linear space complexity). We show that this regularization is particularly well suited to average and cluster time series under the DTW geometry, a task for which our proposal significantly outperforms existing baselines. Next, we propose to tune the parameters of a machine that outputs time series by minimizing its fit with ground-truth labels in a soft-DTW sense.


RuleMatrix #REFR uses a matrix design and Sankey diagram visualization for the content of a rule list showing how data flows through the list.

Recent work has used surrogate models to approximate the behavior of complex ML models locally #OTHEREFR , globally #OTHEREFR , or on all scales #OTHEREFR ; all of these examples provide visual exploration of such surrogates as well.

Closer to our work, SuRE #OTHEREFR uses hierarchical rules to describe the decision space of a given ML model and visually explore its results by an interactive hierarchical visualization of the extracted rules.

However, when checking multiple intertwined rules in the form of if-else statements, participants evaluating SuRE almost always reasoned about less than two conditions at a time, a limitation we surpass with DEFORESTVIS due to the simple nature of one-level decision trees.

Another interesting finding is that their system's users analyzed thoroughly the effect of predictions based on individual features, thus matching well the main design concept of our proposed tool.

Di Castro and Bertini #OTHEREFR use a single surrogate decision tree to replicate a classification model's prediction and visualize it to propose simple yet effective explanations for the original model.

The problem with the above two VA tools is that they use a flat tabular layout #OTHEREFR which disregards the rules' hierarchical structure and the important feature-ordering information captured by the hierarchical structure of decision trees.

In our approach, this is not a problem since AdaBoost produces one-level decision trees (stumps), and we sort stumps for the same feature on the "importance" extracted directly from the AdaBoost algorithm.

DRIL #OTHEREFR presents a rule list for adjusting thresholds and examining relationships between rules and data.

Our VA tool focuses on both the summarized rules and the decision stumps that serve as an extra explanation of how the aggregation of information occurs.

StrategyAtlas #OTHEREFR , a hybrid approach, aims to explain individual data instances by aggregating multiple local surrogates #OTHEREFR .

Deforestvis: Behavior Analysis Of Machine Learning Models With Surrogate Decision Stumps

Abstract: Fig. 1: Components of DEFORESTVIS: (a.1) lollipop plot shows data-rounding effects in the fidelity score; (a.2) dot plot with lines that explains complexity increase as more decision stumps get added and (a.3) selective stump-based explanation; (b.1) segmented bar chart tells the predictive outcome and power of each segment based on automatically computed thresholds and (b.2) detailed stump-based explanation grid; (c.1) bar chart shows the impurity and weighted probability of each decision stump; (c.2) histogram shows the active rule's threshold and distribution of training instances; (d) projection aggregates the global behavior of instances; color shows the local behavior according to the currently selected decision stump; and (e) fragmented bar chart shows the per-feature contribution and influence level for each test case.

Rulematrix: Visualizing And Understanding Classifiers With Rules

Abstract: With the growing adoption of machine learning techniques, there is a surge of research interest towards making machine learning systems more transparent and interpretable. Various visualizations have been developed to help model developers understand, diagnose, and refine machine learning models. However, a large number of potential but neglected users are the domain experts with little knowledge of machine learning but are expected to work with machine learning systems. In this paper, we present an interactive visualization technique to help users with little expertise in machine learning to understand, explore and validate predictive models. By viewing the model as a black box, we extract a standardized rule-based knowledge representation from its input-output behavior. Then, we design RuleMatrix, a matrix-based visualization of rules to help users navigate and verify the rules and the black-box model. We evaluate the effectiveness of RuleMatrix via two use cases and a usability study.


Proof Of Theorem 4.7 And Summary Of Covers

It is an interesting problem to develop a faster algorithm for computing d(ρ * , π) than the one that can be derived from #REFR .

Finally, for certain applications it may be useful to efficiently compute our density statistic.

For example, Even-Zohar and Leng #OTHEREFR devised aÕ(|π|)-time algorithm for computing d(τ * , π) for a permutation π, where τ * is the Bergsma-Dassios statistic defined in the introduction.

This provides an efficient method for testing whether two [0, 1]-valued random variables X and Y with continuous joint cdf are independent.

In fact, the results of #OTHEREFR can be used to efficiently compute d(ρ, π) for a large class of expressions ρ ∈ A, which they refer to as corner trees.

As it turns out, ρ * is not a corner tree, and so the results of #OTHEREFR are insufficient for computing d(ρ * , π) in timeÕ(|π|); however, using other algorithms in #OTHEREFR , one can do much better than the trivial bound of O(n 4 ).

This means that the aforementioned question of Graham on permutation quasirandomness was technically answered, in a different setting, almost 60 years before it was asked (and about 40 years before the term "quasirandomness" entered the combinatorial vernacular).

Six Permutation Patterns Force Quasirandomness

Abstract: A sequence π 1 , π 2 ,. .. of permutations is said to be quasirandom if the induced density of every permutation σ in π n converges to 1/|σ|! as n → ∞. We prove that π 1 , π 2 ,. .. is quasirandom if and only if the density of each permutation σ in the set {123, 321, 2143, 3412, 2413, 3142} converges to 1/|σ|!. Previously, the smallest cardinality of a set with this property, called a quasirandom-forcing set, was known to be between four and eight. In fact, we show that there is a single linear expression of the densities of the six permutations in this set which forces quasirandomness and show that this is best possible in the sense that there is no shorter linear expression of permutation densities with positive coefficients with this property. In the language of theoretical statistics, this expression provides a new nonparametric independence test for bivariate continuous distributions related to Spearman's ρ.

Counting Small Permutation Patterns

Abstract: A sample of n generic points in the xy-plane defines a permutation that relates their ranks along the two axes. Every subset of k points similarly defines a pattern, which occurs in that permutation. The number of occurrences of small patterns in a large permutation arises in many areas, including nonparametric statistics. It is therefore desirable to count them more efficiently than the straightforward ~O(n^k) time algorithm. ::: This work proposes new algorithms for counting patterns. We show that all patterns of order 2 and 3, as well as eight patterns of order 4, can be counted in nearly linear time. To that end, we develop an algebraic framework that we call corner tree formulas. Our approach generalizes the existing methods and allows a systematic study of their scope. ::: Using the machinery of corner trees, we find twenty-three independent linear combinations of order-4 patterns, that can be computed in time ~O(n). We also describe an algorithm that counts another 4-pattern, and hence all 4-patterns, in time ~O(n^(3/2)). ::: As a practical application, we provide a nearly linear time computation of a statistic by Yanagimoto (1970), Bergsma and Dassios (2010). This statistic yields a natural and strongly consistent variant of Hoeffding's test for independence of X and Y, given a random sample as above. This improves upon the so far most efficient ~O(n^2) algorithm.


Flexible Parametric Distribution Estimation Using Modified Normalizing Flows

This is contrary to #REFR , however, in our testing, this constraint relaxation works well and provides no tangible difference in performance.

∆ l,j = v l,j+1 − v l,j k l,j+1 − k l,j , δ l,j = v l,j+1 − v l,j−1 k l,j+1 − k l,j−1 .

Due to our constraint on the minimal differences between consecutive knots and values we do not have to concern ourselves with specific edge cases outlined in #OTHEREFR where the differences in above equalities would be zero.

Additionally, we do not restrict the spline knots to a predetermined interval.

It does, however, eliminate the need for parameter tuning as the knot ranges are determined automatically during optimization.

Ensemble Weather Forecast Post-Processing With A Flexible Probabilistic Neural Network Approach

Abstract: Ensemble forecast post-processing is a necessary step in producing accurate probabilistic forecasts. Conventional post-processing methods operate by estimating the parameters of a parametric distribution, frequently on a perlocation or per-lead-time basis, which limits their expressive power. We propose a novel, neural network-based method, which produces forecasts for all locations and lead times, jointly. To relax the distributional assumption made by many post-processing methods, our approach incorporates normalizing flows as flexible parametric distribution estimators. This enables us to model varying forecast distributions in a mathematically exact way. We demonstrate the effectiveness of our method in the context of the EUPPBench benchmark, where we conduct temperature forecast post-processing for stations in a sub-region of western Europe. We show that our novel method exhibit state-of-the-art performance on the benchmark, outclassing our previous, well-performing entry. Additionally, by providing a detailed comparison of three variants of our novel post-processing method, we elucidate the reasons why our method outperforms per-lead-time-based approaches and approaches with distributional assumptions.

Neural Spline Flows

Abstract: A normalizing flow models a complex probability density as an invertible transformation of a simple base density. Flows based on either coupling or autoregressive transforms both offer exact density evaluation and sampling, but rely on the parameterization of an easily invertible elementwise transformation, whose choice determines the flexibility of these models. Building upon recent work, we propose a fully-differentiable module based on monotonic rational-quadratic splines, which enhances the flexibility of both coupling and autoregressive transforms while retaining analytic invertibility. We demonstrate that neural spline flows improve density estimation, variational inference, and generative modeling of images.


WikiHow #REFR ) is a diverse summarization dataset extracted from the online knowledge base WikiHow, which is written by human authors.

We use three benchmark datasets representing different styles and domains as shown in Table 3 .

CNN/DailyMail (Hermann et al., 2015) is the most popular single-document summarization dataset, comprising online news articles and human written highlights.

XSum #OTHEREFR is an abstractive style summarization dataset built upon new articles with a one-sentence summary written by professional authors.

BART #OTHEREFR is the abstractive baseline that we use to compare with our rewriterfuser model.

BART-Rewriter is a stronger rewriter baseline that follows BERT-Rewriter #OTHEREFR but is fine-tuned on BART, so that we could have a fair comparison between our model and the previous rewriter.

We train GEMINI using separated prefinetuning and fine-tuning steps.

We prefinetuning the randomly-initialized parameters for 8 epochs, and then jointly fine-tune both the new and pretrained parameters until early stopping on development dataset.

Gemini: Controlling The Sentence-Level Writing Style For Abstractive Text Summarization

Abstract: Human experts write summaries using different techniques, including rewriting a sentence in the document or fusing multiple sentences to generate a summary sentence. These techniques are flexible and thus difficult to be imitated by any single method. To address this issue, we propose an adaptive model, GEMINI, that integrates a rewriter and a fuser to mimic the sentence rewriting and fusion techniques, respectively. GEM-INI adaptively chooses to rewrite a specific document sentence or generate a summary sentence from scratch. Experiments demonstrate that our adaptive approach outperforms the pure abstractive and rewriting baselines on various benchmark datasets, especially when the dataset has a balanced distribution of styles. Interestingly, empirical results show that the human writing style of each summary sentence is consistently predictable given its context.

Wikihow: A Large Scale Text Summarization Dataset

Abstract: Sequence-to-sequence models have recently gained the state of the art performance in summarization. However, not too many large-scale high-quality datasets are available and almost all the available ones are mainly news articles with specific writing style. Moreover, abstractive human-style systems involving description of the content at a deeper level require data with higher levels of abstraction. In this paper, we present WikiHow, a dataset of more than 230,000 article and summary pairs extracted and constructed from an online knowledge base written by different human authors. The articles span a wide range of topics and therefore represent high diversity styles. We evaluate the performance of the existing methods on WikiHow to present its challenges and set some baselines to further improve it.


Note that Theorem 5 holds for any algorithm A R ∈ A(η t , T ), and extends established results in prior works for the SGD with batch size 1 #REFR .

w t+1 − w (i) t+1 ≤ w t − w (i) t 1 Jt=J (i) t + w t − w (i) t 1 Jt =J (i) t + 2 m Lη t 1 Jt =J (i) t = w t − w (i) t + 2 m Lη t 1 Jt =J (i) t .

Inequality #OTHEREFR and Lemma 2 give the on-average stability bound of the statement.

This and inequality #OTHEREFR give the generalization error bound for convex smooth and Lipschitz loss for the final iterate for any batch schedule (see Algorithm 1), completing the proof.

Specifically, the generalization error bound of Theorem 5 holds for general stochastic or deterministic gradient schemes (Definition 1) and any batch size choice m.

Next, we present a matching lower bound on the generalization error for the class of Lipschitz and smooth convex problems C L β , instance-wise relative to the parameters L and β.

Select Without Fear: Almost All Mini-Batch Schedules Generalize Optimally

Abstract: We establish matching upper and lower generalization error bounds for mini-batch Gradient Descent (GD) training with either deterministic or stochastic, data-independent, but otherwise arbitrary batch selection rules. We consider smooth Lipschitz-convex/nonconvex/stronglyconvex loss functions, and show that classical upper bounds for Stochastic GD (SGD) also hold verbatim for such arbitrary nonadaptive batch schedules, including all deterministic ones. Further, for convex and strongly-convex losses we prove matching lower bounds directly on the generalization error uniform over the aforementioned class of batch schedules, showing that all such batch schedules generalize optimally. Lastly, for smooth (non-Lipschitz) nonconvex losses, we show that full-batch (deterministic) GD is essentially optimal, among all possible batch schedules within the considered class, including all stochastic ones.

Train Faster, Generalize Better: Stability Of Stochastic Gradient Descent

Abstract: We show that parametric models trained by a stochastic gradient method (SGM) with few iterations have vanishing generalization error. We prove our results by arguing that SGM is algorithmically stable in the sense of Bousquet and Elisseeff. Our analysis only employs elementary tools from convex and continuous optimization. We derive stability bounds for both convex and non-convex optimization under standard Lipschitz and smoothness assumptions. ::: Applying our results to the convex case, we provide new insights for why multiple epochs of stochastic gradient methods generalize well in practice. In the non-convex case, we give a new interpretation of common practices in neural networks, and formally show that popular techniques for training large deep models are indeed stability-promoting. Our findings conceptually underscore the importance of reducing training time beyond its obvious benefit.


As the atmospheric scattering coefficient β increases, the results of Naive-NeRF #REFR are greatly affected, while our results are more robust to different haze concentrations. Table 3 .

Our model can still achieve comparable results under the premise that the training data is several orders of magnitude less than the comparison methods.

Comparison of different haze concentrations.

We added different concentrations of haze to the Lego scene according to ASM.

Specifically, we set A to 0.8, and the atmospheric scattering coefficient β increased from 0.04 to 0.36.

As shown in Figure 6 , in Lego scene with different levels of haze, the results of Naive-NeRF #OTHEREFR are greatly affected, which indicates that the haze affects the 3D reconstruction to a large extent, and our model has a large impact on the different degrees of Haze has strong robustness. Estimation of parameters in ASM. As shown in Table 2 , Figure 6 . Quantitative results at different haze concentrations in Lego scene.

Quantitative results of ablation analysis on photometric errors of Lego scene, which shows that the combination of all the designed components is the best.

in the four synthetic scenes, our model estimates the parameter values in ASM more accurately, which also shows the effectiveness of Dehazing-NeRF in modeling the image degradation process due to haze.

The reason for the biased estimation of the parameter values in this model may be that the data lacks ground truth of parameters.

Dehazing-Nerf: Neural Radiance Fields From Hazy Images

Abstract: Neural Radiance Field (NeRF) has received much attention in recent years due to the impressively high quality in 3D scene reconstruction and novel view synthesis. However, image degradation caused by the scattering of atmospheric light and object light by particles in the atmosphere can significantly decrease the reconstruction quality when shooting scenes in hazy conditions. To address this issue, we propose Dehazing-NeRF, a method that can recover clear NeRF from hazy image inputs. Our method simulates the physical imaging process of hazy images using an atmospheric scattering model, and jointly learns the atmospheric scattering model and a clean NeRF model for both image dehazing and novel view synthesis. Different from previous approaches, Dehazing-NeRF is an unsupervised method with only hazy images as the input, and also does not rely on hand-designed dehazing priors. By jointly combining the depth estimated from the NeRF 3D scene with the atmospheric scattering model, our proposed model breaks through the ill-posed problem of single-image dehazing while maintaining geometric consistency. Besides, to alleviate the degradation of image quality caused by information loss, soft margin consistency regularization, as well as atmospheric consistency and contrast discriminative loss, are addressed during the model training process. Extensive experiments demonstrate that our method outperforms the simple combination of single-image dehazing and NeRF on both image dehazing and novel view image synthesis.

Nerf: Representing Scenes As Neural Radiance Fields For View Synthesis

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.