Social Reasoning Facilitating Communication

The obverter mechanism was implemented in modern neural networks by Choi, Lazaridou, and de Freitas (2018) and #REFR , who also observed an increase in compositionality.

The obverter mechanism makes use of the same signal-meaning mapping for language generation and comprehension.

This guarantees that agents are internally consistent and use the language they learn in the receiver role when they assume the speaker role.

It can be interpreted as a rudimentary form of social reasoning: an agent's own cognitive capabilities serve as the model of another agent.

The symmetry of the roles, together with the selection of speaker-receiver pairs at random from a population of agents, effectively creates a transmission chain similar to the iterated learning setting #OTHEREFR .

This should constitute a pressure for a systematic communication protocol.

The more complex mechanism of social reasoning was explored in #OTHEREFR .

Their research belongs to the domain of artificial intelligence and multi-agent cooperation.

They introduced the concept of social influence as an auxiliary reward for agents that communicate to solve a sequential social dilemma.

Social influence was measured using counterfactual reasoning: an agent considered alternative signals it could have sent and assessed their likely effect on the actions of another agent.

In one variant of their simulations, actions of the other agent were predicted using a specialised internal model trained in a supervised manner.

Models Of Symbol Emergence In Communication: A Conceptual Review And A Guide For Avoiding Local Minima

Abstract: Computational simulations are a popular method for testing hypotheses about the emergence of communication. This kind of research is performed in a variety of traditions including language evolution, developmental psychology, cognitive science, machine learning, robotics, etc. The motivations for the models are different, but the operationalizations and methods used are often similar. We identify the assumptions and explanatory targets of several most representative models and summarise the known results. We claim that some of the assumptions-such as portraying meaning in terms of mapping, focusing on the descriptive function of communication, modelling signals with amodal tokens-may hinder the success of modelling. Relaxing these assumptions and foregrounding the interactions of embodied and situated agents allows one to systematise the multiplicity of pressures under which symbolic systems evolve. In line with this perspective, we sketch the road towards modelling the emergence of meaningful symbolic communication, where symbols are simultaneously grounded in action and perception and form an abstract system.

Emergence Of Communication In An Interactive World With Consistent Speakers

Abstract: Training agents to communicate with one another given task-based supervision only has attracted considerable attention recently, due to the growing interest in developing models for human-agent interaction. Prior work on the topic focused on simple environments, where training using policy gradient was feasible despite the non-stationarity of the agents during training. In this paper, we present a more challenging environment for testing the emergence of communication from raw pixels, where training using policy gradient fails. We propose a new model and training algorithm, that utilizes the structure of a learned representation space to produce more consistent speakers at the initial phases of training, which stabilizes learning. We empirically show that our algorithm substantially improves performance compared to policy gradient. We also propose a new alignment-based metric for measuring context-independence in emerged communication and find our method increases context-independence compared to policy gradient and other competitive baselines.


CURE-TSR #REFR is a traffic sign recognition dataset that includes real-world and simulated challenging conditions of 12 types and five severity levels.

In addition to the widely accepted OOD and adversarial detection setup, we consider corrupted inputs as another type of anomaly.

Deployed in the real world, neural networks are known to suffer from imperfect samples due to the data acquisition process and environmental factors, such as motion blur or weather conditions #OTHEREFR - #OTHEREFR .

We use image classification datasets designed to benchmark the robustness of neural networks under realistic challenging conditions.

CIFAR-10-C #OTHEREFR consists of 19 diverse corruption types in four categories, including noise, blur, weather, and digital, at five different severity levels that are applied to test images of CIFAR-10 dataset.

For each dataset, a ResNet model is trained on corruptionfree images, and the gradients are collected from pristine images in the test sets and their corrupted versions.

We utilize the Mahalanobis method #OTHEREFR for comparison because it showed the best performance among all the compared methods for OOD detection and adversarial detection.

In the experiments, we observed that the AUROC scores are highly saturated for both methods in many cases, particularly for CIFAR-10-C dataset, which calls for a more comprehensive comparison.

To better facilitate the performance comparison, we employ corrected repeated k-fold crossvalidated (CV) paired t-test #OTHEREFR as a measure of statistical significance.

Paired t-test is a statistical test for comparing two different learning schemes, A and B, based on a number of observations; in this case, predictive accuracy a and b.

Probing The Purview Of Neural Networks Via Gradient Analysis

Abstract: We analyze the data-dependent capacity of neural networks and assess anomalies in inputs from the perspective of networks during inference. The notion of data-dependent capacity allows for analyzing the knowledge base of a model populated by learned features from training data. We define purview as the additional capacity necessary to characterize inference samples that differ from the training data. To probe the purview of a network, we utilize gradients to measure the amount of change required for the model to characterize the given inputs more accurately. To eliminate the dependency on ground-truth labels in generating gradients, we introduce confounding labels that are formulated by combining multiple categorical labels. We demonstrate that our gradient-based approach can effectively differentiate inputs that cannot be accurately represented with learned features. We utilize our approach in applications of detecting anomalous inputs, including out-of-distribution, adversarial, and corrupted samples. Our approach requires no hyperparameter tuning or additional data processing and outperforms state-of-the-art methods by up to 2.7%, 19.8%, and 35.6% of AUROC scores, respectively.

Cure-Tsr: Challenging Unreal And Real Environments For Traffic Sign Recognition

Abstract: In this paper, we investigate the robustness of traffic sign recognition algorithms under challenging conditions. Existing datasets are limited in terms of their size and challenging condition coverage, which motivated us to generate the Challenging Unreal and Real Environments for Traffic Sign Recognition (CURE-TSR) dataset. It includes more than two million traffic sign images that are based on real-world and simulator data. We benchmark the performance of existing solutions in real-world scenarios and analyze the performance variation with respect to challenging conditions. We show that challenging conditions can decrease the performance of baseline methods significantly, especially if these challenging conditions result in loss or misplacement of spatial information. We also investigate the effect of data augmentation and show that utilization of simulator data along with real-world data enhance the average recognition performance in real-world scenarios. The dataset is publicly available at this https URL


Since then, the topic gathered significant attention, and the activity sparked with the emergence of polar codes #REFR ; Arıkan mentioned this as one of the major open problems in coding theory at ITW Dublin in 2010.

We refer to Section 9 for the formal definition of BMS channels; for now, it is sufficient to consider the BSC, the main case of interest, as BMS are mixtures of BSCs. Conjecture 1.

For any BMS channel P, and any rate R below the capacity of P given by C(P) = I(U, P) = (1/2) x∈{0,1},y∈Y P(y|x) log 2 P(y|x)/(P(y|0)/2 + P(y|1)/2), where 2 Y is the output alphabet of P (with C(P) = 1 − H( ) when P = BSC( )), a sequence of RM(m i , r i ) codes of rate R i = m i ≤r i 2 −m i tending to R can be decoded successfully with high probability, i.e., any of the 2 nR i codewords can be decoded with probability 1 − o m i (1) despite corruptions from P.

It is hard to track back the first appearance of this belief in the literature, but [KKM + 16a] reports that it was likely already present in the late 60s.

The claim was mentioned explicitly in a 1993 talk by Shu Lin, entitled ?RM Codes are Not So Bad? #OTHEREFR .

A 1993 paper by Dumer and Farrell also contains a discussion on the matter #OTHEREFR , as well as the 1997 paper of Costello and Forney on the 'road to channel capacity' #OTHEREFR .

Due to the broad relevance 3 of RM codes in computer science, electrical engineering and mathematics, the activity scattered in a wide line of works [Dum04, DS06, Dum06, CG05, HKL05, Ari08, Arı09, Ari10, KLP12, ASW15b, ASW15a, KKM + 16a, KKM + 17, MHU14, SSV17, AY19, YA20, SS20, KKM + 16b, Sam18, Sam20, AHN21, HSS21, LHP20, FFHM21, ASY21, RP21, GEE + 21, #OTHEREFR ; see also #OTHEREFR .

Relations to polar codes In a breakthrough paper, Arıkan showed that the explicit class of polar codes achieve Shannon capacity on any BMS channel #OTHEREFR .

Given the close relationship between polar and RM codes, the belief that RM codes could also be proved to achieve capacity on BMS channels intensified.

Polar codes are derived from the same square matrix, i.e., the matrix whose rows correspond to evaluations of monomials, which can also be expressed as G n := ( 1 1 0 1 ) ⊗m , but polar codes use a different row selection that is channel dependant.

One can view the difference as follows: apply G n (over F 2 ) to a vector with i.i.d.

A Proof That Reed-Muller Codes Achieve Shannon Capacity On Symmetric Channels

Abstract: Reed-Muller codes were introduced in 1954, with a simple explicit construction based on polynomial evaluations, and have long been conjectured to achieve Shannon capacity on symmetric channels. Major progress was made towards a proof over the last decades; using combinatorial weight enumerator bounds, a breakthrough on the erasure channel from sharp thresholds, hypercontractivity arguments, and polarization theory. Another major progress recently established that the bit error probability vanishes slowly below capacity. However, when channels allow for errors, the results of Bourgain-Kalai do not apply for converting a vanishing bit to a vanishing block error probability, neither do the known weight enumerator bounds. The conjecture that RM codes achieve Shannon capacity on symmetric channels, with high probability of recovering the codewords, has thus remained open. This paper closes the conjecture's proof. It uses a new recursive boosting framework, which aggregates the decoding of codeword restrictions on 'subspace-sunflowers', handling their dependencies via an L p Boolean Fourier analysis, and using a list-decoding argument with a weight enumerator bound from Sberlo-Shpilka. The proof does not require a vanishing bit error probability for the base case, but only a non-trivial probability, obtained here for general symmetric codes. This gives in particular a shortened and tightened argument for the vanishing bit error probability result of Reeves-Pfister, and with prior works, it implies the strong wire-tap secrecy of RM codes on pure-state classical-quantum channels.

Channel Polarization: A Method For Constructing Capacity-Achieving Codes For Symmetric Binary-Input Memoryless Channels

Abstract: A method is proposed, called channel polarization, to construct code sequences that achieve the symmetric capacity $I(W)$ of any given binary-input discrete memoryless channel (B-DMC) $W$. The symmetric capacity is the highest rate achievable subject to using the input letters of the channel with equal probability. Channel polarization refers to the fact that it is possible to synthesize, out of $N$ independent copies of a given B-DMC $W$, a second set of $N$ binary-input channels $\{W_N^{(i)}:1\le i\le N\}$ such that, as $N$ becomes large, the fraction of indices $i$ for which $I(W_N^{(i)})$ is near 1 approaches $I(W)$ and the fraction for which $I(W_N^{(i)})$ is near 0 approaches $1-I(W)$. The polarized channels $\{W_N^{(i)}\}$ are well-conditioned for channel coding: one need only send data at rate 1 through those with capacity near 1 and at rate 0 through the remaining. Codes constructed on the basis of this idea are called polar codes. The paper proves that, given any B-DMC $W$ with $I(W)>0$ and any target rate $R<I(W)$, there exists a sequence of polar codes $\{{\mathscr C}_n;n\ge 1\}$ such that ${\mathscr C}_n$ has block-length $N=2^n$, rate $\ge R$, and probability of block error under successive cancellation decoding bounded as $P_{e}(N,R) \le \bigoh(N^{-\frac14})$ independently of the code rate. This performance is achievable by encoders and decoders with complexity $O(N\log N)$ for each.


A pPDA is PAST iff the equation system from Theorem 7 has a non-negative solution. This is in contrast to e.g. the equation system from #REFR Ex.

Another important ingredient are weighted PDA over semirings #OTHEREFR .

Related work: Expected runtimes and, more generally, expected rewards in pPDA are studied in #OTHEREFR - #OTHEREFR , #OTHEREFR .

However, these works focus mostly on expectations conditioned on reaching the empty stack while being in a certain state.

In our paper we need such conditional expectations as well; but additionally, we also consider the unconditional expected runtime of pPDA, which is crucial for proving PAST.

More specifically, our Theorem 7 has -to the best of our knowledge -not appeared in previous works.

3.3] which has a solution iff the expected runtime conditioned on termination is finite.

However, this can be the case even if the termination probability is < 1.

Theorem 7 is actually more in the spirit of #OTHEREFR , #OTHEREFR Ch.

4] , where 1-exit recursive simple stochastic games with positive rewards are considered, a model that corresponds to a stateless variant of PDA controlled by two players and a probabilistic environment.

As we consider general pPDA with states, our result does not easily follow from theirs.

On Certificates, Expected Runtimes, And Termination In Probabilistic Pushdown Automata

Abstract: Probabilistic pushdown automata (pPDA) are a natural operational model for a variety of recursive discrete stochastic processes. In this paper, we study certificates-succinct and easily verifiable proofs-for upper and lower bounds on various quantitative properties of a given pPDA. We reveal an intimate, yet surprisingly simple connection between the existence of such certificates and the expected time to termination of the pPDA at hand. This is established by showing that certain intrinsic properties, like the spectral radius of the Jacobian of the pPDA's underlying polynomial equation system, are directly related to expected runtimes. As a consequence, we obtain that there always exist easy-to-check proofs for positive almost-sure termination: does a pPDA terminate in finite expected time?

Model Checking Probabilistic Pushdown Automata

Abstract: We consider the model checking problem for probabilistic ::: pushdown automata (pPDA) and properties expressible in various ::: probabilistic logics. We start with properties that can be ::: formulated as instances of a generalized random walk problem. ::: We prove that both qualitative and quantitative model checking ::: for this class of properties and pPDA is decidable. Then we ::: show that model checking for the qualitative fragment of the ::: logic PCTL and pPDA is also decidable. Moreover, we develop an ::: error-tolerant model checking algorithm for PCTL and the ::: subclass of stateless pPDA. Finally, we consider the class of ::: omega-regular properties and show that both qualitative and ::: quantitative model checking for pPDA is decidable.


Afterwards, another SVM-based model, i.e., SSVM #REFR , proposes to control the empirical risk to ensure a finite risk of the unknown classes.

#OTHEREFR in the domain of computer vision, where the task is defined to jointly achieve the known class classification and novel unknown class detection.

Existing image OSR methods can be grouped into two categories include traditional methods and deep neural networks (DNNs)-based methods.

1) Traditional Methods: The traditional methods are typically implemented by some classic machine learning models such as SVM, nearest-neighbors, sparse-representation, etc., or statistical models such as extreme value machine.

Among them, the 1-vs-set machine #OTHEREFR and W-SVM #OTHEREFR are the pioneer works and both of them are derived from SVM.

The former augments the decision plane of classic SVM from the marginal distances, and the latter integrates useful properties of statistical extreme value theory into SVM to calibrate the classification scores.

On the other hand, OSNN #OTHEREFR extends the nearest-neighbor classifier to the OSR scenario by fully incorporating the ability to recognize samples belonging to classes that are unknown during training.

A generalized sparse-representation based classification algorithm is then proposed to make use of the generalized Pareto extreme value distribution for OSR.

The extreme value theorybased models are a special category derived from the statistical modeling methods for statistical extremes #OTHEREFR .

Among them, EVM #OTHEREFR is the most representative one, which utilizes datadistribution information with a strong theoretical foundation.

2) DNNs-based Methods: With the popularity of deep learning techniques, the deep neural networks have became a powerful alternative to implement OSR systems.

Mdenet: Multi-Modal Dual-Embedding Networks For Malware Open-Set Recognition

Abstract: Malware open-set recognition (MOSR) aims at jointly classifying malware samples from known families and detect the ones from novel unknown families, respectively. Existing works mostly rely on a well-trained classifier considering the predicted probabilities of each known family with a thresholdbased detection to achieve the MOSR. However, our observation reveals that the feature distributions of malware samples are extremely similar to each other even between known and unknown families. Thus the obtained classifier may produce overly high probabilities of testing unknown samples toward known families and degrade the model performance. In this paper, we propose the Multi-modal Dual-Embedding Networks, dubbed MDENet, to take advantage of comprehensive malware features from different modalities to enhance the diversity of malware feature space, which is more representative and discriminative for downstream recognition. Concretely, we first generate a malware image for each observed sample based on their numeric features using our proposed numeric encoder with a redesigned multi-scale CNN structure, which can better explore their statistical and spatial correlations. Besides, we propose to organize tokenized malware features into a sentence for each sample considering its behaviors and dynamics, and utilize language models as the textual encoder to transform it into a representable and computable textual vector. Such parallel multi-modal encoders can fuse the above two components to enhance the feature diversity. Last, to further guarantee the open-set recognition, we dually embed the fused multi-modal representation into one primary space and an associated subspace, i.e., discriminative and exclusive spaces, with contrastive sampling and ρ-bounded enclosing sphere regularizations, which resort to classification and detection, respectively. Moreover, we also enrich our previously proposed large-scaled malware dataset MAL-100 with multi-modal characteristics and contribute an improved version dubbed MAL-100 +. Experimental results on the widely used malware dataset Mailing and the proposed MAL-100 + demonstrate the effectiveness of our method.

Specialized Support Vector Machines For Open-Set Recognition

Abstract: Often, when dealing with real-world recognition problems, we do not need, and often cannot have, knowledge of the entire set of possible classes that might appear during operational testing. In such cases, we need to think of robust classification methods able to deal with the"unknown"and properly reject samples belonging to classes never seen during training. Notwithstanding, almost all existing classifiers to date were mostly developed for the closed-set scenario, i.e., the classification setup in which it is assumed that all test samples belong to one of the classes with which the classifier was trained. In the open-set scenario, however, a test sample can belong to none of the known classes and the classifier must properly reject it by classifying it as unknown. In this work, we extend upon the well-known Support Vector Machines (SVM) classifier and introduce the Specialized Support Vector Machines (SSVM), which is suitable for recognition in open-set setups. SSVM balances the empirical risk and the risk of the unknown and ensures that the region of the feature space in which a test sample would be classified as known (one of the known classes) is always bounded, ensuring a finite risk of the unknown. In this work, we also highlight the properties of the SVM classifier related to the open-set scenario, and provide necessary and sufficient conditions for an RBF SVM to have bounded open-space risk.


B. Block Design Schemes: Locally Differentially Private Schemes Based On Block Designs

Moreover, the following theorem says that the above asymptotic optimality implies the non-asymptotic optimality under the unbiased constraint for the mean squared loss (ℓ #REFR 2 ).

Especially, if e ǫ = E(k, k + 1; v) for some k, then

K opt (v, ǫ) = {k, k + 1}. Otherwise, K opt (v, ǫ) is a singletone.

Theorem 3.7: For given n, let (Q,P n ) be a block design scheme satisfying k ∈ K opt (v, ǫ), and (Q ′ ,P ′ n ) be any unbiased ǫ-LDP scheme. Then, we have

Proof: For each positive integer m, letP ′ nm be the estimator on nm clients given bŷ

Block Design-Based Local Differential Privacy Mechanisms

Abstract: In this paper, we propose a new class of local differential privacy (LDP) schemes based on combinatorial block designs for a discrete distribution estimation. This class not only recovers many known LDP schemes in a unified framework of combinatorial block design, but also suggests a novel way of finding new schemes achieving the optimal (or near-optimal) privacy-utility trade-off with lower communication costs. Indeed, we find many new LDP schemes that achieve both the optimal privacy-utility trade-off and the minimum communication cost among all the unbiased schemes for a certain set of input data size and LDP constraint. Furthermore, to partially solve the sparse existence issue of block design schemes, we consider a broader class of LDP schemes based on regular and pairwise-balanced designs, called RPBD schemes, which relax one of the symmetry requirements on block designs. By considering this broader class of RPBD schemes, we can find LDP schemes achieving near-optimal privacy-utility trade-off with reasonably low communication costs for a much larger set of input data size and LDP constraint.

How To Break Anonymity Of The Netflix Prize Dataset

Abstract: We present a new class of statistical de-anonymization attacks against high-dimensional micro-data, such as individual preferences, recommendations, transaction records and so on. Our techniques are robust to perturbation in the data and tolerate some mistakes in the adversary's background knowledge. ::: We apply our de-anonymization methodology to the Netflix Prize dataset, which contains anonymous movie ratings of 500,000 subscribers of Netflix, the world's largest online movie rental service. We demonstrate that an adversary who knows only a little bit about an individual subscriber can easily identify this subscriber's record in the dataset. Using the Internet Movie Database as the source of background knowledge, we successfully identified the Netflix records of known users, uncovering their apparent political preferences and other potentially sensitive information.


Note that ensemble CAT with other acoustic model #REFR achieves performance comparable to previous work trained on additional modality, e.g.

It is worth mentioning that following the same ensemble protocol, the CAT achieves performance comparable to SOTA ensemble models.

By simply ensemble the CAT with PANN #OTHEREFR , we achieve performance comparable to the previous 9-model ensemble #OTHEREFR results.

This also shows the CAT provides complementary information to ConvNet based model.

ESC50 The CAT achieves SOTA performance on ESC50 dataset as well (Table I) .

We observe a similar pattern that the proposed CAT outperforms previous SOTA #OTHEREFR , #OTHEREFR by 0.8% and 0.2% with and without ensemble; demonstrating the effectiveness of the proposed design.

CAT is 0.7% better than AnT #OTHEREFR which utilizes visualacoustic information.

It can be anticipated that incorporating multimodality features into CAT may further improve performance.

Cat: Causal Audio Transformer For Audio Classification

Abstract: The attention-based Transformers have been increasingly applied to audio classification because of their global receptive field and ability to handle long-term dependency. However, the existing frameworks which are mainly extended from the Vision Transformers are not perfectly compatible with audio signals. In this paper, we introduce a Causal Audio Transformer (CAT) consisting of a Multi-Resolution Multi-Feature (MRMF) feature extraction with an acoustic attention block for more optimized audio modeling. In addition, we propose a causal module that alleviates over-fitting, helps with knowledge transfer and improves interpretability. CAT obtains higher or comparable state-of-the-art classification performance on ESC50, AudioSet and UrbanSound8K datasets, and can be easily generalized to other Transformerbased models.

Panns: Large-Scale Pretrained Audio Neural Networks For Audio Pattern Recognition

Abstract: Audio pattern recognition is an important research topic in the machine learning area, and includes several tasks such as audio tagging, acoustic scene classification and sound event detection. Recently neural networks have been applied to solve audio pattern recognition problems. However, previous systems focus on small datasets, which limits the performance of audio pattern recognition systems. Recently in computer vision and natural language processing, systems pretrained on large datasets have generalized well to several tasks. However, there is limited research on pretraining neural networks on large datasets for audio pattern recognition. In this paper, we propose large-scale pretrained audio neural networks (PANNs) trained on AudioSet. We propose to use Wavegram, a feature learned from waveform, and the mel spectrogram as input. We investigate the performance and complexity of a variety of convolutional neural networks. Our proposed AudioSet tagging system achieves a state-of-the-art mean average precision (mAP) of 0.439, outperforming the best previous system of 0.392. We transferred a PANN to six audio pattern recognition tasks and achieve state-of-the-art performance in many tasks. Source code and pretrained models have been released.


The Transformer was first proposed in #REFR for machine translation and has established state-of-the-art results in many natural language processing (NLP) tasks.

#OTHEREFR tried to generate image layouts and synthesize realistic images from input scene graphs via GCNs. Nauata et al.

#OTHEREFR proposed a graph-constrained generative adversarial network, whose generator and discriminator are built upon a relational architecture.

Our innovation is a novel graph Transformer GAN, where the input constraint is encoded into the graph structure of the proposed graph Transformer-based generator and node classificationbased discriminator.

Experimental results show the effectiveness of GTGAN over all the leading methods.

Recently, the Vision Transformer (ViT) #OTHEREFR equipped with global self-attention has achieved state-of-the-art results on the classification task.

Since then, Transformer-based approaches have been shown to be efficient in many computer vision tasks including image segmentation #OTHEREFR , object detection #OTHEREFR , depth estimation #OTHEREFR , pose estimation #OTHEREFR , video inpainting #OTHEREFR , vision-and-language navigation #OTHEREFR , video classification #OTHEREFR , human reaction generation #OTHEREFR , 3D pose transfer #OTHEREFR .

Different from these methods, in this paper, we adopt a Transformer-based network to tackle the graphconstrained house generation task.

However, integrating graph convolutional networks and ViTs is not trivial.

To this end, we propose a graph Transformer-based generator to capture both local and global relations across nodes in a graph.

Graph Transformer Gans For Graph-Constrained House Generation

Abstract: We present a novel graph Transformer generative adversarial network (GTGAN) to learn effective graph node relations in an end-to-end fashion for the challenging graphconstrained house generation task. The proposed graph-Transformer-based generator includes a novel graph Transformer encoder that combines graph convolutions and selfattentions in a Transformer to model both local and global interactions across connected and non-connected graph nodes. Specifically, the proposed connected node attention (CNA) and non-connected node attention (NNA) aim to capture the global relations across connected nodes and non-connected nodes in the input graph, respectively. The proposed graph modeling block (GMB) aims to exploit local vertex interactions based on a house layout topology. Moreover, we propose a new node classification-based discriminator to preserve the high-level semantic and discriminative node features for different house components. Finally, we propose a novel graph-based cycle-consistency loss that aims at maintaining the relative spatial relationships between ground truth and predicted graphs. Experiments on two challenging graph-constrained house generation tasks (i.e., house layout and roof generation) with two public datasets demonstrate the effectiveness of GTGAN in terms of objective quantitative scores and subjective visual realism. New state-of-the-art results are established by large margins on both tasks.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


It simulates the train-test process with few examples during training and optimizes the models to reduce the simulated test errors #REFR .

There are two common paradigms for creating such pretext tasks, i.e., self-supervised learning and intermediate-finetuning.

Specifically, in self-supervised methods, pseudo annotations are crafted by some heuristics on unannotated data to train models in a supervised learning manner #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR .

On the other hand, for intermediate finetuning methods, task-related annotated datasets are used to finetune language models before finetuning on the target task #OTHEREFR .

Although transfer learning methods are usually effective, they are prone to be affected by the data properties or learning objectives in the pretext tasks #OTHEREFR , #OTHEREFR .

Different from transfer learning, meta learning provides a framework that transfers task-level knowledge instead of data-level one.

In such schemes, the behaviors of learning with few examples are specifically considered.

The ability of "learning to learn" allows the model to adapt to unseen tasks quickly.

Nevertheless, two challenges arise in designing such a meta learning framework for abstractive summarization. 1) Expensive computation.

Meta learning involves the computation of second derivatives, which is expensive for summarization models due to the typically large model size. 2) Meta-dataset construction. Meta learning requires crafted tasks for learning.

In applications such as image classification #OTHEREFR , tasks can be directly defined as different class combinations in a single dataset, which is not suitable for abstractive summarization.

Spec: Summary Preference Decomposition For Low-Resource Abstractive Summarization

Abstract: Neural abstractive summarization has been widely studied and achieved great success with large-scale corpora. However, the considerable cost of annotating data motivates the need for learning strategies under low-resource settings. In this paper, we investigate the problems of learning summarizers with only few examples and propose corresponding methods for improvements. First, typical transfer learning methods are prone to be affected by data properties and learning objectives in the pretext tasks. Therefore, based on pretrained language models, we further present a meta learning framework to transfer few-shot learning processes from source corpora to the target corpus. Second, previous methods learn from training examples without decomposing the content and preference. The generated summaries could therefore be constrained by the preference bias in the training set, especially under low-resource settings. As such, we propose decomposing the contents and preferences during learning through the parameter modulation, which enables control over preferences during inference. Third, given a target application, specifying required preferences could be non-trivial because the preferences may be difficult to derive through observations. Therefore, we propose a novel decoding method to automatically estimate suitable preferences and generate corresponding summary candidates from the few training examples. Extensive experiments demonstrate that our methods achieve state-of-the-art performance on six diverse corpora with 30.11%/33.95%/27.51% and 26.74%/31.14%/24.48% average improvements on ROUGE-1/2/L under 10-and 100example settings.

Model-Agnostic Meta-Learning For Fast Adaptation Of Deep Networks

Abstract: We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learning problems, including classification, regression, and reinforcement learning. The goal of meta-learning is to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples. In our approach, the parameters of the model are explicitly trained such that a small number of gradient steps with a small amount of training data from a new task will produce good generalization performance on that task. In effect, our method trains the model to be easy to fine-tune. We demonstrate that this approach leads to state-of-the-art performance on two few-shot image classification benchmarks, produces good results on few-shot regression, and accelerates fine-tuning for policy gradient reinforcement learning with neural network policies.


In order to further refine the color information, we additionally add several 1×1 convolution blocks with channel attention mechanism #REFR . More details are provided in the supplementary material.

wherex 1 andx 2 is the output of BNN and LAN, respectively. sg(•) denotes stop gradient operation #OTHEREFR .

When training is done, LAN will have a transfer ability to predict the supervision for textured areas.

The network structure of LAN is simple yet delicate.

We stack 3×3 convolution layers to create the local receptive field, specifically, k 3×3 layers can make up (2k+1)×(2k+1) receptive field.

Spatially Adaptive Self-Supervised Learning For Real-World Image Denoising

Abstract: Significant progress has been made in self-supervised image denoising (SSID) in the recent few years. However, most methods focus on dealing with spatially independent noise, and they have little practicality on real-world sRGB images with spatially correlated noise. Although pixelshuffle downsampling has been suggested for breaking the noise correlation, it breaks the original information of images, which limits the denoising performance. In this paper, we propose a novel perspective to solve this problem, i.e., seeking for spatially adaptive supervision for real-world sRGB image denoising. Specifically, we take into account the respective characteristics of flat and textured regions in noisy images, and construct supervisions for them separately. For flat areas, the supervision can be safely derived from non-adjacent pixels, which are much far from the current pixel for excluding the influence of the noise-correlated ones. And we extend the blind-spot network to a blindneighborhood network (BNN) for providing supervision on flat areas. For textured regions, the supervision has to be closely related to the content of adjacent pixels. And we present a locally aware network (LAN) to meet the requirement, while LAN itself is selectively supervised with the output of BNN. Combining these two supervisions, a denoising network (e.g., U-Net) can be well-trained. Extensive experiments show that our method performs favorably against state-of-the-art SSID methods on real-world sRGB photographs. The code is available at https://github. com/nagejacob/SpatiallyAdaptiveSSID.

Image Super-Resolution Using Very Deep Residual Channel Attention Networks

Abstract: Convolutional neural network (CNN) depth is of crucial importance for image super-resolution (SR). However, we observe that deeper networks for image SR are more difficult to train. The low-resolution inputs and features contain abundant low-frequency information, which is treated equally across channels, hence hindering the representational ability of CNNs. To solve these problems, we propose the very deep residual channel attention networks (RCAN). Specifically, we propose a residual in residual (RIR) structure to form very deep network, which consists of several residual groups with long skip connections. Each residual group contains some residual blocks with short skip connections. Meanwhile, RIR allows abundant low-frequency information to be bypassed through multiple skip connections, making the main network focus on learning high-frequency information. Furthermore, we propose a channel attention mechanism to adaptively rescale channel-wise features by considering interdependencies among channels. Extensive experiments show that our RCAN achieves better accuracy and visual improvements against state-of-the-art methods.

several 1×1 convolution
channel attention mechanism


The work in #REFR tried to solve the problem by capturing with stopand-motion manner, which captured each noisy-clean pair with a static scene, and concatenated frames with temporal order to generate paired noisy-clean raw videos, building the CRVD dataset.

For image denoising, there are mainly two strategies to capture paired noisy-clean images.

One strategy is capturing noisy-clean image pairs using high and low ISO settings, and the other strategy is averaging a lot of noisy images to get a clean image.

The two strategies both require the objects to be static and the camera is static.

Otherwise, it will lead to motion blur or large displacements between noisy and clean images.

This makes it difficult to capture noisy-clean pairs for videos.

But #OTHEREFR mainly captured toys which were controlled by people, the motions are monotonous and unnatural, and there are only few scenes (11 scenes) in the dataset.

The work in #OTHEREFR moved cameras rather than moving the objects to create translation motions, and they only captured sRGB video pairs.

Neither moving toys nor moving cameras can simulate the various motions in real world.

Training with the two datasets limits the model's ability in dealing with real-world noisy videos.

Based on the above observations, we propose to recapture the existing high-resolution videos displayed on a 4K screen to construct our dataset.

Rvideformer: Efficient Raw Video Denoising Transformer With A Larger Benchmark Dataset

Abstract: In recent years, raw video denoising has garnered increased attention due to the consistency with the imaging process and well-studied noise modeling in the raw domain. Despite these advancements, two problems still hinder the denoising performance. Firstly, there is no large dataset with realistic motions for supervised raw video denoising, as capturing noisy and clean frames for real dynamic scenes is difficult. To address this, we propose recapturing existing high-resolution videos displayed on a 4K screen. Specifically, we recapture the screen content with high-low ISO settings to construct noisy-clean paired frames. Afterward, we introduce intensity, spatial, and color correction strategies to make the paired frames well-aligned. Then, the aligned frames are concatenated with temporal order to construct paired videos. In this way, we construct a video denoising dataset (named as ReCRVD) with 120 groups of noisy-clean videos, whose ISO values ranging from 1600 to 25600. Secondly, while non-local temporal-spatial attention is beneficial for denoising, it often leads to heavy computation costs. In this work, we propose an efficient raw video denoising transformer network (RViDeformer) that explores both short and long-distance correlations. Specifically, we introduce Low-Resolution-Window Self-Attention (LWSA), Global-Window Self-Attention (GWSA), and Neighbour-Window Self-Attention (NWSA) to build a multi-branch spatial self-attention for spatial reconstruction. Similarly, Global-Window Temporal Mutual Attention (GTMA) and Neighbour-Window Temporal Mutual Attention (NTMA) are proposed to build multi-branch temporal self-attention for temporal reconstruction. We employ reparameterization to reduce computation costs. Our network is trained in both supervised and unsupervised manners, achieving the best performance compared with state-of-the-art methods. Additionaly, the model trained with our proposed dataset (ReCRVD) outperforms the model trained with previous benchmark dataset (CRVD) when evaluated on the real-world outdoor noisy videos. Our code and dataset will be released after the acceptance of this work. Index Terms-Raw video denoising, ReCRVD dataset, Raw video denoising transformer (RViDeformer), unsupervised learning.

Supervised Raw Video Denoising With A Benchmark Dataset On Dynamic Scenes

Abstract: In recent years, the supervised learning strategy for real noisy image denoising has been emerging and has achieved promising results. In contrast, realistic noise removal for raw noisy videos is rarely studied due to the lack of noisy-clean pairs for dynamic scenes. Clean video frames for dynamic scenes cannot be captured with a long-exposure shutter or averaging multi-shots as was done for static images. In this paper, we solve this problem by creating motions for controllable objects, such as toys, and capturing each static moment for multiple times to generate clean video frames. In this way, we construct a dataset with 55 groups of noisy-clean videos with ISO values ranging from 1600 to 25600. To our knowledge, this is the first dynamic video dataset with noisy-clean pairs. Correspondingly, we propose a raw video denoising network (RViDeNet) by exploring the temporal, spatial, and channel correlations of video frames. Since the raw video has Bayer patterns, we pack it into four sub-sequences, i.e RGBG sequences, which are denoised by the proposed RViDeNet separately and finally fused into a clean video. In addition, our network not only outputs a raw denoising result, but also the sRGB result by going through an image signal processing (ISP) module, which enables users to generate the sRGB result with their favourite ISPs. Experimental results demonstrate that our method outperforms state-of-the-art video and raw image denoising algorithms on both indoor and outdoor videos.


We follow #REFR to drop the matting refinement stage in DIM for simplicity and fair comparison. The results are illustrated in Tab. 7.

In contrast, thanks to the large receptive field and tri-token guided design, our proposed model learns image context and object structure effectively, contributing to sophisticated performance in such a challenging region.

Besides, our proposed model can also accurately estimate the TT-type objects whose background is complex. For example, in the second row of Fig.

3 , the background is so complicated that the previous methods are hard to describe the details.

It is worth noting that our TransMattingV2 achieves more high-quality alpha mattes than TransMattingV1, indicating that our tri-token helps to distinguish the foreground and background in deep features.

To evaluate the universality of our proposed tri-token, we introduce it to three current SOTA methods: DIM #OTHEREFR , GCAMatting #OTHEREFR , and MGMatting #OTHEREFR .

As we can see, our proposed tri-token boosts the performance of all three SOTA methods with a big margin.

Especially for recent proposed GCAMatting and MGMAtting, by replacing the traditional concatenation with our proposed tri-token, the SAD is decreased by 3.98 and 3.18, respectively.

This proves the effectiveness of the proposed tri-token strategy in existing SOTA methods and indicates a great potential to replace the concatenation strategy with the proposed tritoken in a wide range of image matting methods.

1, to further verify our method in boosting the performance of TT-type objects, we analyze two types of objects separately.

Transmatting: Tri-Token Equipped Transformer Model For Image Matting

Abstract: Image matting aims to predict alpha values of elaborate uncertainty areas of natural images, like hairs, smoke, and spider web. However, existing methods perform poorly when faced with highly transparent foreground objects due to the large area of uncertainty to predict and the small receptive field of convolutional networks. To address this issue, we propose a Transformer-based network (TransMatting) to model transparent objects with long-range features and collect a high-resolution matting dataset of transparent objects (Transparent-460) for performance evaluation. Specifically, to utilize semantic information in the trimap flexibly and effectively, we also redesign the trimap as three learnable tokens, named tri-token. Both Transformer and convolution matting models could benefit from our proposed tri-token design. By replacing the traditional trimap concatenation strategy with our tri-token, existing matting methods could achieve about 10% improvement in SAD and 20% in MSE. Equipped with the new tri-token design, our proposed TransMatting outperforms current state-of-the-art methods on several popular matting benchmarks and our newly collected Transparent-460.

Indices Matter: Learning To Index For Deep Image Matting

Abstract: We show that existing upsampling operators can be unified using the notion of the index function. This notion is inspired by an observation in the decoding process of deep image matting where indices-guided unpooling can often recover boundary details considerably better than other upsampling operators such as bilinear interpolation. By viewing the indices as a function of the feature map, we introduce the concept of 'learning to index', and present a novel index-guided encoder-decoder framework where indices are self-learned adaptively from data and are used to guide the pooling and upsampling operators, without extra training supervision. At the core of this framework is a flexible network module, termed IndexNet, which dynamically generates indices conditioned on the feature map. Due to its flexibility, IndexNet can be used as a plug-in applying to almost all off-the-shelf convolutional networks that have coupled downsampling and upsampling stages. We demonstrate the effectiveness of IndexNet on the task of natural image matting where the quality of learned indices can be visually observed from predicted alpha mattes. Results on the Composition-1k matting dataset show that our model built on MobileNetv2 exhibits at least 16.1% improvement over the seminal VGG-16 based deep matting baseline, with less training data and lower model capacity. Code and models have been made available at: https://tinyurl.com/IndexNetV1.


A neighbor-based tracking control scheme with distributed estimators is applied to advise distributed tracking strategies in #REFR .

They provide an alternative solution based on relaxing the inter-dependency requirements among the agents, such as the communication range of each agent, the amount of information required about its surrounding agents and the environment, etc.

A key aspect about any distributed controller is its ability to optimize a set of simultaneously conflicting local and team objectives.

The underlying control methods are often based on theories of feedback mechanisms and communication graphs #OTHEREFR , #OTHEREFR .

The behavior of large numbers of interacting agents, such as flocks of flying birds and fish schools, is used in the study the cooperative dynamical systems #OTHEREFR , #OTHEREFR .

Sliding mode methods are employed to achieve coordination of multi-agent systems in #OTHEREFR , #OTHEREFR .

Relevant control approaches relied on the existence of a virtual leader or a moving target in mobile sensor networks #OTHEREFR , #OTHEREFR .

A common shortcoming to all these techniques is their dependence on the prior knowledge of a system model that is free of unstructured uncertainties.

A distributed adaptive strategy that is based on a backstepping scheme is adopted to solve a consensus control problem in #OTHEREFR .

The tracking problem of fractional-order multi-agent systems is addressed in #OTHEREFR using adaptive strategies with neural network approximators.

Reinforcement Learning (RL) is a machine learning tool that allows agents to learn the best strategies to solve certain problems upon interacting with unknown environments #OTHEREFR , #OTHEREFR .

An Adaptive Fuzzy Reinforcement Learning Cooperative Approach For The Autonomous Control Of Flock Systems

Abstract: The flock-guidance problem enjoys a challenging structure where multiple optimization objectives are solved simultaneously. This usually necessitates different control approaches to tackle various objectives, such as guidance, collision avoidance, and cohesion. The guidance schemes, in particular, have long suffered from complex tracking-error dynamics. Furthermore, techniques that are based on linear feedback strategies obtained at equilibrium conditions either may not hold or degrade when applied to uncertain dynamic environments. Pre-tuned fuzzy inference architectures lack robustness under such unmodeled conditions. This work introduces an adaptive distributed technique for the autonomous control of flock systems. Its relatively flexible structure is based on online fuzzy reinforcement learning schemes which simultaneously target a number of objectives; namely, following a leader, avoiding collision, and reaching a flock velocity consensus. In addition to its resilience in the face of dynamic disturbances, the algorithm does not require more than the agent position as a feedback signal. The effectiveness of the proposed method is validated with two simulation scenarios and benchmarked against a similar technique from the literature.

Distributed Tracking Control Of Leader-Follower Multi-Agent Systems Under Noisy Measurement

Abstract: In this paper, a distributed tracking control scheme with distributed estimators has been developed for a leader-follower multi-agent system with measurement noises and directed interconnection topology. It is supposed that each follower can only measure relative positions of its neighbors in a noisy environment, including the relative position of the second-order active leader. A neighbor-based tracking protocol together with distributed estimators is designed based on a novel velocity decomposition technique. It is shown that the closed loop tracking control system is stochastically stable in mean square and the estimation errors converge to zero in mean square as well. A simulation example is finally given to illustrate the performance of the proposed control scheme.

distributed tracking strategies
neighbor-based tracking control


In implementation, CRA not only learns discriminative visual and textual features based on a set of orthonormal basis and a dual-gate mechanism, but also globally aligns both type of features under the supervision of a triplet contrastive loss #REFR .

To this end, although there has emerged progresses for improving either global CMA #OTHEREFR or local CMA #OTHEREFR , how to exploit multi-level CMA to enhance radiology report generation is underexplored.

Considering the above issues, we propose a Unify, Align and Refine (UAR) framework to facilitate multi-level CMA for generating faithful and believable radiology reports.

Within UAR, we design a Latent Space Unifier (LSU) to tokenize images into discrete visual tokens with a discrete variational autoencoder (dVAE) #OTHEREFR .

By doing so, LSU unifies vision and text modalities into discrete tokens, making it flexible to design a shared network that seamlessly processes both modalities to learn common knowledge #OTHEREFR .

Next, we introduce a modality-agnostic Cross-modal Representation Aligner (CRA) to learn global CMA.

What's more, we improve local CMA by integrating Transformer #OTHEREFR with our proposed Text-to-Image Refiner (TIR).

Different from the vanilla attention mechanism in Transformer, TIR additionally contains a learnable mask to re-calibrate text-to-image attention activations.

TIR constrains the learnable mask with an auxiliary loss to focus word prediction on useful visual information.

Last but not least, we design a two-stage training procedure to make the full model grasp CMA at different levels gradually.

We conduct experiments on two widely-adopted radiology report generation benchmarks, i.e., IU-Xray #OTHEREFR and MIMIC-CXR #OTHEREFR .

Unify, Align And Refine: Multi-Level Semantic Alignment For Radiology Report Generation

Abstract: Automatic radiology report generation has attracted enormous research interest due to its practical value in reducing the workload of radiologists. However, simultaneously establishing global correspondences between the image (e.g., Chest X-ray) and its related report and local alignments between image patches and keywords remains challenging. To this end, we propose an Unify, Align and then Refine (UAR) approach to learn multi-level cross-modal alignments and introduce three novel modules: Latent Space Unifier (LSU), Cross-modal Representation Aligner (CRA) and Textto-Image Refiner (TIR). Specifically, LSU unifies multimodal data into discrete tokens, making it flexible to learn common knowledge among modalities with a shared network. The modality-agnostic CRA learns discriminative features via a set of orthonormal basis and a dual-gate mechanism first and then globally aligns visual and textual representations under a triplet contrastive loss. TIR boosts token-level local alignment via calibrating text-to-image attention with a learnable mask. Additionally, we design a two-stage training procedure to make UAR gradually grasp cross-modal alignments at different levels, which imitates radiologists' workflow: writing sentence by sentence first and then checking word by word. Extensive experiments and analyses on IU-Xray and MIMIC-CXR benchmark datasets demonstrate the superiority of our UAR against varied state-of-the-art methods.

Facenet: A Unified Embedding For Face Recognition And Clustering

Abstract: Despite significant recent advances in the field of face recognition [10, 14, 15, 17], implementing face verification and recognition efficiently at scale presents serious challenges to current approaches. In this paper we present a system, called FaceNet, that directly learns a mapping from face images to a compact Euclidean space where distances directly correspond to a measure of face similarity. Once this space has been produced, tasks such as face recognition, verification and clustering can be easily implemented using standard techniques with FaceNet embeddings as feature vectors.

triplet contrastive loss
discriminative visual


Their parameters are written as ((n, K, d)) q , where K is the dimension and d the distance. The most common QECCs are stabilizer codes #REFR .

Quantum error-correcting codes (QECCs) are an important tool in quantum computation since they allow us to protect quantum information from quantum noise and, in particular, from decoherence.

The existence of quantum processors that improve the behaviour of classical ones #OTHEREFR explains the importance of obtaining good QECCs.

A q-ary QECC of length n, q being a power of a prime p, is a subspace of the Hilbert space C q n , where C denotes the complex numbers.

Many of these codes satisfy K = q k , for some nonnegative integer k, and then their parameters are expressed as [[n, k, d ]] q ; abusing the notation we say that k is the dimension of these codes.

QECCs were first introduced in the binary case #OTHEREFR and later in the general case #OTHEREFR , where we have cited only some references of a vast literature on the subject.

QECCs in the nonbinary case are convenient for fault-tolerant quantum computation #OTHEREFR .

Stabilizer quantum codes are intimately related to self-orthogonal additive codes, where duality is given by a trace-symplectic form #OTHEREFR .

As particular cases of the above construction, stabilizer quantum codes can be obtained from self-orthogonal linear codes with respect to Hermitian or Euclidean inner products.

Steane Enlargement Of Entanglement-Assisted Quantum Error-Correcting Codes

Abstract: We introduce a Steane-like enlargement procedure for entanglement-assisted quantum error-correcting codes (EAQECCs) obtained by considering Euclidean inner product. We give formulae for the parameters of these enlarged codes and apply our results to explicitly compute the parameters of enlarged EAQECCs coming from some BCH codes.

Nonbinary Stabilizer Codes Over Finite Fields

Abstract: One formidable difficulty in quantum communication and computation is to protect information-carrying quantum states against undesired interactions with the environment. To address this difficulty, many good quantum error-correcting codes have been derived as binary stabilizer codes. Fault-tolerant quantum computation prompted the study of nonbinary quantum codes, but the theory of such codes is not as advanced as that of binary quantum codes. This paper describes the basic theory of stabilizer codes over finite fields. The relation between stabilizer codes and general quantum codes is clarified by introducing a Galois theory for these objects. A characterization of nonbinary stabilizer codes over Fq in terms of classical codes over Fq 2 is provided that generalizes the well-known notion of additive codes over F4 of the binary case. This paper also derives lower and upper bounds on the minimum distance of stabilizer codes, gives several code constructions, and derives numerous families of stabilizer codes, including quantum Hamming codes, quadratic residue codes, quantum Melas codes, quantum Bose-Chaudhuri-Hocquenghem (BCH) codes, and quantum character codes. The puncturing theory by Rains is generalized to additive codes that are not necessarily pure. Bounds on the maximal length of maximum distance separable stabilizer codes are given. A discussion of open problems concludes this paper


One tentative workaround to quantitatively proxying for imitation is proposed by the framework of generative adversarial optimization, recently made popular in the context of gradient optimization of neural networks #REFR .

In essence, instead of manually quantifying performance, this framework introduces a second agent optimized to discriminate between real collectable data and artificially produced data.

Oppositely, the original data-producing agent is now optimized to generate data that the new agent cannot easily discern.

When optimization conditions are right, generating agents eventually produce synthesized data so similar to real data that discriminator agents can no longer tell the difference. Figure 10 . Metric-Based and Adversarial Optimization for Data Imitation.

(left) A similarity metric is manually defined from some target dataset for an agent to be optimized based on.

(right) Instead of a metric, a second agent is put in place and optimized to discriminate between generated and target data.

Evolving Artificial Neural Networks To Imitate Human Behaviour In Shinobi Iii : Return Of The Ninja Master Présenté Par

Abstract: Our society is increasingly fond of computational tools. This phenomenon has greatly increased over the past decade following, among other factors, the emergence of a new Artificial Intelligence paradigm. Specifically, the coupling of two algorithmic techniques, Deep Neural Networks and Stochastic Gradient Descent, thrusted by an exponentially increasing computing capacity, has and is continuing to become a major asset in many modern technologies. However, as progress takes its course, some still wonder whether other methods could similarly or even more greatly benefit from these various hardware advances. In order to further this study, we delve in this thesis into Evolutionary Algorithms and their application to Dynamic Neural Networks, two techniques which despite enjoying many advantageous properties have yet to find their niche in contemporary Artificial Intelligence. We find that by elaborating new methods while exploiting strong computational resources, it becomes possible to develop strongly performing agents on a variety of benchmarks but also some other agents behaving very similarly to human subjects on the video game Shinobi III : Return of The Ninja Master, typical complex tasks previously out of reach for non-gradient-based optimization.

Generative Adversarial Nets

Abstract: We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to ½ everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.

neural networks
generative adversarial optimization


The proposed efficient knowledge distillation scheme can achieve the same or even better accuracy compared with that of training with vanilla knowledge distillation #REFR yet at a similar training speed as the standard training, as verified in Sec. 4.3.

where S sample and S batch denote the uncertainty score of one sample and a batch of samples, s is the batch size, Yi is the output probability of the model for class i,Ŷi is the prior probability of the labels for class i, and n is the number of classes.

To simplify the measurement, all our experiments are based on the assumption that different classes have the same prior probability, i.e.,Ŷi =Ŷj for any i and j. As shown in Fig.

2 , if S batch is larger than the pre-defined threshold, then a query to the teacher model will be made; otherwise, this iteration will proceed with the standard training without knowledge distillation.

Ersam: Neural Architecture Search For Energy-Efficient And Real-Time Social Ambiance Measurement

Abstract: Social ambiance describes the context in which social interactions happen, and can be measured using speech audio by counting the number of concurrent speakers. This measurement has enabled various mental health tracking and human-centric IoT applications. While on-device Socal Ambiance Measure (SAM) is highly desirable to ensure user privacy and thus facilitate wide adoption of the aforementioned applications, the required computational complexity of state-of-the-art deep neural networks (DNNs) powered SAM solutions stands at odds with the often constrained resources on mobile devices. Furthermore, only limited labeled data is available or practical when it comes to SAM under clinical settings due to various privacy constraints and the required human effort, further challenging the achievable accuracy of on-device SAM solutions. To this end, we propose a dedicated neural architecture search framework for Energy-efficient and Real-time SAM (ERSAM). Specifically, our ERSAM framework can automatically search for DNNs that push forward the achievable accuracy vs. hardware efficiency frontier of mobile SAM solutions. For example, ERSAM-delivered DNNs only consume 40 mW • 12 h energy and 0.05 seconds processing latency for a 5 seconds audio segment on a Pixel 3 phone, while only achieving an error rate of 14.3% on a social ambiance dataset generated by LibriSpeech. We can expect that our ERSAM framework can pave the way for ubiquitous on-device SAM solutions which are in growing demand.

Distilling The Knowledge In A Neural Network

Abstract: A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.


The authors of #REFR also observed that as opposed to the size variations of normal traffic, TCP SYN flood packets are quite small (typically under 100 bytes), largely in order to maximize the number of connection requests per second for victim exhaustion.

1) INB and OUT: Inbound and Outbound Network Traffic: Among the direct/detectable features, inbound network traffic features describe the traffic whose destination is the IoT device in terms of packet count and size, encryption, etc.

The outbound network traffic features are similar to the inbound ones, however in this case, the IoT device is the source of the traffic for the features rather than the destination.

In past research, e.g., #OTHEREFR , #OTHEREFR , #OTHEREFR , #OTHEREFR , these features were found informative for IoT attack detection.

For instance, most normal IoT packets are sent at regular time intervals #OTHEREFR (e.g., automated network activities).

In contrast, these intervals are dramatically shorter #OTHEREFR for most DoS attack traffic and thus informative.

2) SRD: Communication Sources & Destinations: As opposed to PCs and smartphones, IoT devices normally communicate with a limited number of endpoints (IP destination) #OTHEREFR , e.g., for (de)activation from the cloud, retrieving firmware updates, and logging their status; thus, an increased number of destination endpoints might be indicative of attack traffic (e.g., sending messages to the botnet victims), while extraneous source IP addresses might indicate logging attempts into the device, as in Mirai #OTHEREFR .

Moreover, the set of destination IPs rarely changes over time #OTHEREFR .

Also, the traffic associated with DDoS attacks has different protocol distributions (roughly equivalent to the number of destination ports), and they also include fewer total protocols #OTHEREFR , thus highly likely to be informative for IoT attack detection.

D-Score: An Expert-Based Method For Assessing The Detectability Of Iot-Related Cyber-Attacks

Abstract: IoT devices are known to be vulnerable to various cyber-attacks, such as data exfiltration and the execution of flooding attacks as part of a DDoS attack. When it comes to detecting such attacks using network traffic analysis, it has been shown that some attack scenarios are not always equally easy to detect if they involve different IoT models. That is, when targeted at some IoT models, a given attack can be detected rather accurately, while when targeted at others the same attack may result in too many false alarms. In this research, we attempt to explain this variability of IoT attack detectability and devise a risk assessment method capable of addressing a key question: how easy is it for an anomaly-based network intrusion detection system to detect a given cyber-attack involving a specific IoT model? In the process of addressing this question we (a) investigate the predictability of IoT network traffic, (b) present a novel taxonomy for IoT attack detection which also encapsulates traffic predictability aspects, (c) propose an expertbased attack detectability estimation method which uses this taxonomy to derive a detectability score (termed 'D-Score') for a given combination of IoT model and attack scenario, and (d) empirically evaluate our method while comparing it with a datadriven method.

Machine Learning Ddos Detection For Consumer Internet Of Things Devices

Abstract: An increasing number of Internet of Things (IoT) devices are connecting to the Internet, yet many of these devices are fundamentally insecure, exposing the Internet to a variety of attacks. Botnets such as Mirai have used insecure consumer IoT devices to conduct distributed denial of service (DDoS) attacks on critical Internet infrastructure. This motivates the development of new techniques to automatically detect consumer IoT attack traffic. In this paper, we demonstrate that using IoT-specific network behaviors (e.g., limited number of endpoints and regular time intervals between packets) to inform feature selection can result in high accuracy DDoS detection in IoT network traffic with a variety of machine learning algorithms, including neural networks. These results indicate that home gateway routers or other network middleboxes could automatically detect local IoT device sources of DDoS attacks using low-cost machine learning algorithms and traffic data that is flow-based and protocol-agnostic.


For both experiments, we use 3D models from the PartNet-Mobility #REFR object set to generate a training and test data set.

Categories: While PartNet-Mobility provides more than 2000 objects from 46 categories, as done in previous work [5, Table 1 . Overview of Selected Objects.

We select a subset of the PartNet-Mobility #OTHEREFR object set and report the amount of instances we selected per category for our training and test set.

Our final object set has 92 objects instances for training and 25 for testing. objects. In Tab.

1 we list the number of objects per category we selected as well as the context in which they can be used.

Carto: Category And Joint Agnostic Reconstruction Of Articulated Objects

Abstract: Figure 1. Visualization of CARTO on unseen object instances. We first use CARTO to jointly detect all objects in the scene and then articulate them while keeping the predicted shape code constant.

Sapien: A Simulated Part-Based Interactive Environment

Abstract: Building home assistant robots has long been a pursuit for vision and robotics researchers. To achieve this task, a simulated environment with physically realistic simulation, sufficient articulated objects, and transferability to the real robot is indispensable. Existing environments achieve these requirements for robotics simulation with different levels of simplification and focus. We take one step further in constructing an environment that supports household tasks for training robot learning algorithm. Our work, SAPIEN, is a realistic and physics-rich simulated environment that hosts a large-scale set for articulated objects. Our SAPIEN enables various robotic vision and interaction tasks that require detailed part-level understanding.We evaluate state-of-the-art vision algorithms for part detection and motion attribute recognition as well as demonstrate robotic interaction tasks using heuristic approaches and reinforcement learning algorithms. We hope that our SAPIEN can open a lot of research directions yet to be explored, including learning cognition through interaction, part motion discovery, and construction of robotics-ready simulated game environment.


In the single scale setting, compared with the second best CNN-based method BDCN #REFR , we obtain a large performance gain by 2.3%, 2.1% and 4.5% in terms of ODS, OIS and AP.

In this section, we compare the performance of the proposed UAED with existing excellent edge detectors, including traditional detectors such as Canny #OTHEREFR , CNN-based detectors such as HED #OTHEREFR and RCF #OTHEREFR , and transformerbased detector EDTER #OTHEREFR .

BSDS results. The results are summarized in Table 1 .

We can see that our proposed UAED outperforms other previous CNN-based methods.

We also achieve ODS=0.844, OIS=0.864 and AP=0.905 under the MS-VOC setting, which also surpasses BDCN #OTHEREFR by a large margin (1.6%, 2.0% and 1.5%).

Compared with transformer-based EDTER #OTHEREFR , we still achieve the best performance and increase the scores by 0.5%, 0.6%, and 1.2% in ODS, OIS and AP metrics when testing on a single scale input.

The results are only slightly lower than EDTER under multi-scale settings.

The possible reason is that EDTER is learned in a patch-based manner, so the results are greatly improved when testing under the multi-scale setting.

To show the results more intuitively, we give the Precision-Recall curve in Figure 3 . The precision-recall curves on BSDS500. Fig. 3 .

The Treasure Beneath Multiple Annotations: An Uncertainty-Aware Edge Detector

Abstract: Deep learning-based edge detectors heavily rely on pixel-wise labels which are often provided by multiple annotators. Existing methods fuse multiple annotations using a simple voting process, ignoring the inherent ambiguity of edges and labeling bias of annotators. In this paper, we propose a novel uncertainty-aware edge detector (UAED), which employs uncertainty to investigate the subjectivity and ambiguity of diverse annotations. Specifically, we first convert the deterministic label space into a learnable Gaussian distribution, whose variance measures the degree of ambiguity among different annotations. Then we regard the learned variance as the estimated uncertainty of the predicted edge maps, and pixels with higher uncertainty are likely to be hard samples for edge detection. Therefore we design an adaptive weighting loss to emphasize the learning from those pixels with high uncertainty, which helps the network to gradually concentrate on the important pixels. UAED can be combined with various encoder-decoder backbones, and the extensive experiments demonstrate that UAED achieves superior performance consistently across multiple edge detection benchmarks. The source code is available at https://github.com/ZhouCX117/ UAED.

Bi-Directional Cascade Network For Perceptual Edge Detection

Abstract: Exploiting multi-scale representations is critical to improve edge detection for objects at different scales. To extract edges at dramatically different scales, we propose a Bi-Directional Cascade Network (BDCN) structure, where an individual layer is supervised by labeled edges at its specific scale, rather than directly applying the same supervision to all CNN outputs. Furthermore, to enrich multi-scale representations learned by BDCN, we introduce a Scale Enhancement Module (SEM) which utilizes dilated convolution to generate multi-scale features, instead of using deeper CNNs or explicitly fusing multi-scale edge maps. These new approaches encourage the learning of multi-scale representations in different layers and detect edges that are well delineated by their scales. Learning scale dedicated layers also results in compact network with a fraction of parameters. We evaluate our method on three datasets, i.e., BSDS500, NYUDv2, and Multicue, and achieve ODS Fmeasure of 0.828, 1.3% higher than current state-of-the art on BSDS500.


The session length search part is a Neural Architecture Search (NAS) algorithm based on DARTS #REFR .

Now we introduce the shortterm interest module for capturing the short-term sequential dependencies in users' historical sequences.

The short-term interest module follows a similar framework as the long-term interest module, meaning it also contains a sequence-mixer and channel-mixer.

The key difference is the short-term interest length search section that can automatically select optimal short-term interest length in a data-driven manner. Session Length Search.

To be specific, we first define some candidate short-term lengths before the last item , representing the possible range of users' short-term interest length, i.e., = { 1 , ..., , ..., }, as shown in Figure 5 . Then we create SRSMLPs, each for individual candidate .

Next, as shown in Figure 5 , we assign a set of learnable architectural weights = { 1 , ..., , ..., } to the outputs from different SRSMLPs respectively.

After that, we apply softmax to transform weights as continuous and differentiable approximations:

… Short-term Length Search = 2 … … Α 0.1 0.2 T-2 T-1 T T-2 T-1 T T … T-2 T-1 SRSMLP Layers SRSMLP Layers SRSMLP Layers SRSMLP Layers 0.2 0.5

Automlp: Automated Mlp For Sequential Recommendations

Abstract: Sequential recommender systems aim to predict users' next interested item given their historical interactions. However, a longstanding issue is how to distinguish between users' long/short-term interests, which may be heterogeneous and contribute differently to the next recommendation. Existing approaches usually set predefined short-term interest length by exhaustive search or empirical experience, which is either highly inefficient or yields subpar results. The recent advanced transformer-based models can achieve state-of-the-art performances despite the aforementioned issue, but they have a quadratic computational complexity to the length of the input sequence. To this end, this paper proposes a novel sequential recommender system, AutoMLP, aiming for better modeling users' long/short-term interests from their historical interactions. In addition, we design an automated and adaptive search algorithm for preferable short-term interest length via end-to-end optimization. Through extensive experiments, we show that AutoMLP has competitive performance against state-of-the-art methods, while maintaining linear computational complexity. CCS CONCEPTS • Information systems → Recommender systems.

Darts: Differentiable Architecture Search

Abstract: This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques.


The total training will terminate when the accuracy loss of the global model is below the preset threshold #REFR .

To save computing resources and protect users' privacy, Federated Learning (FL) allows resource-constrained discrete clients to collaboratively learn the global model under the orchestration of a central server while keeping privacy-sensitive data locally #OTHEREFR . * corresponding author.

The current popular FL algorithms are the Federated Averaging (FedAvg) and its variants #OTHEREFR .

It usually consists of a central server and a group of distributed clients.

In each training round, discrete clients first obtain the global model parameters of the previous round from the central server and then uses their own local data to generate the local model parameters.

Then the central server will aggregate the local model parameters from discrete clients and generate the new global parameters.

Under this framework, the server effectively utilizes clients' computing power and storage resources while realizing the separation of privacy data #OTHEREFR .

Although FedAvg performs well on Gboard application #OTHEREFR , recent works #OTHEREFR have pointed out its convergence issues in some settings.

For example, FedAvg #OTHEREFR will converge slowly and the accuracy will be reduced on the non-independent-andidentically-distributed (Non-IID) client dataset #OTHEREFR . Reddi et al.

#OTHEREFR show that this is because of client parameter drift and lack of adaptivity to the different models. Moreover, Tan et al.

#OTHEREFR point out that FedAvg can not learn the personalized FL model for each client.

Adaptive Federated Learning Via Entropy Approach

Abstract: Federated Learning (FL) has recently emerged as a popular framework, which allows resource-constrained discrete clients to cooperatively learn the global model under the orchestration of a central server while storing privacysensitive data locally. However, due to the difference in equipment and data divergence of heterogeneous clients, there will be parameter deviation between local models, resulting in a slow convergence rate and a reduction of the accuracy of the global model. The current FL algorithms use the static client learning strategy pervasively and can not adapt to the dynamic training parameters of different clients. In this paper, by considering the deviation between different local model parameters, we propose an adaptive learning rate scheme for each client based on entropy theory to alleviate the deviation between heterogeneous clients and achieve fast convergence of the global model. It's difficult to design the optimal dynamic learning rate for each client as the local information of other clients is unknown, especially during the local training epochs without communications between local clients and the central server. To enable a decentralized learning rate design for each client, we first introduce meanfield schemes to estimate the terms related to other clients' local model parameters. Then the decentralized adaptive learning rate for each client is obtained in closed form by constructing the Hamilton equation. Moreover, we prove that there exist fixed point solutions for the mean-field estimators, and an algorithm is proposed to obtain them. Finally, extensive experimental results on real datasets show that our algorithm can effectively eliminate the deviation between local model parameters compared to other recent FL algorithms.

Communication-Efficient Learning Of Deep Networks From Decentralized Data

Abstract: Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning. We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.


Ii. Machine Learning Methods For Optimization Problems In Graphs

In 2016, researchers presented in #REFR a framework for combinatorial optimization problems using neural networks and reinforcement learning; the work focused on the TSP, which is a graph problem, but the approach was designed to work with any combinatorial optimization problem.

In the last few years, many machine learning-based approaches that effectively solve several different optimization problems in graphs have been proposed, although none of them designed for solving KEP.

Graph optimization problems already solved with the help of machine learning include the Set Covering Problem #OTHEREFR , Graph Colouring #OTHEREFR , #OTHEREFR , Minimum Vertex Cover #OTHEREFR , #OTHEREFR , Maximum Cut #OTHEREFR , Graph Partitioning #OTHEREFR , Maximum Independent Set #OTHEREFR , Maximum Common Subgraph #OTHEREFR , and the Travelling Salesperson Problem (also called the travelling salesman problem or TSP), one of the most famous NP-hard problems, often used to represent the class, and some variants of it #OTHEREFR - #OTHEREFR .

In 2015, the authors of #OTHEREFR presented a new type of neural network called Pointer Networks, designed to learn how to reorder the elements of an input sequence; they validated the method by using it to solve 3 problems, including the TSP.

In 2017, the authors of #OTHEREFR proposed the utilization of a combination of graph representation learning and reinforcement learning to solve graph optimization problems; they showed that their proposed approach effectively learns to solve at least three of those problems: Minimum Vertex Cover, Maximum Cut, and TSP.

In 2018, the decision variant of the TSP, called Decision Traveling Salesman Problem (DTSP), which is to decide if a given TSP instance admits a Hamiltonian route with a cost no greater than a given threshold C, and is also NP-Hard, was solved in #OTHEREFR with a GNN.

In 2019, the authors of #OTHEREFR tried to solve the TSP using a two stage technique that is very similar to the one presented in this study (described at section IV-C2 and illustrated at Figure  1 ): firstly, a GNN processes the input graph and create scores for each edge of the graph; then, a non-learned search heuristic, which in this case was beam search, uses these scores to construct a solution.

There have been other approaches that use similar techniques: in #OTHEREFR the authors review graph learning methods for solving CO problems, with a focus on two-stage techniques, where the first is based on graph representation learning, which embeds the input graph into low-dimension vectors, and the second uses the embeddings learned in the first stage; #OTHEREFR surveys the use of GNNs as a model of neuralsymbolic computing and their applications, which includes combinatorial optimization problems; #OTHEREFR unifies and refines several of such two stage techniques for neural combinatorial optimization, and test it on the TSP.

Solving The Kidney-Exchange Problem Via Graph Neural Networks With No Supervision

Abstract: This paper introduces a new learning-based approach for approximately solving the Kidney-Exchange Problem (KEP), an NP-hard problem on graphs. The problem consists of, given a pool of kidney donors and patients waiting for kidney donations, optimally selecting a set of donations to optimize the quantity and quality of transplants performed while respecting a set of constraints about the arrangement of these donations. The proposed technique consists of two main steps: the first is a Graph Neural Network (GNN) trained without supervision; the second is a deterministic non-learned search heuristic that uses the output of the GNN to find a valid solution. To allow for comparisons, we also implemented and tested an exact solution method using integer programming, two greedy search heuristics without the machine learning module, and the GNN alone without a heuristic. We analyze and compare the methods and conclude that the learning-based two-stage approach is the best solution quality, outputting approximate solutions on average 1.1 times more valuable than the ones from the deterministic heuristic alone.

Neural Combinatorial Optimization With Reinforcement Learning

Abstract: This paper presents a framework to tackle combinatorial optimization problems using neural networks and reinforcement learning. We focus on the traveling salesman problem (TSP) and train a recurrent neural network that, given a set of city coordinates, predicts a distribution over different city permutations. Using negative tour length as the reward signal, we optimize the parameters of the recurrent neural network using a policy gradient method. We compare learning the network parameters on a set of training graphs against learning them on individual test graphs. Without much engineering and heuristic designing, Neural Combinatorial Optimization achieves close to optimal results on 2D Euclidean graphs with up to 100 nodes. Applied to the KnapSack, another NP-hard problem, the same method obtains optimal solutions for instances with up to 200 items. These results, albeit still far from state-of-the-art, give insights into how neural networks can be used as a general tool for tackling combinatorial optimization problems.


As an example, Shao et al., in #REFR , have highlighted the role of Twitter bots, showing how bots were primarily responsible for the early spread of disinformation, interacting with influential accounts through mentions and replies.

With the advent of the internet and Online Social Networks (OSNs), production and fruition of information feature less mediated procedures, where content and quality do not always go through a rigorous editorial process #OTHEREFR .

Thus, although OSNs make our lives easier by giving us immediate access to information and allowing us to exchange opinions about anything, the danger of being exposed to false or misleading news is high #OTHEREFR .

The promotion of disinformation on OSNs has often been juxtaposed with the existence of automated accounts known as bots.

The struggle between bots hunters and bots creators has been going on for many years now #OTHEREFR , and the actions of these automated accounts with malicious intent have influenced even the purchase of Twitter itself -just remember the $44 billion deal that went up in smoke precisely because of concerns about the unquantified presence of bots on the platform #OTHEREFR .

In this study, we provide a novel approach to bot detection, by leveraging the remarkable advancements in the field of image recognition #OTHEREFR .

To the best of the authors' knowledge, no methodology or tool so far defined for bot detection leverage image recognition.

In particular, we will exploit the potential of Convolutional Neural Networks (CNNs) to classify Twitter accounts as bots or not.

Based on the premise that automated accounts are often programmed to carry out spam and/or disinformation campaigns, numerous works in the literature have proposed detection approaches that leverage coordination and synchronism features of accounts, see, e.g. #OTHEREFR .

From Online Behaviours To Images: A Novel Approach To Social Bot Detection

Abstract: Online Social Networks have revolutionized how we consume and share information, but they have also led to a proliferation of content not always reliable and accurate. One particular type of social accounts is known to promote unreputable content, hyperpartisan, and propagandistic information. They are automated accounts, commonly called bots. Focusing on Twitter accounts, we propose a novel approach to bot detection: we first propose a new algorithm that transforms the sequence of actions that an account performs into an image; then, we leverage the strength of Convolutional Neural Networks to proceed with image classification. We compare our performances with state-of-the-art results for bot detection on genuine accounts / bot accounts datasets well known in the literature. The results confirm the effectiveness of the proposal, because the detection capability is on par with the state of the art, if not better in some cases. This work was partially supported by project SERICS (PE00000014) under the NRRP MUR program funded by the EU-NGEU; by the Integrated Activity Project TOFFEe (TOols for Fighting FakEs) https://toffee.imtlucca.it/; by the IIT-CNR funded Project re-DESIRE (DissEmination of ScIentific REsults 2.0); by 'Prebunking: predicting and mitigating coordinated inauthentic behaviors in social media' project, funded by Sapienza University of Rome.

Anatomy Of An Online Misinformation Network

Abstract: Massive amounts of fake news and conspiratorial content have spread over social media before and after the 2016 US Presidential Elections despite intense fact-checking efforts. How do the spread of misinformation and fact-checking compete? What are the structural and dynamic characteristics of the core of the misinformation diffusion network, and who are its main purveyors? How to reduce the overall amount of misinformation? To explore these questions we built Hoaxy, an open platform that enables large-scale, systematic studies of how misinformation and fact-checking spread and compete on Twitter. Hoaxy captures public tweets that include links to articles from low-credibility and fact-checking sources. We perform k-core decomposition on a diffusion network obtained from two million retweets produced by several hundred thousand accounts over the six months before the election. As we move from the periphery to the core of the network, fact-checking nearly disappears, while social bots proliferate. The number of users in the main core reaches equilibrium around the time of the election, with limited churn and increasingly dense connections. We conclude by quantifying how effectively the network can be disrupted by penalizing the most central nodes. These findings provide a first look at the anatomy of a massive online misinformation diffusion network.