Attention layer: The attention layer is a key component of the transformer model used in natural language processing tasks #REFR , Parikh et al. [2016] .

Both the encoder (left-side) and decoder (right-side) are shown in Figure 2 .

Both Encoder and Decoder are composed of layers and sub-layers that can be stacked on top of each other multiple times, which N x describes in Figure 2 .

The first is a multi-head self-attention mechanism, and the second is a simple, position-wise, fully connected feed-forward network.

The inputs and outputs are first embedded into an n-dimensional space.

foreach client k ∈ S t in parallel do 7 W k t+1 ← ClientU pdate(k, W t , N l ) 8 W t+1 ← k k=1 n k n W k t+1 end W t ← (W t−1 + (W t − W t−1 )/t) end return M (W t )

The attention mechanism enables the model to selectively focus on the relevant parts of the input sequence, rather than treating all inputs equally.

The attention layer consists of three sets of parameters -query (Q), key (K), and value (V) -that are learned during the training process.

In this paper, by applying the self-attention mechanism, we aim at capturing the long dependencies in the input sentence following these steps: • Obtain each word's weight by calculating the similitude between the query matrix and every key matrix (Q, K, V ).

The dot product, concatenation, and perceptron are typically applied as similarity functions.

• Normalize the similitude score of the preceding step and compute the weights with the SoftMax function.

Fedbot: Enhancing Privacy In Chatbots With Federated Learning

Abstract: Chatbots are mainly data-driven and usually based on utterances that might be sensitive. However, training deep learning models on shared data can violate user privacy. Such issues have commonly existed in chatbots since their inception. In the literature, there have been many approaches to deal with privacy, such as differential privacy and secure multi-party computation, but most of them need to have access to users' data. In this context, Federated Learning (FL) aims to protect data privacy through distributed learning methods that keep the data in its location. This paper presents Fedbot, a proof-of-concept (POC) privacy-preserving chatbot that leverages large-scale customer support data. The POC combines Deep Bidirectional Transformer models and federated learning algorithms to protect customer data privacy during collaborative model training. The results of the proof-of-concept showcase the potential for privacy-preserving chatbots to transform the customer support industry by delivering personalized and efficient customer service that meets data privacy regulations and legal requirements. Furthermore, the system is specifically designed to improve its performance and accuracy over time by leveraging its ability to learn from previous interactions.

Disan: Directional Self-Attention Network For Rnn/Cnn-Free Language Understanding

Abstract: Recurrent neural nets (RNN) and convolutional neural nets (CNN) are widely used on NLP tasks to capture the long-term and local dependencies, respectively. Attention mechanisms have recently attracted enormous interest due to their highly parallelizable computation, significantly less training time, and flexibility in modeling dependencies. We propose a novel attention mechanism in which the attention between elements from input sequence(s) is directional and multi-dimensional (i.e., feature-wise). A light-weight neural net, "Directional Self-Attention Network (DiSAN)", is then proposed to learn sentence embedding, based solely on the proposed attention without any RNN/CNN structure. DiSAN is only composed of a directional self-attention with temporal order encoded, followed by a multi-dimensional attention that compresses the sequence into a vector representation. Despite its simple form, DiSAN outperforms complicated RNN models on both prediction quality and time efficiency. It achieves the best test accuracy among all sentence encoding methods and improves the most recent best result by 1.02% on the Stanford Natural Language Inference (SNLI) dataset, and shows state-of-the-art test accuracy on the Stanford Sentiment Treebank (SST), Multi-Genre natural language inference (MultiNLI), Sentences Involving Compositional Knowledge (SICK), Customer Review, MPQA, TREC question-type classification and Subjectivity (SUBJ) datasets.


As illustrated in Figure 5 , AdaIN #REFR transfers the style patterns but losses important content details (1st, 2nd rows).

We randomly collect 17, 124 content images from ImageNet #OTHEREFR and 17, 124 style images from WikiArt #OTHEREFR which are separated from the training set to generate 17, 124 stylization results.

Similar to Stytr 2 [8], we use the mean euclidean distance and the mean instance statistics difference mentioned in 3.4 as the metrics for content preservation and stylization degree.

As presented in Table 1 , compared with existing methods, UniST achieves the best performance in both content and style differences.

SANet #OTHEREFR fails to align the distribution of the style patterns, leading to the distorted object boundaries (1st, 3rd, 5th rows) and inconsistent content background structures (2nd, 4th rows).

The stylization of Linear #OTHEREFR is not satisfactory enough, resulting relatively light migration effects such as pink in the 5th row.

As with the linear transformation, MCCNet #OTHEREFR learns the correlation of each channel through the transformed self-attention, slightly improving the transfer effect, but there are serious overflow problems around the object boundaries (1st, 3rd, 4th rows).

MAST #OTHEREFR distorts the content background structure with excessive style transfer (3rd, 4th rows).

Based on WCT #OTHEREFR in our setting, Artflow #OTHEREFR leads to conspicuous vertical artifacts at the edges of the generated results(3rd, 5th rows).

Two Birds, One Stone: A Unified Framework For Joint Learning Of Image And Video Style Transfers

Abstract: Current arbitrary style transfer models are limited to either image or video domains. In order to achieve satisfying image and video style transfers, two different models are inevitably required with separate training processes on image and video domains, respectively. In this paper, we show that this can be precluded by introducing UniST, a Unified Style Transfer framework for both images and videos. At the core of UniST is a domain interaction transformer (DIT), which first explores context information within the specific domain and then interacts contextualized domain information for joint learning. In particular, DIT enables exploration of temporal information from videos for the image style transfer task and meanwhile allows rich appearance texture from images for video style transfer, thus leading to mutual benefits. Considering heavy computation of traditional multihead self-attention, we present a simple yet effective axial multi-head self-attention (AMSA) for DIT , which improves computational efficiency while maintains style transfer performance. To verify the effectiveness of UniST, we conduct extensive experiments on both image and video style transfer tasks and show that UniST performs favorably against state-of-the-art approaches on both tasks. Our code and results will be released.

Arbitrary Style Transfer In Real-Time With Adaptive Instance Normalization

Abstract: Gatys et al. recently introduced a neural algorithm that renders a content image in the style of another image, achieving so-called style transfer. However, their framework requires a slow iterative optimization process, which limits its practical application. Fast approximations with feed-forward neural networks have been proposed to speed up neural style transfer. Unfortunately, the speed improvement comes at a cost: the network is usually tied to a fixed set of styles and cannot adapt to arbitrary new styles. In this paper, we present a simple yet effective approach that for the first time enables arbitrary style transfer in real-time. At the heart of our method is a novel adaptive instance normalization (AdaIN) layer that aligns the mean and variance of the content features with those of the style features. Our method achieves speed comparable to the fastest existing approach, without the restriction to a pre-defined set of styles. In addition, our approach allows flexible user controls such as content-style trade-off, style interpolation, color & spatial controls, all using a single feed-forward neural network.


Faster Estimation Of Stabilizer Fidelity

We note that Aaronson studied PAC learning of quantum states in the so-called realizable setting #REFR .

Finally, we remark that the problem we solve is similar in spirit to the agnostic probably approximately correct (PAC) learning framework #OTHEREFR .

In the agnostic PAC model, a learner is given labeled training data {(x 1 , y 1 ), . . .

, (x m , y m )} from some unknown distribution D, as well as some concept class C to choose a hypothesis from.

The goal of the learner is to find a hypothesis function h ∈ C that approximates the best fit for the training data, even though no function in C will necessarily fit the training data perfectly.

In an analogous fashion, our algorithm finds a stabilizer state |φ that approximates the best fit for |ψ over the set of stabilizer states, which need not contain |ψ .

However, agnostic PAC learning of quantum states has not yet appeared in the literature.

Improved Stabilizer Estimation Via Bell Difference Sampling

Abstract: We study the complexity of learning quantum states in various models with respect to the stabilizer formalism and obtain the following results: • We prove that Ω(n) T-gates are necessary for any Clifford+T circuit to prepare computationally pseudorandom quantum states, an exponential improvement over the previously known bound. This bound is asymptotically tight if linear-time quantum-secure pseudorandom functions exist. • Given an n-qubit pure quantum state |ψ that has fidelity at least τ with some stabilizer state, we give an algorithm that outputs a succinct description of a stabilizer state that witnesses fidelity at least τ − ε. The algorithm uses O(n/(ε 2 τ 4)) samples and exp O(n/τ 4) /ε 2 time. In the regime of τ constant, this algorithm estimates stabilizer fidelity substantially faster than the naïve exp(O(n 2))-time brute-force algorithm over all stabilizer states. • We improve the soundness analysis of the stabilizer state property testing algorithm due to Gross, Nezami, and Walter [Comms. Math. Phys. 385 (2021)]. As an application, we exhibit a tolerant property testing algorithm for stabilizer states. The underlying algorithmic primitive in all of our results is Bell difference sampling. To prove our results, we establish and/or strengthen connections between Bell difference sampling, symplectic Fourier analysis, and graph theory.

The Learnability Of Quantum States

Abstract: A method of cladding a steel roll, such as a felt roll with a non-corrosive shell, ensures that the shell is tightly fitted to the core and is free to expand and contract on the core to avoid distortion problems. The method comprises the steps of preparing two stainless steel shells each having a length equivalent to the circumference of the core plus a predetermined extra, wrapping the shells about the core, side by side, leaving a predetermined circumferential gap between the shells, heating and clamping the shells tightly around the core until the longitudinal joints each attain a predetermined longitudinal gap, welding the longitudinal gaps, and circumferential gap between the shells while the shells are hot, ensuring the sheets are welded together and not welded to the core.


Cityscapes #REFR is an autonomous driving dataset that contains urban street scenes recorded from 50 different cities with precise pixel-wise annotations of 19 semantic categories.

DensePASS #OTHEREFR is a panoramic dataset and contains 2,000 images for training and 100 precise annotated images for testing.

WildPASS #OTHEREFR is a panoramic dataset designed to capture diverse scenes from all around the globe and contains 2500 panoramas. Evaluation.

We take the mean Intersection-over-Union (mIoU) as the evaluation metric in both the source and target domains.

Our framework is evaluated on the DenseP-ASS / WildPASS validation set via test at a single scale, and the resolution is 400 × 2048. Implementation details.

Our framework is implemented with Pytorch and trained on multiple NVIDIA GPUs.

Both Style And Distortion Matter: Dual-Path Unsupervised Domain Adaptation For Panoramic Semantic Segmentation

Abstract: The ability of scene understanding has sparked active research for panoramic image semantic segmentation. However, the performance is hampered by distortion of the equirectangular projection (ERP) and a lack of pixel-wise annotations. For this reason, some works treat the ERP and pinhole images equally and transfer knowledge from the pinhole to ERP images via unsupervised domain adaptation (UDA). However, they fail to handle the domain gaps caused by: 1) the inherent differences between camera sensors and captured scenes; 2) the distinct image formats (e.g., ERP and pinhole images). In this paper, we propose a novel yet flexible dual-path UDA framework, DPPASS, taking ERP and tangent projection (TP) images as inputs. To reduce the domain gaps, we propose cross-projection and intra-projection training. The cross-projection training includes tangent-wise feature contrastive training and prediction consistency training. That is, the former formulates the features with the same projection locations as positive examples and vice versa, for the models' awareness of distortion, while the latter ensures the consistency of cross-model predictions between the ERP and TP. Moreover, adversarial intra-projection training is proposed to reduce the inherent gap, between the features of the pinhole images and those of the ERP and TP images, respectively. Importantly, the TP path can be freely removed after training, leading to no additional inference cost. Extensive experiments on two benchmarks show that our DPPASS achieves +1.06% mIoU increment than the state-of-the-art approaches. https: //vlis2022.github.io/cvpr23/DPPASS

The Cityscapes Dataset For Semantic Urban Scene Understanding

Abstract: Visual understanding of complex urban street scenes is an enabling factor for a wide range of applications. Object detection has benefited enormously from large-scale datasets, especially in the context of deep learning. For semantic urban scene understanding, however, no current dataset adequately captures the complexity of real-world urban scenes. To address this, we introduce Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling. Cityscapes is comprised of a large, diverse set of stereo video sequences recorded in streets from 50 different cities. 5000 of these images have high quality pixel-level annotations, 20 000 additional images have coarse annotations to enable methods that leverage large volumes of weakly-labeled data. Crucially, our effort exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity. Our accompanying empirical study provides an in-depth analysis of the dataset characteristics, as well as a performance evaluation of several state-of-the-art approaches based on our benchmark.

precise pixel-wise annotations
autonomous driving dataset


Recent supervised learning work RAFT #REFR demonstrates the strength of recurrent blocks in optical flow estimation.

In continuous motion of objects, the loss of pixels information due to occlusion and blurring, which may not be a problem in supervised learning due to the presence of ground truth from God's view, is untraceable and tricky in unsupervised learning.

The predictive coding structure based on temporal dynamics has at least the following potential advantages for alleviating the problem.

First, Based on the motion temporal smoothness, temporal dynamic networks can propagate the previous motion prior through hidden states to provide a more reliable reference for untraceable pixels.

Second, the motion estimation of the occluded object is practically equivalent to the prediction of the location of the occluded object in the latter frame, which has a strong connection with predictive coding mechanism as one of the motivations for introducing temporal dynamics.

Accordingly, this work constructs an CNN-RNN based model that understands optical flow estimation in dynamic environments.

Instead of recurrent refinement of a single frame, we introduce both spatial and temporal recurrences into the model, constructing a spatial-temporal dual recurrent block with predictive coding arrangement.

The final resulting network has a small number of parameters yet stably converges in the sequence of arbitrary length and demonstrates higher performance in ablation study compared to other models. We trained the model in the temporal causal sequence.

To make the model correctly grasp the object occlusion, brightness variation, color variation, and various content blurring caused by the movement in the dynamic environment, three training enhancers based on self-supervised learning are proposed, including Dynamic Occlusion Enhancer (DOE), Content Variation Enhancer (CVE), and Spatial Variation Enhancer (SVE).

In which DOE extracts sub-object blocks from the original data distribution and simulates the random natural motion of objects in multi-frame images, allowing the model to understand object occlusion patterns over prolonged temporal environments.

Furthermore, we propose a mixed supervision strategy by combining the unsupervised loss and self-supervised loss in the DOE, which simultaneously provides reliable supervision on the occluded region and the occlusion per se, facilitating a better generalization of the network to occlusion scenes.

Unsupervised Learning Optical Flow In Multi-Frame Dynamic Environment Using Temporal Dynamic Modeling

Abstract: For visual estimation of optical flow, a crucial function for many vision tasks, unsupervised learning, using the supervision of view synthesis, has emerged as a promising alternative to supervised methods, since ground-truth flow is not readily available in many cases. However, unsupervised learning is likely to be unstable when pixel tracking is lost due to occlusion and motion blur, or the pixel matching is impaired due to variation in image content and spatial structure over time. In natural environments, dynamic occlusion or object variation is a relatively slow temporal process spanning several frames. We, therefore, explore the optical flow estimation from multiple-frame sequences of dynamic scenes, whereas most of the existing unsupervised approaches are based on temporal static models. We handle the unsupervised optical flow estimation with a temporal dynamic model by introducing a spatial-temporal dual recurrent block based on the predictive coding structure, which feeds the previous high-level motion prior to the current optical flow estimator. Assuming temporal smoothness of optical flow, we use motion priors of the adjacent frames to provide more reliable supervision of the occluded regions. To grasp the essence of challenging scenes, we simulate various scenarios across long sequences, including dynamic occlusion, content variation, and spatial variation, and adopt self-supervised distillation to make the model understand the object's motion patterns in a prolonged dynamic environment. Experiments on KITTI 2012, KITTI 2015, Sintel Clean, and Sintel Final datasets demonstrate the effectiveness of our methods on unsupervised optical flow estimation. The proposal reaches state-of-the-art performance with advantages in memory overhead.

Raft: Recurrent All-Pairs Field Transforms For Optical Flow

Abstract: We introduce Recurrent All-Pairs Field Transforms (RAFT), a new deep network architecture for optical flow. RAFT extracts per-pixel features, builds multi-scale 4D correlation volumes for all pairs of pixels, and iteratively updates a flow field through a recurrent unit that performs lookups on the correlation volumes. RAFT achieves state-of-the-art performance, with strong cross-dataset generalization and high efficiency in inference time, training speed, and parameter count. Code is available \url{https://github.com/princeton-vl/RAFT}.


Motivated by the Sinc-Net's #REFR promising performance and interpretability, the time branch of our framework is designed based on SincNet.

The time branch is designed to extract the time-domain feature from the raw speech signals.

Following SincNet, the first layer of the time branch is designed as the learnable band pass filters to model the frequency characteristics for speaker recognition. A band pass filter can be formulated as:

where f 1 /f 2 is the low/high cutoff frequency, sinc(x) = sin x/x.

By designing the filters as band pass filters, the model has fewer parameters and better interpretability.

Except for the first layer, all other layers in our time branch are the typical 1dimensional convolutional layer (Conv), followed by the batch normalization layer (BN) #OTHEREFR and ReLU layer (ReLU) #OTHEREFR .

Time-Frequency Network For Robust Speaker Recognition

Abstract: The wide deployment of speech-based biometric systems usually demands high-performance speaker recognition algorithms. However, most of the prior works for speaker recognition either process the speech in the frequency domain or time domain, which may produce suboptimal results because both time and frequency domains are important for speaker recognition. In this paper, we attempt to analyze the speech signal in both time and frequency domains and propose the timefrequency network (TFN) for speaker recognition by extracting and fusing the features in the two domains. Based on the recent advance of deep neural networks, we propose a convolution neural network to encode the raw speech waveform and the frequency spectrum into domain-specific features, which are then fused and transformed into a classification feature space for speaker recognition. Experimental results on the publicly available datasets TIMIT and LibriSpeech show that our framework is effective to combine the information in the two domains and performs better than the state-of-the-art methods for speaker recognition.

Speech And Speaker Recognition From Raw Waveform With Sincnet

Abstract: Deep neural networks can learn complex and abstract representations, that are progressively obtained by combining simpler ones. A recent trend in speech and speaker recognition consists in discovering these representations starting from raw audio samples directly. Differently from standard hand-crafted features such as MFCCs or FBANK, the raw waveform can potentially help neural networks discover better and more customized representations. The high-dimensional raw inputs, however, can make training significantly more challenging. This paper summarizes our recent efforts to develop a neural architecture that efficiently processes speech from audio waveforms. In particular, we propose SincNet, a novel Convolutional Neural Network (CNN) that encourages the first layer to discover meaningful filters by exploiting parametrized sinc functions. In contrast to standard CNNs, which learn all the elements of each filter, only low and high cutoff frequencies of band-pass filters are directly learned from data. This inductive bias offers a very compact way to derive a customized front-end, that only depends on some parameters with a clear physical meaning. Our experiments, conducted on both speaker and speech recognition, show that the proposed architecture converges faster, performs better, and is more computationally efficient than standard CNNs.

time branch
Sinc-Net's promising performance


Comparison With State-Of-The-Art Methods

Our ADP also got +2.6%/+7.6% improvement in Rank-1 and +8.7%/+10.5% in mAP compared with SOTA partial Re-ID method VPM #REFR .

Though our proposed module is designed to solve occlusion problems rather than holistic problems, we still get comparable performance in holistic datasets compared with other holistic Re-ID methods.

For example, the proposed ADP can achieve +0.3%/+1.6% improvement in Rank-1 and +0.9%/+3.1% in mAP on Market-1501 and DukeMTMC-ReID datasets, respectively, Table 4 .

Ablation study of the dual-path loss used in DPC on Occluded-Duke dataset.

AM denotes the module use shared angular softmax with single-path structure, while DP represents the module with an asymmetric dual-path structure.

compared with the state-of-the-art method ISP #OTHEREFR .

In comparison with occluded state-ofthe-art methods, our method is naturally at a disadvantage in performance due to the reason that PFD #OTHEREFR is based on part feature while our method is based on global feature.

But we still achieved a very competitive and close performance to PFD.

Attention Disturbance And Dual-Path Constraint Network For Occluded Person Re-Identification

Abstract: Occluded person re-identification (Re-ID) aims to address the potential occlusion problem when matching occluded or holistic pedestrians from different camera views. Many methods use the background as artificial occlusion and rely on attention networks to exclude noisy interference. However, the significant discrepancy between simple background occlusion and realistic occlusion can negatively impact the generalization of the network. To address this issue, we propose a novel transformer-based Attention Disturbance and Dual-Path Constraint Network (ADP) to enhance the generalization of attention networks. Firstly, to imitate real-world obstacles, we introduce an Attention Disturbance Mask (ADM) module that generates an offensive noise, which can distract attention like a realistic occluder, as a more complex form of occlusion. Secondly, to fully exploit these complex occluded images, we develop a Dual-Path Constraint Module (DPC) that can obtain preferable supervision information from holistic images through dualpath interaction. With our proposed method, the network can effectively circumvent a wide variety of occlusions using the basic ViT baseline. Comprehensive experimental evaluations conducted on person re-ID benchmarks demonstrate the superiority of ADP over state-of-the-art methods.

Perceive Where To Focus: Learning Visibility-Aware Part-Level Features For Partial Person Re-Identification

Abstract: This paper considers a realistic problem in person re-identification (re-ID) task, i.e., partial re-ID. Under partial re-ID scenario, the images may contain a partial observation of a pedestrian. If we directly compare a partial pedestrian image with a holistic one, the extreme spatial misalignment significantly compromises the discriminative ability of the learned representation. We propose a Visibility-aware Part Model (VPM) for partial re-ID, which learns to perceive the visibility of regions through self-supervision. The visibility awareness allows VPM to extract region-level features and compare two images with focus on their shared regions (which are visible on both images). VPM gains two-fold benefit toward higher accuracy for partial re-ID. On the one hand, compared with learning a global feature, VPM learns region-level features and thus benefits from fine-grained information. On the other hand, with visibility awareness, VPM is capable to estimate the shared regions between two images and thus suppresses the spatial misalignment. Experimental results confirm that our method significantly improves the learned feature representation and the achieved accuracy is on par with the state of the art.


Formative works like data sheets and model cards #REFR brought this approach to the fore, complementing evaluations by articulating design decisions and developer positions involved in creating assets.

ImageNet; #OTHEREFR , others stress-test models [e.g.

CheckList; #OTHEREFR or adversarially identify failures [e.g.

red-teaming; #OTHEREFR and still others characterize models along a range of dimensions [e.g. HELM; .

In general, while some efforts expand evaluation to datasets #OTHEREFR , Swayamdipta et al., 2020 #OTHEREFR or adopt methodologies from human-computer interaction to consider human factors like user experience [Lee et al., 2022a,b] , for the most part, evaluation aims to characterize a specific model in isolation.

Second, documentation is a growing practice for specifying metadata about the broader context that situates model and system development.

Subsequent efforts have enriched these approaches to make these documentation artifacts more useful, accessible, or otherwise aligned to specific informational needs [Crisan et al., 2022] .

14 In general, documentation efforts aim to contextualize a specific asset against a broader social backdrop, often with an inclination towards how the asset came to be and with greater uptake for research-centric assets to our knowledge.

Third, analyses and critiques have become increasingly relevant, showcasing much of the latent and oft-overlooked underpinnings of AI development and deployment.

These works often bring questions of values and power to the fore, frequently appealing to concepts or methodologies from the social sciences and disciplines beyond computer science [e.g.

#OTHEREFR , Koch et al., 2021 #OTHEREFR .

Ecosystem Graphs: The Social Footprint Of Foundation Models

Abstract: Foundation models (e.g. ChatGPT, StableDiffusion) pervasively influence society, warranting immediate social attention. While the models themselves garner much attention, to accurately characterize their impact, we must consider the broader sociotechnical ecosystem. We propose Ecosystem Graphs as a documentation framework to transparently centralize knowledge of this ecosystem. Ecosystem Graphs is composed of assets (datasets, models, applications) linked together by dependencies that indicate technical (e.g. how Bing relies on GPT-4) and social (e.g. how Microsoft relies on OpenAI) relationships. To supplement the graph structure, each asset is further enriched with fine-grained metadata (e.g. the license or training emissions). We document the ecosystem extensively at https://crfm. stanford.edu/ecosystem-graphs: as of March 16, 2023, we annotate 262 assets (64 datasets, 128 models, 70 applications) from 63 organizations linked by 356 dependencies. We show Ecosystem Graphs functions as a powerful abstraction and interface for achieving the minimum transparency required to address myriad use cases. Therefore, we envision Ecosystem Graphs will be a community-maintained resource that provides value to stakeholders spanning AI researchers, industry professionals, social scientists, auditors and policymakers.

Model Cards For Model Reporting

Abstract: Trained machine learning models are increasingly used to perform high-impact tasks in areas such as law enforcement, medicine, education, and employment. In order to clarify the intended use cases of machine learning models and minimize their usage in contexts for which they are not well suited, we recommend that released models be accompanied by documentation detailing their performance characteristics. In this paper, we propose a framework that we call model cards, to encourage such transparent model reporting. Model cards are short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups (e.g., race, geographic location, sex, Fitzpatrick skin type [15]) and intersectional groups (e.g., age and race, or sex and Fitzpatrick skin type) that are relevant to the intended application domains. Model cards also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information. While we focus primarily on human-centered machine learning models in the application fields of computer vision and natural language processing, this framework can be used to document any trained machine learning model. To solidify the concept, we provide cards for two supervised models: One trained to detect smiling faces in images, and one trained to detect toxic comments in text. We propose model cards as a step towards the responsible democratization of machine learning and related artificial intelligence technology, increasing transparency into how well artificial intelligence technology works. We hope this work encourages those releasing trained machine learning models to accompany model releases with similar detailed evaluation numbers and other relevant documentation.


Gender Bias Analysis Through Text Markers

In total, 97.66% of the captions generated contained gender-marked terms, versus #REFR appearance predictions: this is consistent given the fact that VQA mostly consists of single word predictions, whereas captions are full sentences.

As described in Section 3.1, we used captions and open-ended VQA to obtain textual representations of the generated images, whose discrete nature makes identifying trends related to social attribute variation more tractable.

More specifically, we base our evaluation on the likelihood that a caption or VQA answer for a given profession contains gender-marked words such as 'man' and 'woman' or gender-unspecified descriptors such as 'person' or the profession name -we present these results in Table 1 .

To put the percentage of predictions that contain gender-marked terms such as 'man' and 'woman' into perspective, we compare them with the percentages of men and women in these professions provided by the BLS #OTHEREFR .

We find that Dall•E 2 has the largest discrepancy compared to the BLS-provided numbers, with its captions mentioning women on average 27% less, and its VQA mentioning them 25% less, and Stable Diffusion v.1.4 having the least (approximately 9% for both captions and VQA).

The professions with the biggest discrepancy between the BLS and both captions and VQA across all models are: clerk (57 and 55% less), data entry keyer (55/53% less) and real estate broker (52/54% less), whereas those that have more captions that mention women are: singer (29/36% more), cleaner (20/16% more) and dispatcher (19/16% more).

Very few of the generated image captions mention gender-neutral terms such as 'person' (an average of less than 1% of captions for any of the models, distributed equally across professions), and none use the 'non-binary' gender marker.

Also, less than 0.5% of captions and 2% of VQA generations explicitly mention the profession in the prompt, but it is interesting to note that a single profession, police officer, had explicit mentions of the profession name in 80.95% of captions.

Stable Bias: Analyzing Societal Representations In Diffusion Models

Abstract: As machine learning-enabled Text-to-Image (TTI) systems are becoming increasingly prevalent and seeing growing adoption as commercial services, characterizing the social biases they exhibit is a necessary first step to lowering their risk of discriminatory outcomes. This evaluation, however, is made more difficult by the synthetic nature of these systems' outputs; since artificial depictions of fictive humans have no inherent gender or ethnicity nor do they belong to socially-constructed groups, we need to look beyond common categorizations of diversity or representation. To address this need, we propose a new method for exploring and quantifying social biases in TTI systems by directly comparing collections of generated images designed to showcase a system's variation across social attributes-gender and ethnicity-and target attributes for bias evaluation-professions and gender-coded adjectives. Our approach allows us to (i) identify specific bias trends through visualization tools, (ii) provide targeted scores to directly compare models in terms of diversity and representation, and (iii) jointly model interdependent social variables to support a multidimensional analysis. We use this approach to analyze over 96,000 images generated by 3 popular TTI systems (Dall•E 2 , Stable Diffusion v 1.4 and v 2) and find that all three significantly over-represent the portion of their latent space associated with whiteness and masculinity across target attributes; among the systems studied, Dall•E 2 shows the least diversity, followed by Stable Diffusion v2 then v1.4.

Microsoft Coco: Common Objects In Context

Abstract: We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.


The results show that our code has a lower edit distance error compared to the code in #REFR , for a higher code rate.

Note that this improvement comes at the expense of a lower code rate; however, different trade-offs can be achieved by varying the parameters δ and .

3 , we also compare the performance our code to the VT-based code in #OTHEREFR in the regime where the average number of deletions per segment is a constant (i.e., α = 1).

The code in #OTHEREFR is constructed by concatenating n/ = 17 VT codewords each of size = 59, resulting in a code with rate ≈ 0.9.

The parameters of our code are set to δ = 3, n = 994, and = 71, resulting in a code with rate ≈ 0.934.

Optimal Codes Detecting Deletions In Concatenated Binary Strings Applied To Trace Reconstruction

Abstract: Consider two or more strings x 1 , x 2 ,. .. , that are concatenated to form x = x 1 , x 2 ,. .. . Suppose that up to δ deletions occur in each of the concatenated strings. Since deletions alter the lengths of the strings, a fundamental question to ask is: how much redundancy do we need to introduce in x in order to recover the boundaries of x 1 , x 2 ,. . .? This boundary problem is equivalent to the problem of designing codes that can detect the exact number of deletions in each concatenated string. In this work, we answer the question above by first deriving converse results that give lower bounds on the redundancy of deletion-detecting codes. Then, we present a marker-based code construction whose redundancy is asymptotically optimal in δ among all families of deletion-detecting codes, and exactly optimal among all block-by-block decodable codes. To exemplify the usefulness of such deletion-detecting codes, we apply our code to trace reconstruction and design an efficient coded reconstruction scheme that requires a constant number of traces.

Coding For Deletion Channels With Multiple Traces

Abstract: Motivated by the sequence reconstruction problem from traces in DNA-based storage, we consider the problem of designing codes for the deletion channel when multiple observations (or traces) are available to the decoder. We propose simple binary and non-binary codes based on Varshamov-Tenengolts (VT) codes. The proposed codes split the codeword in blocks and employ a VT code in each block. The availability of multiple traces helps the decoder to identify deletion-free copies of a block, and to avoid mis-synchronization while decoding. The encoding complexity of the proposed scheme is linear in the codeword length; the decoding complexity is linear in the codeword length, and quadratic in the number of deletions and the number of traces. The proposed scheme offers an explicit low-complexity technique for correcting deletions using multiple traces.


Iii. Interview For Engineering Experience

Participants who were categorized as elite developers were invited through email or text messages sent to public channels of their respective organizations #REFR .

The study employed a purposive sampling technique and a semi-structured approach to conduct interviews with participants selected from a list of OSS projects with bot deployment #OTHEREFR .

The interview process consisted of 10 pre-defined questions grouped into three main sections, namely introduction, workflow, and expectations of SE bots (see Tab. I).

All interviews, which lasted between 25 to 45 minutes each, were conducted across 2021 and 2022.

Participants' responses were analyzed using an open coding procedure, and the raw interview transcripts were annotated for thematic analysis to extract meaningful findings #OTHEREFR Ch.4] .

To ensure coding schema validity, two researchers assessed one participant's transcript, and substantial agreement was achieved (Cohen's κ = 0.66).

The study's primary objectives were to determine the workflow and expectations of SE bots, which were the major findings of the research.

Optimizing Workflow For Elite Developers: Perspectives On Leveraging Se Bots

Abstract: Small-scale automation services in Software Engineering, known as SE Bots, have gradually infiltrated every aspect of daily software development with the goal of enhancing productivity and well-being. While leading the OSS development, elite developers have often burned out from holistic responsibilities in projects and looked for automation support. Building on prior research in BotSE and our interviews with elite developers, this paper discusses how to design and implement SE bots that integrate into the workflows of elite developers and meet their expectations. We present six main design guidelines for implementing SE bots for elite developers, based on their concerns about noise, security, simplicity, and other factors. Additionally, we discuss the future directions of SE bots, especially in supporting elite developers' increasing workload due to rising demands.

Unveiling Elite Developers' Activities In Open Source Projects

Abstract: Open-source developers, particularly the elite developers, maintain a diverse portfolio of contributing activities. They do not only commit source code but also spend a significant amount of effort on other communicative, organizational, and supportive activities. However, almost all prior research focuses on a limited number of specific activities and fails to analyze elite developers' activities in a comprehensive way. To bridge this gap, we conduct an empirical study with fine-grained event data from 20 large open-source projects hosted on GitHub. Thus, we investigate elite developers' contributing activities and their impacts on project outcomes. Our analyses reveal three key findings: (1) they participate in a variety of activities while technical contributions (e.g., coding) accounting for a small proportion only; (2) they tend to put more effort into supportive and communicative activities and less effort into coding as the project grows; (3) their participation in non-technical activities is negatively associated with the project's outcomes in term of productivity and software quality. These results provide a panoramic view of elite developers' activities and can inform an individual's decision making about effort allocation, thus leading to finer project outcomes. The results also provide implications for supporting these elite developers.


Then, collected real data displays a sharp change in the true harvesting function, e.g., Figure 4 in #REFR .

Our previous work #OTHEREFR accomplishes the learning approach and theoretical guarantees simultaneously by using splines and arXiv:2304.01560v1 [cs.IT] 4 Apr 2023 local polynomials.

The work is motivated by the fact that for low-dimensional functions, classic function interpolation methods perform as good as deep learning regression but also provide theoretical guarantees.

Specifically, assuming that a smooth harvesting function-formally, belonging to a class of Sobolev functions-is known only at sample points, it characterizes asymptotic performance loss of information rate and energy incurred by such sampled knowledge #OTHEREFR .

Instead of deep learning regression, it suggests using the spline and local polynomial methods to reconstruct the unknown harvesting function and using it as a proxy for the true one.

The results are promising as performance losses from the true optimal performance are asymptotically negligible with the number of samples, but the smoothness assumption is restrictive because the harvesting function occasionally varies very quickly as a signal varies, e.g., due to diode's mode transition, hardware defects, or unexpected resonance of circuits if frequency modulation is adopted.

Even when the true function is (formally Sobolev) smooth, such sharp change implies a large magnitude of its derivative, which is in turn captured by a hidden constant factor of the loss characterization that depends on derivatives #OTHEREFR . Therefore, the resulting loss bounds are vacuous.

In such a case, it is more reasonable to model it as a function of bounded variation that may have discontinuities, but its total variation is bounded.

This is a larger class of functions than the Sobolev functions, for which the spline reconstruction method does not perform well.

This work considers a class of harvesting functions of bounded variation, which may have a finite number of jumps (discontinuities).

In addition, we assume that the harvesting function is accessible only at discrete sample points, either in the absence of noise or in the presence of noise.

Information And Energy Transmission With Wavelet-Reconstructed Harvesting Functions

Abstract: In practical simultaneous information and energy transmission (SIET), the exact energy harvesting function is usually unavailable because an energy harvesting circuit is nonlinear and nonideal. In this work, we consider a SIET problem where the harvesting function is accessible only at experimentally-taken sample points and study how close we can design SIET to the optimal system with such sampled knowledge. Assuming that the harvesting function is of bounded variation that may have discontinuities, we separately consider two settings where samples are taken without and with additive noise. For these settings, we propose to design a SIET system as if a waveletreconstructed harvesting function is the true one and study its asymptotic performance loss of energy and information delivery from the true optimal one. Specifically, for noiseless samples, it is shown that designing SIET as if the wavelet-reconstructed harvesting function is the truth incurs asymptotically vanishing energy and information delivery loss with the number of samples. For noisy samples, we propose to reconstruct wavelet coefficients via soft-thresholding estimation. Then, we not only obtain similar asymptotic losses to the noiseless case but also show that the energy loss by wavelets is asymptotically optimal up to a logarithmic factor.

Learning To Communicate And Energize: Modulation, Coding And Multiple Access Designs For Wireless Information-Power Transmission

Abstract: The explosion of the number of low-power devices in the next decades calls for a re-thinking of wireless network design, namely, unifying wireless transmission of information and power so as to make the best use of the RF spectrum, radiation, and infrastructure for the dual purpose of communicating and energizing. This paper provides a novel learning-based approach towards such wireless network design. To that end, a parametric model of a practical energy harvester, accounting for various sources of nonlinearities, is proposed using a nonlinear regression algorithm applied over collected real data. Relying on the proposed model, the learning problem of modulation design for Simultaneous Wireless Information-Power Transmission (SWIPT) over a point-to-point link is studied. Joint optimization of the transmitter and the receiver is implemented using Neural Network (NN)-based autoencoders. The results reveal that by increasing the receiver power demand, the baseband transmit modulation constellation converges to an On-Off keying signalling. Utilizing the observations obtained via learning, an algorithmic SWIPT modulation design is proposed. It is observed via numerical results that the performance loss of the proposed modulations are negligible compared to the ones obtained from learning. Extension of the studied problem to learning modulation design for multi-user SWIPT scenarios and coded modulation design for point-to-point SWIPT are considered. The major conclusion of this work is to utilize learning-based results to design non learning-based algorithms, which perform as well. In particular, inspired by the results obtained via learning, an algorithmic approach for coded modulation design is proposed, which performs very close to its learning counterparts, and is significantly superior due to its high real-time adaptability to new system design parameters.


We reported TPR at 0.1% and 1% FPR for comparison to #REFR , where it can be seen our model performed better in both.

The table also contains results using loss as a performance metric for SOREL-20M and EMBER-2018; this is to show the difference in F1-score and loss as a performance metric in the final stage, this will be brought up in the discussion section of this section.

Some other works shown in the tables only report a subset of the metrics, but are still shown for comparison.

In particular, for SOREL-20M, Table 4 shows our AUC results are on par with the FFNN ensemble from #OTHEREFR and slightly exceeds #OTHEREFR , the work that presented the auxiliary model heads for SOREL-20M.

Our model significantly exceeds the AUC under 0.1% FPR of the only other work #OTHEREFR , which reported this parameter.

The accuracy of our model is similar but higher than #OTHEREFR .

With respect to EMBER-2018 in Table 5 , #OTHEREFR performs slightly better in their reported metrics, AUC and accuracy, whereas the rest of their metrics are not reported.

Our model chosen in the final training phase is similar to other results in AUC and accuracy, surpassing #OTHEREFR in AUC, and surpassing both #OTHEREFR , #OTHEREFR in accuracy.

The AUC under 0.1% FPR of our model far surpasses the results of #OTHEREFR .

The TPR at 0.1% is the only reported metric of #OTHEREFR , which is significantly Fig. 4 . ROC using F1-Score for Selection higher than our results.

Due to limited metrics provided by other related works, it is difficult to compare the efficacy of our AutoML method in a holistic sense.

Automated Machine Learning For Deep Learning Based Malware Detection

Abstract: Deep learning (DL) has proven to be effective in detecting sophisticated malware that is constantly evolving. Even though deep learning has alleviated the feature engineering problem, finding the most optimal DL model, in terms of neural architecture search (NAS) and the model's optimal set of hyper-parameters, remains a challenge that requires domain expertise. In addition, many of the proposed state-of-the-art models are very complex and may not be the best fit for different datasets. A promising approach, known as Automated Machine Learning (AutoML), can reduce the domain expertise required to implement a custom DL model. AutoML reduces the amount of human trial-and-error involved in designing DL models, and in more recent implementations can find new model architectures with relatively low computational overhead. Research on the feasibility of using AutoML for malware detection is very limited. This work provides a comprehensive analysis and insights on using AutoML for static and online malware detection. For static, our analysis is performed on two widely used malware datasets: SOREL-20M to demonstrate efficacy on large datasets; and EMBER-2018, a smaller dataset specifically curated to hinder the performance of machine learning models. In addition, we show the effects of tuning the NAS process parameters on finding a more optimal malware detection model on these static analysis datasets. Further, we also demonstrate that AutoML is performant in online malware detection scenarios using Convolutional Neural Networks (CNNs) for cloud IaaS. We compare an AutoML technique to six existing state-of-the-art CNNs using a newly generated online malware dataset with and without other applications running in the background during malware execution. We show that the AutoML technique is more performant than the state-of-the-art CNNs with little overhead in finding the architecture. In general, our experimental results show that the performance of AutoML based static and online malware detection models are on par or even better than state-of-the-art models or hand-designed models presented in literature.

Aloha: Auxiliary Loss Optimization For Hypothesis Augmentation

Abstract: Malware detection is a popular application of Machine Learning for Information Security (ML-Sec), in which an ML classifier is trained to predict whether a given file is malware or benignware. Parameters of this classifier are typically optimized such that outputs from the model over a set of input samples most closely match the samples' true malicious/benign (1/0) target labels. However, there are often a number of other sources of contextual metadata for each malware sample, beyond an aggregate malicious/benign label, including multiple labeling sources and malware type information (e.g., ransomware, trojan, etc.), which we can feed to the classifier as auxiliary prediction targets. In this work, we fit deep neural networks to multiple additional targets derived from metadata in a threat intelligence feed for Portable Executable (PE) malware and benignware, including a multi-source malicious/benign loss, a count loss on multi-source detections, and a semantic malware attribute tag loss. We find that incorporating multiple auxiliary loss terms yields a marked improvement in performance on the main detection task. We also demonstrate that these gains likely stem from a more informed neural network representation and are not due to a regularization artifact of multi-target learning. Our auxiliary loss architecture yields a significant reduction in detection error rate (false negatives) of 42.6% at a false positive rate (FPR) of $10^{-3}$ when compared to a similar model with only one target, and a decrease of 53.8% at $10^{-5}$ FPR.


Whether the infinite clique problem over arbitrary ω-synchronous relations is decidable is a longstanding open problem #REFR .

In fact, the infinite clique problem for arbitrary synchronous relations was shown to be NL-complete #OTHEREFR (later simplified in #OTHEREFR ).

However, in certain settings we need to exploit the fact that ∼ R j is the complement of an equivalence relation ∼ R j .

#OTHEREFR While the authors of #OTHEREFR did not analyze the complexity of their algorithm, it is easy to see that their algorithm runs in elementary time for fixed arity k.

For example, Löding and Spinrath #OTHEREFR have shown that the infinite clique problem for ω-synchronous co-equivalence relations is decidable in double exponential time.

This yields a double (triple) exponential time algorithm for the ωrecognizability problem for ω-synchronous relations given by (non)deterministic ω-automata.

Another example where the difference between co-equivalence relations and arbitrary relations becomes apparent is the case of tree-automatic relations.

It was proven in #OTHEREFR that the infinite clique problem for treeautomatic relations is EXP-complete; however, restricted to complements of transitive relations the infinite clique problem becomes P-complete.

This yields optimal complexity for the recognizability problem for tree-automatic relations: Recognizability is P-complete for relations given by deterministic bottom-up or top-down tree automata, and EXP-complete for nondeterministic tree automata.

Contributions: We provide improved complexity and new decidability results for the membership problems in subclasses of rational relations over finite and infinite words.

To do so, we refine the existing analyses in #OTHEREFR , #OTHEREFR and identify patterns in the transducers which witness non-membership in the subclass.

Revisiting Membership Problems In Subclasses Of Rational Relations

Abstract: We revisit the membership problem for subclasses of rational relations over finite and infinite words: Given a relation R in a class C2, does R belong to a smaller class C1? The subclasses of rational relations that we consider are formed by the deterministic rational relations, synchronous (also called automatic or regular) relations, and recognizable relations. For almost all versions of the membership problem, determining the precise complexity or even decidability has remained an open problem for almost two decades. In this paper, we provide improved complexity and new decidability results. (i) Testing whether a synchronous relation over infinite words is recognizable is NL-complete (PSPACE-complete) if the relation is given by a deterministic (nondeterministic) ω-automaton. This fully settles the complexity of this recognizability problem, matching the complexity of the same problem over finite words. (ii) Testing whether a deterministic rational binary relation is recognizable is decidable in polynomial time, which improves a previously known double exponential time upper bound. For relations of higher arity, we present a randomized exponential time algorithm. (iii) We provide the first algorithm to decide whether a deterministic rational relation is synchronous. For binary relations the algorithm even runs in polynomial time.

Is Ramsey'S Theorem Omega-Automatic?

Abstract: We study the existence of infinite cliques in omega-automatic (hyper-)graphs. It turns out that the situation is much nicer than in general uncountable graphs, but not as nice as for automatic graphs. More specifically, we show that every uncountable omega-automatic graph contains an uncountable co-context-free clique or anticlique, but not necessarily a context-free (let alone regular) clique or anticlique. We also show that uncountable omega-automatic ternary hypergraphs need not have uncountable cliques or anticliques at all.

arbitrary ω-synchronous relations
infinite clique problem


Unfortunately, they are often neglected by operators due to the difficulties of assessment even though job slowdowns are inevitable #REFR .

Figure 1 shows a set of recurring jobs in Cosmos #OTHEREFR , a Big Data analytics platform at Microsoft, submitted with different frequencies.

We can see that some jobs have more stable runtime while some have occasional slow downs with non-regular patterns.

But, it is not apparent why such variations are happening, how they can be mitigated, or how likely it is for the next job run to be an outlier compared to historic runs.

In production systems, jobs are often scheduled or pipelined with strong data dependencies (jobs using other jobs' output data as inputs) #OTHEREFR .

Stability and predictability of job runtimes are important factors that affect the fundamental design and architecture of data processing pipelines.

Even with massive amounts of telemetry data, cloud providers still default to a manual triage process due to the difficulty of capturing the compounding factors that impact job runtime and its stability, which is not scalable and error-prone.

Although prior works #OTHEREFR have empirically characterized runtime variation, they do not propose methods to predict the variation nor the likelihood of a new run being an outlier compared to the average or median runtimes.

Other works such as Griffon #OTHEREFR used machine learning models to predict the minor slowdown in runtimes for a limited number of job templates.

They are unable to predict significant slowdowns that appear as outliers.

As ML models are notoriously bad at handling outliers especially with a low existence, prior time-series based approaches #OTHEREFR are not applicable.

Runtime Variation In Big Data Analytics

Abstract: The dynamic nature of resource allocation and runtime conditions on Cloud can result in high variability in a job's runtime across multiple iterations, leading to a poor experience. Identifying the sources of such variation and being able to predict and adjust for them is crucial to cloud service providers to design reliable data processing pipelines, provision and allocate resources, adjust pricing services, meet SLOs and debug performance hazards. In this paper, we analyze the runtime variation of millions of production SCOPE jobs on Cosmos, an exabyte-scale internal analytics platform at Microsoft. We propose an innovative 2-step approach to predict job runtime distribution by characterizing typical distribution shapes combined with a classification model with an average accuracy of >96%, out-performing traditional regression models and better capturing long tails. We examine factors such as job plan characteristics and inputs, resource allocation, physical cluster heterogeneity and utilization, and scheduling policies. To the best of our knowledge, this is the first study on predicting categories of runtime distributions for enterprise analytics workloads at scale. Furthermore, we examine how our methods can be used to analyze what-if scenarios, focusing on the impact of resource allocation, scheduling, and physical cluster provisioning decisions on a job's runtime consistency and predictability. CCS Concepts: • Computer systems organization → Cloud computing; • Computing methodologies → Causal reasoning and diagnostics; • Information systems → Data analytics.

Griffon: Reasoning About Job Anomalies With Unlabeled Data In Cloud-Based Platforms

Abstract: Microsoft's internal big data analytics platform is comprised of hundreds of thousands of machines, serving over half a million jobs daily, from thousands of users. The majority of these jobs are recurring and are crucial for the company's operation. Although administrators spend significant effort tuning system performance, some jobs inevitably experience slowdowns, i.e., their execution time degrades over previous runs. Currently, the investigation of such slowdowns is a labor-intensive and error-prone process, which costs Microsoft significant human and machine resources, and negatively impacts several lines of businesses. In this work, we present Griffin, a system we built and have deployed in production last year to automatically discover the root cause of job slowdowns. Existing solutions either rely on labeled data (i.e., resolved incidents with labeled reasons for job slowdowns), which is in most cases non-existent or non-trivial to acquire, or on time-series analysis of individual metrics that do not target specific jobs holistically. In contrast, in Griffin we cast the problem to a corresponding regression one that predicts the runtime of a job, and show how the relative contributions of the features used to train our interpretable model can be exploited to rank the potential causes of job slowdowns. Evaluated over historical incidents, we show that Griffin discovers slowdown causes that are consistent with the ones validated by domain-expert engineers, in a fraction of the time required by them.


A Framework For Describing And Analyzing Fairness: Acrocpolis

In an exposé by ProPublica, the system was found to yield different results based on race, in effect recommending harsher penalties and higher bail payments for Black people than white people #REFR .

Describing the above categories is, in many cases, not sufficient to assess or describe the fairness of a process.

Indeed, fairness assessments often hinge on the relationships between components.

For this reason, Links, denoting such components, are an integral part of the framework.

The next sections detail the different elements of ACROCPoLis.

As a running example we use the well-known COMPAS case, where an AI decision support system ostensibly evaluated the recidivism risk of court defendants based on various demographic data.

In particular, the perspectives of both Northpointe (the company that built the COMPAS system) and ProPublica will be described, as well as the factors these Actors took as most relevant, revealing the differences in the fairness assessments they provided.

Acrocpolis: A Descriptive Framework For Making Sense Of Fairness

Abstract: Fairness is central to the ethical and responsible development and use of AI systems, with a large number of frameworks and formal notions of algorithmic fairness being available. However, many of the fairness solutions proposed revolve around technical considerations and not the needs of and consequences for the most impacted communities. We therefore want to take the focus away from definitions and allow for the inclusion of societal and relational aspects to represent how the effects of AI systems impact and are experienced by individuals and social groups. In this paper, we do this by means of proposing the ACROCPoLis framework to represent allocation processes with a modeling emphasis on fairness aspects. The framework provides a shared vocabulary in which the factors relevant to fairness assessments for different situations and procedures are made explicit, as well as their interrelationships. This enables us to compare analogous situations, to highlight the differences in dissimilar situations, and to capture differing interpretations of the same situation by different stakeholders.

A Survey On Bias And Fairness In Machine Learning

Abstract: With the widespread use of AI systems and applications in our everyday lives, it is important to take fairness issues into consideration while designing and engineering these types of systems. Such systems can be used in many sensitive environments to make important and life-changing decisions; thus, it is crucial to ensure that the decisions do not reflect discriminatory behavior toward certain groups or populations. We have recently seen work in machine learning, natural language processing, and deep learning that addresses such challenges in different subdomains. With the commercialization of these systems, researchers are becoming aware of the biases that these applications can contain and have attempted to address them. In this survey we investigated different real-world applications that have shown biases in various ways, and we listed different sources of biases that can affect AI applications. We then created a taxonomy for fairness definitions that machine learning researchers have defined in order to avoid the existing bias in AI systems. In addition to that, we examined different domains and subdomains in AI showing what researchers have observed with regard to unfair outcomes in the state-of-the-art methods and how they have tried to address them. There are still many future directions and solutions that can be taken to mitigate the problem of bias in AI systems. We are hoping that this survey will motivate researchers to tackle these issues in the near future by observing existing work in their respective fields.


Implications Of The Performance Anomalies Found

A similar observation has been shown in IRN #REFR , but they mainly focus on innetwork behaviors.

For example, many anomalies (e.g., #9 and #12) show that the host limitation can slow down RNIC's outbound rate (dispatching received data to host memory).

This makes the receiver cannot consume packets as fast as the sender sends.

Without end-to-end flow control, the Ro-CEv2 now can only rely on PFC, the hop-by-hop flow control mechanism.

PFC helps to avoid such overflow packet drop but can cause catastrophic consequences #OTHEREFR .

Note that RDMA congestion control #OTHEREFR mainly targets innetwork congestion, so it is orthogonal.

Collie shows that, in addition to switches, the hosts can also generate PFC pause frames, which requires attention when deploying RDMA in production.

Collie: Finding Performance Anomalies In Rdma Subsystems

Abstract: High-speed RDMA networks are getting rapidly adopted in the industry for their low latency and reduced CPU overheads. To verify that RDMA can be used in production, system administrators need to understand the set of application workloads that can potentially trigger abnormal performance behaviors (e.g., unexpected low throughput, PFC pause frame storm). We design and implement Collie, a tool for users to systematically uncover performance anomalies in RDMA subsystems without the need to access hardware internal designs. Instead of individually testing each hardware device (e.g., NIC, memory, PCIe), Collie is holistic, constructing a comprehensive search space for application workloads. Collie then uses simulated annealing to drive RDMA-related performance and diagnostic counters to extreme value regions to find workloads that can trigger performance anomalies. We evaluate Collie on combinations of various RDMA NIC, CPU, and other hardware components. Collie found 15 new performance anomalies. All of them are acknowledged by the hardware vendors. 7 of them are already fixed after we reported them. We also present our experience in using Collie to avoid performance anomalies for an RDMA RPC library and an RDMA distributed machine learning framework.

Revisiting Network Support For Rdma

Abstract: The advent of RoCE (RDMA over Converged Ethernet) has led to a significant increase in the use of RDMA in datacenter networks. To achieve good performance, RoCE requires a lossless network which is in turn achieved by enabling Priority Flow Control (PFC) within the network. However, PFC brings with it a host of problems such as head-of-the-line blocking, congestion spreading, and occasional deadlocks. Rather than seek to fix these issues, we instead ask: is PFC fundamentally required to support RDMA over Ethernet? ::: We show that the need for PFC is an artifact of current RoCE NIC designs rather than a fundamental requirement. We propose an improved RoCE NIC (IRN) design that makes a few simple changes to the RoCE NIC for better handling of packet losses. We show that IRN (without PFC) outperforms RoCE (with PFC) by 6-83% for typical network scenarios. Thus not only does IRN eliminate the need for PFC, it improves performance in the process! We further show that the changes that IRN introduces can be implemented with modest overheads of about 3-10% to NIC resources. Based on our results, we argue that research and industry should rethink the current trajectory of network support for RDMA.


For this regime, our code has a lower redundancy compared to the Θ(log n) redundancy of the code introduced in #REFR .

We focus on the regime where p = k/n α , where k > 1 and α ∈ (0.5, 1] are constants.

We use the deletion-detecting code introduced in Section IV-B (Construction 1) to design a novel code C ⊆ F n 2 for trace reconstruction. The contributions that we present are the following.

• For all α ∈ (0.5, 1], the redundancy of our code is o(n 1−α log n), i.e., grows strictly slower than the optimal redundancy of a deletion-correcting code, which is Θ(n 1−α log n) for d = Θ(np) = Θ(n 1−α ).

Furthermore, for the particular regime where the average number of deletions is fixed, i.e., α = 1 and p = k/n, we show that our code can efficiently reconstruct any codeword x ∈ C with any redundancy that is ω(1).

This regime is similar to the regime where the number of deletions per trace is fixed, which was studied in #OTHEREFR .

• The reconstruction algorithm that we propose runs in linear time O(n).

• We provide a theoretical guarantee on the performance of our code in the form of an upper bound on the probability of reconstruction error.

This bound shows that for all α ∈ (0.5, 1], and for any codeword x ∈ C , the probability of reconstruction error vanishes asymptotically in n for a constant number of traces t = O(1).

• We evaluate the numerical performance of our code in terms of the edit distance error, which was adopted as a performance metric in several recent related works, e.g., #OTHEREFR - #OTHEREFR .

We compare the performance of our code to the code in #OTHEREFR and to a simple coded version of the Bitwise Majority Alignment (BMA) reconstruction algorithm #OTHEREFR .


Impact Of Multiple Transformer Layers On Training Convergence:

There seems to be a contradiction with the conclusion on the generative model focusing on StyleGAN #REFR : the model becomes more robust with more parameters.

One drawback of the vanilla Transformer model in style transfer is that the multi-layer structure can lead to difficult training convergence. As shown in Fig.

16 , with more layers adopted, the loss may converge more slowly, and it even fails in the 5-layer case.

In fact, instead of generating new contents unconditionally in StyleGAN, style transfer aims to preserve contents and migrate style patterns at the same time.

Stacking more layers in Transformer models may increase the complexity of the transfer function and tends to learn more abstract information.

Thus, with more layers, it becomes harder to preserve original content structures during training.

Sharing parameters for different layers kills three birds with one stone: it makes a light-weight, easy-to-train, and easy-to-control model.

Master: Meta Style Transformer For Controllable Zero-Shot And Few-Shot Artistic Style Transfer

Abstract: specific style. Text-guided few-shot style transfer is firstly achieved with the proposed framework. Extensive experiments demonstrate the superiority of Master under both zero-shot and few-shot style transfer settings.

Analyzing And Improving The Image Quality Of Stylegan

Abstract: The style-based GAN architecture (StyleGAN) yields state-of-the-art results in data-driven unconditional generative image modeling. We expose and analyze several of its characteristic artifacts, and propose changes in both model architecture and training methods to address them. In particular, we redesign the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images. In addition to improving image quality, this path length regularizer yields the additional benefit that the generator becomes significantly easier to invert. This makes it possible to reliably attribute a generated image to a particular network. We furthermore visualize how well the generator utilizes its output resolution, and identify a capacity problem, motivating us to train larger models for additional quality improvements. Overall, our improved model redefines the state of the art in unconditional image modeling, both in terms of existing distribution quality metrics as well as perceived image quality.


The similarity between groups is higher for shade than for bark, but the similarity within groups is higher The training and test subsets were randomly selected (using the train_test_split function with test_size=0.5 and random_state=0; sci-kit learn library; #REFR .

The proxigram overlays of bark and shade highlight similar differences in their group structures.

Whereas the proxigram of the contextualized embeddings of shade identifies multiple interconnected groups, the one for bark highlights two separate groups with very few connections between them, supporting previous verbal descriptions of these lexicographic categories (e.g. Rodd, 2020).

To evaluate quantitatively the degree to which the contextual embeddings for bark and shade contain meaningful, disambiguating information in the multidimensional space, we computed the intra-and inter-group cosine similarity of their contextual embeddings (with 'group' defined by translation labels; see Figure 4c ).

This analysis revealed greater similarity within each group than between groups ( = −0.03, < 0.01, = −0.31, = −186.27, < 0.001; = 0.50; ℎ = 0.56).

Moreover, we found a significant interaction between group status and type of ambiguity on the similarity of contextual embeddings ( = −0.01, < 0.01, = −0.08, = −46.4, < 0.001).

The classifier had an accuracy of 94% and 74% in distinguishing the meanings of bark and shade respectively.

To avoid any potential for circularity (e.g.

if the translation algorithm is based on technology similar to BERT), we repeated the classification analysis using ground truth labels derived from the human judgments described above.

We took as ground truth the meanings selected for each context by at least two of our human raters (where this was not 'other') and

Shades Of Meaning 1 Shades Of Meaning: Uncovering The Geometry Of Ambiguous Word Representations Through Contextualised Language Models

Abstract: Lexical ambiguity presents a profound and enduring challenge to the language sciences. Researchers for decades have grappled with the problem of how language users learn, represent and process words with more than one meaning. Our work offers new insight into psychological understanding of lexical ambiguity through a series of simulations that capitalise on recent advances in contextual language models. These models have no grounded understanding of the meanings of words at all; they simply learn to predict words based on the surrounding context provided by other words. Yet, our analyses show that their representations capture fine-grained meaningful distinctions between unambiguous, homonymous, and polysemous words that align with lexicographic classifications and psychological theorising. These findings provide quantitative support for modern psychological conceptualisations of lexical ambiguity and raise new challenges for understanding of the way that contextual information shapes the meanings of words across different timescales.

Scikit-Learn: Machine Learning In Python

Abstract: Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing machine learning to non-specialists using a general-purpose high-level language. Emphasis is put on ease of use, performance, documentation, and API consistency. It has minimal dependencies and is distributed under the simplified BSD license, encouraging its use in both academic and commercial settings. Source code, binaries, and documentation can be downloaded from http://scikit-learn.sourceforge.net.


JointDNN #REFR ] is a new method that allows computation on either platform for each layer independently of the other layers; this may allow one more partition point across the mobile device and cloud.

The DNN is split into parts at the granularity of layers, input data, or sub-layers.

Layer Partition. In general, the DNN is split into two parts.

Neurosurgeon #OTHEREFR , Edgent #OTHEREFR , JALAD #OTHEREFR , and PriPro #OTHEREFR are typical partition methods that split a DNN into two parts.

Some DNN partition approaches #OTHEREFR are similar in that the output of the partition is only two parts.

In addition, the DNN can be subdivided into two more parts at the granularity of layers.

However, compared with the general solutions, the approach in #OTHEREFR is more flexible with regard to the fine-grained DNN computation partitioning mechanism. The number of DNNs influences DNN partition.

The joint optimization of multiple DNN partition and scheduling for mobile cloud #OTHEREFR splits each DNN into two parts at the granularity of each DNN layer and the different partition points.

Moreover, to address the multiple DNN partition problem, a DNN partition strategy with layer partition operations is also considered to be efficient #OTHEREFR .

DNN partition can be employed at the granularity of the input data.

A Survey On Deep Neural Network Partition Over Cloud, Edge And End Devices

Abstract: "Deep neural network (DNN) partition" is a research problem that involves splitting a DNN into multiple parts and offloading them to specific locations. Because of the recent advancement in multi-access edge computing and edge intelligence, DNN partition has been considered as a powerful tool for improving DNN inference performance when the computing resources of edge and end devices are limited and the remote transmission of data from these devices to clouds is costly. This paper provides a comprehensive survey on the recent advances and challenges in DNN partition approaches over the cloud, edge, and end devices based on a detailed literature collection. We review how DNN partition works in various application scenarios, and provide a unified mathematical model of the DNN partition problem. We developed a five-dimensional classification framework for DNN partition approaches, consisting of deployment locations, partition granularity, partition constraints, optimization objectives, and optimization algorithms. Each existing DNN partition approache can be perfectly defined in this framework by instantiating each dimension into specific values. In addition, we suggest a set of metrics for comparing and evaluating the DNN partition approaches. Based on this, we identify and discuss research challenges that have not yet been investigated or fully addressed. We hope that this work helps DNN partition researchers by highlighting significant future research directions in this domain.

Jointdnn: An Efficient Training And Inference Engine For Intelligent Mobile Cloud Computing Services

Abstract: Deep learning models are being deployed in many mobile intelligent applications. End-side services, such as intelligent personal assistants, autonomous cars, and smart home services often employ either simple local models on the mobile or complex remote models on the cloud. However, recent studies have shown that partitioning the DNN computations between the mobile and cloud can increase the latency and energy efficiencies. In this paper, we propose an efficient, adaptive, and practical engine, JointDNN, for collaborative computation between a mobile device and cloud for DNNs in both inference and training phase. JointDNN not only provides an energy and performance efficient method of querying DNNs for the mobile side but also benefits the cloud server by reducing the amount of its workload and communications compared to the cloud-only approach. Given the DNN architecture, we investigate the efficiency of processing some layers on the mobile device and some layers on the cloud server. We provide optimization formulations at layer granularity for forward- and backward-propagations in DNNs, which can adapt to mobile battery limitations and cloud server load constraints and quality of service. JointDNN achieves up to 18 and 32 times reductions on the latency and mobile energy consumption of querying DNNs compared to the status-quo approaches, respectively.


We also compare our results with an in-memory NVM accelerator #REFR where the M V M Dynamic operations are executed in the temporal 1-D SIMD lanes instead of the NVM tiles due to limited endurance.

With the proposed sequence blocking (SB=64) dataflow.

First, without sequence blocking, we observe upto 10x benefits on average for both GLUE and SQuAD benchmarks over GPU.

This is because X-Former is a highly parallel spatial in-memory architecture.

Moreover, we also observe improvements compared to a traditional NVM in-memory accelerator since the Attention Engine is more efficient in processing the MVM operation compared to the 1-D SIMD module (vector-vector operations).

Next, we notice additional improvements with sequence blocking, upto 85x and 81x for GLUE and SQuAD benchmarks respectively because the Attention Engine is inactive only until the Projection Engine processes a subset of the input instead of the entire sequence length.

Therefore, both the engines work efficiently with higher utilization, resulting in better throughput.

X-Former: In-Memory Acceleration Of Transformers

Abstract: Transformers have achieved great success in a wide variety of natural language processing (NLP) tasks due to the attention mechanism, which assigns an importance score for every word relative to other words in a sequence. However, these models are very large, often reaching hundreds of billions of parameters, and therefore require a large number of DRAM accesses. Hence, traditional deep neural network (DNN) accelerators such as GPUs and TPUs face limitations in processing Transformers efficiently. In-memory accelerators based on non-volatile memory promise to be an effective solution to this challenge, since they provide high storage density while performing massively parallel matrix vector multiplications within memory arrays. However, attention score computations, which are frequently used in Transformers (unlike CNNs and RNNs), require matrix vector multiplications (MVM) where both operands change dynamically for each input. As a result, conventional NVM-based accelerators incur high write latency and write energy when used for Transformers, and further suffer from the low endurance of most NVM technologies. To address these challenges, we present X-Former, a hybrid in-memory hardware accelerator that consists of both NVM and CMOS processing elements to execute transformer workloads efficiently. To improve the hardware utilization of X-Former, we also propose a sequence blocking dataflow, which overlaps the computations of the two processing elements and reduces execution time. Across several benchmarks, we show that X-Former achieves upto 85x and 7.5x improvements in latency and energy over a NVIDIA GeForce GTX 1060 GPU and upto 10.7x and 4.6x improvements in latency and energy over a state-of-theart in-memory NVM accelerator.

Puma: A Programmable Ultra-Efficient Memristor-Based Accelerator For Machine Learning Inference

Abstract: Memristor crossbars are circuits capable of performing analog matrix-vector multiplications, overcoming the fundamental energy efficiency limitations of digital logic. They have been shown to be effective in special-purpose accelerators for a limited set of neural network applications. We present the Programmable Ultra-efficient Memristor-based Accelerator (PUMA) which enhances memristor crossbars with general purpose execution units to enable the acceleration of a wide variety of Machine Learning (ML) inference workloads. PUMA's microarchitecture techniques exposed through a specialized Instruction Set Architecture (ISA) retain the efficiency of in-memory computing and analog circuitry, without compromising programmability. We also present the PUMA compiler which translates high-level code to PUMA ISA. The compiler partitions the computational graph and optimizes instruction scheduling and register allocation to generate code for large and complex workloads to run on thousands of spatial cores. We have developed a detailed architecture simulator that incorporates the functionality, timing, and power models of PUMA's components to evaluate performance and energy consumption. A PUMA accelerator running at 1 GHz can reach area and power efficiency of 577 GOPS/s/mm 2 and 837~GOPS/s/W, respectively. Our evaluation of diverse ML applications from image recognition, machine translation, and language modelling (5M-800M synapses) shows that PUMA achieves up to 2,446× energy and 66× latency improvement for inference compared to state-of-the-art GPUs. Compared to an application-specific memristor-based accelerator, PUMA incurs small energy overheads at similar inference latency and added programmability.


Finally and anecdotically, the NGD method can also be viewed as an approximate Newton method since the FIM and the GGN matrix are equivalent when the model predictive distribution P y|x (θ) belongs to exponential family distributions #REFR .

This is useful in high-dimensional cases where the choice of parameters is more or less arbitrary.

Strictly speaking, invariance with respect to parameters only occurs at the continuous level, i.e., in the limit of α k → 0.

This minor drawback does not undermine the theoretical soundness of the method.

In fact, the real drawback of the NGD (2.15)-(2.16) lies in the cost of computing and inverting the Fisher matrix.

This is why it is capital to consider suitable approximations of the FIM such as KFAC (cf. §2.3).

Analysis And Comparison Of Two-Level Kfac Methods For Training Deep Neural Networks

Abstract: As a second-order method, the Natural Gradient Descent (NGD) has the ability to accelerate training of neural networks. However, due to the prohibitive computational and memory costs of computing and inverting the Fisher Information Matrix (FIM), efficient approximations are necessary to make NGD scalable to Deep Neural Networks (DNNs). Many such approximations have been attempted. The most sophisticated of these is KFAC, which approximates the FIM as a block-diagonal matrix, where each block corresponds to a layer of the neural network. By doing so, KFAC ignores the interactions between different layers. In this work, we investigate the interest of restoring some low-frequency interactions between the layers by means of two-level methods. Inspired from domain decomposition, several two-level corrections to KFAC using different coarse spaces are proposed and assessed. The obtained results show that incorporating the layer interactions in this fashion does not really improve the performance of KFAC. This suggests that it is safe to discard the off-diagonal blocks of the FIM, since the block-diagonal approach is sufficiently robust, accurate and economical in computation time.

New Insights And Perspectives On The Natural Gradient Method

Abstract: Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent. In this paper we critically analyze this method and its properties, and show how it can be viewed as a type of approximate 2nd-order optimization method, where the Fisher information matrix can be viewed as an approximation of the Hessian. This perspective turns out to have significant implications for how to design a practical and robust version of the method. Additionally, we make the following contributions to the understanding of natural gradient and 2nd-order methods: a thorough analysis of the convergence speed of stochastic natural gradient descent (and more general stochastic 2nd-order methods) as applied to convex quadratics, a critical examination of the oft-used "empirical" approximation of the Fisher matrix, and an analysis of the (approximate) parameterization invariance property possessed by natural gradient methods, which we show still holds for certain choices of the curvature matrix other than the Fisher, but notably not the Hessian.


As revealed by #REFR , the optimal value of reflecting attenuation is β sn = 1, ∀s, n to maximize the received power and simplify the hardware implementation.

In contrast to hand-held UE, a surface mounted on a vehicle is generally high enough to get the line-of-sight (LOS) path of the BS without any blockage, resulting in Rician fading, i.e.,

with the Rician factor Γ, the LOS component H LOS , the multipath component H N LOS consisting of independent entries that follow CN (0, 1), and the BS-IRVS path loss σ 2 s , s ∈ S.

Each reflecting surface is equipped with a smart controller, which can adaptively adjust the phase shift of each element in terms of the CSI acquired through periodic channel estimation #OTHEREFR .

We write c sn = β sn e jφsn to denote the reflection coefficient of the n th element of the s th surface, with the induced phase shift φ sn ∈ [0, 2π) and the amplitude attenuation β sn ∈ [0, 1].

Because of severe path loss, the signals that are reflected by the IRVS twice or more are negligible.

By ignoring impairments such as channel aging #OTHEREFR and phase noise #OTHEREFR , the k th UE observes the received signal

where x denotes the vector of transmitted signals over the BS antenna array, P d expresses the power constraint of the BS, n k is additive white Gaussian noise (AWGN) with zero mean and variance σ 2 n , namely n k ∼ CN (0, σ 2 n ).

Define the diagonal phase-shift matrix for surface s as Φ s = diag e jφs1 , e jφs2 , . . . , e jφ sNs , ∀s ∈ S, Fig. 2 . The structure of a TDMA frame.

Intelligent Reflecting Vehicle Surface: A Novel Irs Paradigm For Moving Vehicular Networks

Abstract: Intelligent reflecting surface (IRS) has recently received much attention from the research community due to its potential to achieve high spectral and power efficiency costeffectively. In addition to traditional cellular networks, the use of IRS in vehicular networks is also considered. Prior works on IRSaided vehicle-to-everything communications focus on deploying reflection surfaces on the facades of buildings along the road for sidelink performance enhancement. This paper goes beyond the state of the art by presenting a novel paradigm coined Intelligent Reflecting Vehicle Surface (IRVS). It embeds a massive number of reflection elements on vehicles' surfaces to aid moving vehicular networks in military and emergency communications. We propose an alternative optimization method to optimize jointly active beamforming at the base station and passive reflection across multiple randomly-distributed vehicle surfaces. Performance evaluation in terms of sum spectral efficiency under continuous, discrete, and random phase shifts is conducted. Numerical results reveal that IRVS can substantially improve the capacity of a moving vehicular network.

Intelligent Reflecting Surface Enhanced Wireless Network Via Joint Active And Passive Beamforming

Abstract: Intelligent reflecting surface (IRS) is a revolutionary and transformative technology for achieving spectrum and energy efficient wireless communication cost-effectively in the future. Specifically, an IRS consists of a large number of low-cost passive elements each being able to reflect the incident signal independently with an adjustable phase shift so as to collaboratively achieve three-dimensional (3D) passive beamforming without the need of any transmit radio-frequency (RF) chains. In this paper, we study an IRS-aided single-cell wireless system where one IRS is deployed to assist in the communications between a multi-antenna access point (AP) and multiple single-antenna users. We formulate and solve new problems to minimize the total transmit power at the AP by jointly optimizing the transmit beamforming by active antenna array at the AP and reflect beamforming by passive phase shifters at the IRS, subject to users’ individual signal-to-interference-plus-noise ratio (SINR) constraints. Moreover, we analyze the asymptotic performance of IRS’s passive beamforming with infinitely large number of reflecting elements and compare it to that of the traditional active beamforming/relaying. Simulation results demonstrate that an IRS-aided MIMO system can achieve the same rate performance as a benchmark massive MIMO system without using IRS, but with significantly reduced active antennas/RF chains. We also draw useful insights into optimally deploying IRS in future wireless systems.


However, recent advances in GANs architecture and training techniques have shown promise in addressing these issues and improving the stability and performance of GANs #REFR .

In other words, each neural network tries to minimize its cost.

In practice, GANs is commonly trained using gradient-based optimization on and , alternatively.

Despite GANs exhibiting the capacity to produce realistic and high-quality images, GANs is notoriously difficult to train due to the adversarial nature of the training process.

GANs requires and to compete with each other, which can lead to "mode collapse", i.e.

finds a way to generate only a limited ranges of outputs to mislead . GANs is also sensitive to hyperparameters.

Generative process using a decoder from simple trained autoencoders (AEs) is extremely difficult since its latent space is normally not "well-organized".

Contrary to encoder in AEs (encoding an input to a single point in the latent space), VAEs [Kingma and Welling 2013] encodes an input as a distribution.

VAEs is then trained along with a regularization term (Eq. 2), i.e. Kullback-Leibler (KL) divergence, to provide exploitable latent space.

Formally, given x representing the input data, and z representing the compressed latent space representation.

Text-Guided Image-And-Shape Editing And Generation: A Short Survey

Abstract: Image and shape editing are ubiquitous among digital artworks. Graphics algorithms facilitate artists and designers to achieve desired editing intents without going through manually tedious retouching. In the recent advance of machine learning, artists' editing intents can even be driven by text, using a variety of well-trained neural networks. They have seen to be receiving an extensive success on such as generating photorealistic images, artworks and human poses, stylizing meshes from text, or auto-completion given image and shape priors. In this short survey, we provide a overview over 50 papers on state-of-the-art (text-guided) image-and-shape generation techniques. We start with an overview on recent editing algorithms in the introduction. Then, we provide a comprehensive review on text-guided editing techniques for 2D and 3D independently, where each of its subsection begins with a brief background introduction. We also contextualize editing algorithms under recent implicit neural representations. Finally, we conclude the survey with the discussion over existing methods and potential research ideas.

Progressive Growing Of Gans For Improved Quality, Stability, And Variation

Abstract: We describe a new training methodology for generative adversarial networks. The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses. This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CelebA images at 1024^2. We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10. Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator. Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation. As an additional contribution, we construct a higher-quality version of the CelebA dataset.