Similar results are then achieved in #REFR for deep fully-connected networks as all hidden layer widths go to infinity.

#OTHEREFR gives a constructive method, but only for target and activation functions in L 1 .

In #OTHEREFR and #OTHEREFR , they propose constructive methods for a class of target functions with unit step and ReLU activations respectively.

In #OTHEREFR , functions are approximated using trigonometric polynomial ridge functions, which can then be shown in expectation to be equivalent to randomly initialized ReLU activations.

There are several interesting and important results in the literature having to do with neural networks with random (typically Gaussian) parameter initialization, sometimes as a consequence of using randomly initialized gradient descent for training the network.

A classic result by #OTHEREFR shows that the output of a single hidden-layer network with Gaussian randomly initialized parameters goes to a Gaussian Process as the width goes to infinity.

Also for deep fully-connected networks, in #OTHEREFR the authors define the Neural Tangent Kernel and propose that its limit, as the hidden layer widths go to infinity, can be used to study the timestep evolution and dynamics of the parameters, and the corresponding network output function, in gradient descent.

In #OTHEREFR , the authors show that single hidden-layer networks cannot achieve the same rates of increase in a measure of curvature produced by the network output, as deep networks can, with parameters Gaussian randomly initialized and bounded activation functions.

In #OTHEREFR , the authors show that single hidden-layer networks of a sufficient width can use Gaussian randomly initialized gradient descent on values of a target function, and achieve guaranteed generalization to the entire function.

And in #OTHEREFR , the authors show that deep fully-connected networks, where each hidden layer width meets a sufficient size, can be trained with Gaussian randomly initialized gradient descent and be guaranteed to reach the global minimum at a linear rate.

Our primary contributions in this paper are: developing a novel method for bridging convex combinations of activation functions (eg.

Function Approximation With Randomly Initialized Neural Networks For Approximate Model Reference Adaptive Control

Abstract: Classical results in neural network approximation theory show how arbitrary continuous functions can be approximated by networks with a single hidden layer, under mild assumptions on the activation function. However, the classical theory does not give a constructive means to generate the network parameters that achieve a desired accuracy. Recent results have demonstrated that for specialized activation functions, such as ReLUs, high accuracy can be achieved via linear combinations of randomly initialized activations. These recent works utilize specialized integral representations of target functions that depend on the specific activation functions used. This paper defines mollified integral representations, which provide a means to form integral representations of target functions using activations for which no direct integral representation is currently known. The new construction enables approximation guarantees for randomly initialized networks using any activation for which there exists an established base approximation which may not be constructive. We extend the results to the supremum norm and show how this enables application to an extended, approximate version of (linear) model reference adaptive control.

Deep Neural Networks As Gaussian Processes

Abstract: A deep fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP) in the limit of infinite network width. This correspondence enables exact Bayesian inference for neural networks on regression tasks by means of straightforward matrix computations. For single hidden-layer networks, the covariance function of this GP has long been known. Recently, kernel functions for multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified the correspondence between using these kernels as the covariance function for a GP and performing fully Bayesian prediction with a deep neural network. In this work, we derive this correspondence and develop a computationally efficient pipeline to compute the covariance functions. We then use the resulting GP to perform Bayesian inference for deep neural networks on MNIST and CIFAR-10. We find that the GP-based predictions are competitive and can outperform neural networks trained with stochastic gradient descent. We observe that the trained neural network accuracy approaches that of the corresponding GP-based computation with increasing layer width, and that the GP uncertainty is strongly correlated with prediction error. We connect our observations to the recent development of signal propagation in random neural networks.


Per pixel and per instance evaluation for the detection of small hyperreflective specks and foci in comparison to pixel-wise results for larger conventional hyperreflective foci reported by Schlegl et al. #REFR for data acquired on a Cirrus/Spectralis OCT scanner.

However, on their figures, they appear to be much larger, corresponding to conventional HRF.

In addition, they also include features in different retinal layers.

Due to the difference in sizes and scale, the performance of their algorithm is not directly comparable to our results, yet their results show the performance of a different algorithm on a similar task.

A more detailed comparison of our algorithm and the ground truth yields the following observations (Fig. 2) .

Labeled features with slightly higher intensity values than the background are difficult to detect and the reader agreement on these is not high. Tab. 1.

This also applies to boundary regions of the features because of the smooth transition to background intensities and a missing consensus of where to delineate feature boundaries.

Another situation of disagreement can occur on much higher intensity values on the ELM compared to the rest of the ELM.

They are usually detected by the algorithm, but usually not annotated by the human experts.

In addition, for features that are close to IS/OS and OPL, it is sometimes unclear, if they are detached from these layers.

It occurs that only either the algorithm or the readers include or exclude them, which has a high impact on the scores as the regions are comparably large.

Unsupervised Detection Of Small Hyperreflective Features In Ultrahigh Resolution Optical Coherence Tomography

Abstract: Recent advances in optical coherence tomography such as the development of high speed ultrahigh resolution scanners and corresponding signal processing techniques may reveal new potential biomarkers in retinal diseases. Newly visible features are, for example, small hyperreflective specks in age-related macular degeneration. Identifying these new markers is crucial to investigate potential association with disease progression and treatment outcomes. Therefore, it is necessary to reliably detect these features in 3D volumetric scans. Because manual labeling of entire volumes is infeasible a need for automatic detection arises. Labeled datasets are often not publicly available and there are usually large variations in scan protocols and scanner types. Thus, this work focuses on an unsupervised approach that is based on local peak-detection and random walker segmentation to detect small features on each B-scan of the volume.

Fully Automated Segmentation Of Hyperreflective Foci In Optical Coherence Tomography Images

Abstract: The automatic detection of disease related entities in retinal imaging data is relevant for disease- and treatment monitoring. It enables the quantitative assessment of large amounts of data and the corresponding study of disease characteristics. The presence of hyperreflective foci (HRF) is related to disease progression in various retinal diseases. Manual identification of HRF in spectral-domain optical coherence tomography (SD-OCT) scans is error-prone and tedious. We present a fully automated machine learning approach for segmenting HRF in SD-OCT scans. Evaluation on annotated OCT images of the retina demonstrates that a residual U-Net allows to segment HRF with high accuracy. As our dataset comprised data from different retinal diseases including age-related macular degeneration, diabetic macular edema and retinal vein occlusion, the algorithm can safely be applied in all of them though different pathophysiological origins are known.


Three-Dimensional Convolutional Neural Network (Cnn) Architecture

Our approach is adopted from #REFR , whereby a 2D × kernel can be inflated to a 3D × × kernel that spreads across frames.

Resnet18 was used as the backbone of our CNN model.

It serves as our feature extractor which compresses the image representation into a vector with dimensionality of 512 as shown in Fig. 5 .

The base architecture, which was designed for image classification was extended to a three-dimensional (3D) model by "inflating" the kernels #OTHEREFR .

The weights from the pretrained 2D Resnet 18 model are used to initialize the kernels, whereby each of the planes in the × × kernel are initialized with the pre-trained × weights, rescaled by 1/ .

This initialization setup produces the same results as the 2D pre-trained model run on a single static frame, repeated in the time domain.

In our setup, the first convolutional kernel in each Resnet layer is initialized with a 3 × 3 × 3 kernel.

The rest of the kernels are simply extended to their 3D counterpart that perform the same operation, by initializing them as a 1 × 3 × 3 kernel.

This approach does not increase the parameters significantly, thus alleviating overfitting. The CNN architecture is visualized in Fig. 2 .

Radifusion: A Multi-Radiomics Deep Learning Based Breast Cancer Risk Prediction Model Using Sequential Mammographic Images With Image Attention And Bilateral Asymmetry Refinement

Abstract: Breast cancer is a significant public health concern and early detection is critical for triaging high risk patients. Sequential screening mammograms can provide important spatiotemporal information about changes in breast tissue over time. In this study, we propose a deep learning architecture called RADIFUSION that utilizes sequential mammograms and incorporates a linear image attention mechanism, radiomic features, a new gating mechanism to combine different mammographic views, and bilateral asymmetry-based finetuning for breast cancer risk assessment. We evaluate our model on a screening dataset called Cohort of Screen-Aged Women (CSAW) dataset. Based on results obtained on the independent testing set consisting of 1,749 women, our approach achieved superior performance compared to other state-of-the-art models with area under the receiver operating characteristic curves (AUCs) of 0.905, 0.872 and 0.866 in the three respective metrics of 1-year AUC, 2-year AUC and > 2-year AUC. Our study highlights the importance of incorporating various deep learning mechanisms, such as image attention, radiomic features, gating mechanism, and bilateral asymmetry-based fine-tuning, to improve the accuracy of breast cancer risk assessment. We also demonstrate that our model's performance was enhanced by leveraging spatiotemporal information from sequential mammograms. Our findings suggest that RADIFUSION can provide clinicians with a powerful tool for breast cancer risk assessment.

Non-Local Neural Networks

Abstract: Both convolutional and recurrent operations are building blocks that process one local neighborhood at a time. In this paper, we present non-local operations as a generic family of building blocks for capturing long-range dependencies. Inspired by the classical non-local means method in computer vision, our non-local operation computes the response at a position as a weighted sum of the features at all positions. This building block can be plugged into many computer vision architectures. On the task of video classification, even without any bells and whistles, our non-local models can compete or outperform current competition winners on both Kinetics and Charades datasets. In static image recognition, our non-local models improve object detection/segmentation and pose estimation on the COCO suite of tasks. Code is available at this https URL .


Other works such as Griffon #REFR used machine learning models to predict the minor slowdown in runtimes for a limited number of job templates.

In production systems, jobs are often scheduled or pipelined with strong data dependencies (jobs using other jobs' output data as inputs) #OTHEREFR .

Stability and predictability of job runtimes are important factors that affect the fundamental design and architecture of data processing pipelines.

Unfortunately, they are often neglected by operators due to the difficulties of assessment even though job slowdowns are inevitable #OTHEREFR .

Even with massive amounts of telemetry data, cloud providers still default to a manual triage process due to the difficulty of capturing the compounding factors that impact job runtime and its stability, which is not scalable and error-prone.

Although prior works #OTHEREFR have empirically characterized runtime variation, they do not propose methods to predict the variation nor the likelihood of a new run being an outlier compared to the average or median runtimes.

They are unable to predict significant slowdowns that appear as outliers.

As ML models are notoriously bad at handling outliers especially with a low existence, prior time-series based approaches #OTHEREFR are not applicable.

In this paper, we aim to address this gap for production data analytics systems by developing a novel and systematic approach for modeling, predicting, and explaining the job runtime variation, allowing for finer-grained differentiation in characteristics.

For our study, we comprehensively examine the runtime variation for millions of production SCOPE #OTHEREFR jobs on Cosmos #OTHEREFR , an exabyte-scale analytics platform at Microsoft that supports a broad spectrum of Microsoft products #OTHEREFR .

Our key contribution is a framework for systematically analyzing, predicting and explaining runtime variation that includes:

Runtime Variation In Big Data Analytics

Abstract: The dynamic nature of resource allocation and runtime conditions on Cloud can result in high variability in a job's runtime across multiple iterations, leading to a poor experience. Identifying the sources of such variation and being able to predict and adjust for them is crucial to cloud service providers to design reliable data processing pipelines, provision and allocate resources, adjust pricing services, meet SLOs and debug performance hazards. In this paper, we analyze the runtime variation of millions of production SCOPE jobs on Cosmos, an exabyte-scale internal analytics platform at Microsoft. We propose an innovative 2-step approach to predict job runtime distribution by characterizing typical distribution shapes combined with a classification model with an average accuracy of >96%, out-performing traditional regression models and better capturing long tails. We examine factors such as job plan characteristics and inputs, resource allocation, physical cluster heterogeneity and utilization, and scheduling policies. To the best of our knowledge, this is the first study on predicting categories of runtime distributions for enterprise analytics workloads at scale. Furthermore, we examine how our methods can be used to analyze what-if scenarios, focusing on the impact of resource allocation, scheduling, and physical cluster provisioning decisions on a job's runtime consistency and predictability. CCS Concepts: • Computer systems organization → Cloud computing; • Computing methodologies → Causal reasoning and diagnostics; • Information systems → Data analytics.

Griffon: Reasoning About Job Anomalies With Unlabeled Data In Cloud-Based Platforms

Abstract: Microsoft's internal big data analytics platform is comprised of hundreds of thousands of machines, serving over half a million jobs daily, from thousands of users. The majority of these jobs are recurring and are crucial for the company's operation. Although administrators spend significant effort tuning system performance, some jobs inevitably experience slowdowns, i.e., their execution time degrades over previous runs. Currently, the investigation of such slowdowns is a labor-intensive and error-prone process, which costs Microsoft significant human and machine resources, and negatively impacts several lines of businesses. In this work, we present Griffin, a system we built and have deployed in production last year to automatically discover the root cause of job slowdowns. Existing solutions either rely on labeled data (i.e., resolved incidents with labeled reasons for job slowdowns), which is in most cases non-existent or non-trivial to acquire, or on time-series analysis of individual metrics that do not target specific jobs holistically. In contrast, in Griffin we cast the problem to a corresponding regression one that predicts the runtime of a job, and show how the relative contributions of the features used to train our interpretable model can be exploited to rank the potential causes of job slowdowns. Evaluated over historical incidents, we show that Griffin discovers slowdown causes that are consistent with the ones validated by domain-expert engineers, in a fraction of the time required by them.


We contrast this with the work of Kiefer & Neuen #REFR Theorem 6 .4], who showed that k-WL identifies graphs of treewidth k, though they did not control for rounds.

We showed that the (3k + 4)-WL identifies graphs of treewidth k in O(log n) rounds, improving upon the work of Grohe & Verbitsky #OTHEREFR , who established the analogous result for (4k + 3)-WL.

As a corollary, we obtained that graphs of treewidth k are identified by FO + C formulas with (3k + 5)-variables and quantifier depth O(log n).

Naturally, it would be of interest to close the gap between our upper bound of (3k + 4) on the Weisfeiler-Leman dimension required to achieve O(log n) rounds and the best known upper bound of k on the Weisfeiler-Leman dimension (without controlling for rounds) of graphs of treewidth k.

One approach would be to improve the result on #OTHEREFR , to provide a tree decomposition of width ≤ 3k + 2 and height O(log n) for graphs of treewidth k.

It would also be of interest to examine special families of graphs with bounded treewidth that possess additional structure, which Weisfeiler-Leman can exploit to achieve O(log n) rounds with only k + O(1) pebbles.

Kiefer & Neuen #OTHEREFR Theorem 6 .1] also established a lower bound of ⌈k/2⌉ − 2 on the Weisfeiler-Leman dimension (again, without controlling for rounds) of graphs of treewidth k.

It would be of interest to strengthen their lower bound on the Weisfeiler-Leman dimension when restricting to O(log n) rounds.

Logarithmic Weisfeiler-Leman And Treewidth *

Abstract: In this paper, we show that the (3k + 4)-dimensional Weisfeiler-Leman algorithm can identify graphs of treewidth k in O(log n) rounds. This improves the result of Grohe & Verbitsky (ICALP 2006), who previously established the analogous result for (4k + 3)-dimensional Weisfeiler-Leman. In light of the equivalence between Weisfeiler-Leman and the logic FO + C (Cai, Fürer, & Immerman, Combinatorica 1992), we obtain an improvement in the descriptive complexity for graphs of treewidth k. Precisely, if G is a graph of treewidth k, then there exists a (3k + 5)-variable formula ϕ in FO + C with quantifier depth O(log n) that identifies G up to isomorphism.

The Power Of The Weisfeiler-Leman Algorithm To Decompose Graphs

Abstract: The Weisfeiler-Leman procedure is a widely-used approach for graph isomorphism testing that works by iteratively computing an isomorphism-invariant coloring of vertex tuples. Meanwhile, a fundamental tool in structural graph theory, which is often exploited in approaches to tackle the graph isomorphism problem, is the decomposition into 2- and 3-connected components. ::: We prove that the 2-dimensional Weisfeiler-Leman algorithm implicitly computes the decomposition of a graph into its 3-connected components. Thus, the dimension of the algorithm needed to distinguish two given graphs is at most the dimension required to distinguish the corresponding decompositions into 3-connected components (assuming it is at least 2). ::: This result implies that for k >= 2, the k-dimensional algorithm distinguishes k-separators, i.e., k-tuples of vertices that separate the graph, from other vertex k-tuples. As a byproduct, we also obtain insights about the connectivity of constituent graphs of association schemes. ::: In an application of the results, we show the new upper bound of k on the Weisfeiler-Leman dimension of graphs of treewidth at most k. Using a construction by Cai, Furer, and Immerman, we also provide a new lower bound that is asymptotically tight up to a factor of 2.


For question and answer features, we pre-trained BERT #REFR on our generated training set and extracted QA features adhering to the NExT-QA setting.

9 OpenAI text-davinci-003 API https://openai.com/api/ #OTHEREFR Our empirical findings indicate that adjusting the max len and top k parameters in the prompt for GPT-3 yields favorable results.

As such, we utilize the default max len and top k settings and use prompts in the format of "what is the intention of {Cap}? Provide {top k} answers within vide examples of few-shot prompting by randomly sampling 5 QAs from NExT-QA and transferring the question to the declared sentence.

For the distillation experiments (Section 3.4), we distill from GPT-3 to T5-large 11 model.

Video QA training In all of our experiments, we followed the NExT-QA #OTHEREFR video preprocessing method, where we uniformly sampled eight segments of 16 consecutive frames.

For visual features, we used Resnet101 #OTHEREFR pre-trained on ImageNet #OTHEREFR and inflated 3D ResNeXt-101 #OTHEREFR pre-trained on Kinetics #OTHEREFR as our feature extractors.

To adapt an open-ended QA model for multiple-choice QA, we concatenated each candidate answer with the question and optimized the model with Hinge Loss, following the NExT-QA implementation.

For video QA training, we employ the default NeXT-QA implementation #OTHEREFR with the exception of setting the patience in the ReduceLROnPlateau to 2 instead of 5, and the maximum number of epochs to 25 instead of 50, as we observed a faster convergence during training.

We conduct the training on a single NVIDIA TITAN RTX GPU, and each experiment takes around 18 to 24 hours at most.

Language Models Are Causal Knowledge Extractors For Zero-Shot Video Question Answering

Abstract: Causal Video Question Answering (CVidQA) queries not only association or temporal relations but also causal relations in a video. Existing question synthesis methods pretrained question generation (QG) systems on reading comprehension datasets with text descriptions as inputs. However, QG models only learn to ask association questions (e.g., "what is someone doing...") and result in inferior performance due to the poor transfer of association knowledge to CVidQA, which focuses on causal questions like "why is someone doing ...". Observing this, we proposed to exploit causal knowledge to generate question-answer pairs, and proposed a novel framework, Causal Knowledge Extraction from Language Models (CaKE-LM), leveraging causal commonsense knowledge from language models to tackle CVidQA. To extract knowledge from LMs, CaKE-LM generates causal questions containing two events with one triggering another (e.g., "score a goal" triggers "soccer player kicking ball") by prompting LM with the action (soccer player kicking ball) to retrieve the intention (to score a goal). CaKE-LM significantly outperforms conventional methods by 4% to 6% of zero-shot CVidQA accuracy on NExT-QA and Causal-VidQA datasets. We also conduct comprehensive analyses and provide key findings for future research.

Bert: Pre-Training Of Deep Bidirectional Transformers For Language Understanding

Abstract: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).


Autoencoding models are trained similarly, except that the masked word can be located anywhere in the randomly sampled suite of words #REFR .

From the point of view of a Transformer-based LLM, generating new text is equivalent to the generation of a translation, except that there is not necessarily an embedding space representation.

Instead, it's the continuation of an initial word sequence, where the word to be generated can be located either at the end of the original sentence or in the middle of it.

These two cases correspond to the two main approaches to training Generative Deep Neural Language Models.

Autoregressive models are trained by a random sampling suite of words in the source dataset ("utterances"), masking a word and training the model to predict the masked word accurately #OTHEREFR .

Autoregressive models are best thought of as autocomplete -they are trained to complete sentences in a way that is representative of their training dataset.

Autoencoding models are best thought of as search or autocorrect suggestions -they are trained to rewrite whole sentences in a way that is representative of their training dataset.

Just as Google search suggestions, they can suggest words at the end of a query to refine it, but they can also add one at the beginning or even rewrite a word (for instance, to correct a typo).

While Autoencoding models can be used to generate text based on utterances, their strength is rather in applications that require understanding the utterance as a whole.

While the autoencoding models are generally considered to be more powerful than autoregressive ones, their generative capabilities are not necessarily optimal for the size of the model, training dataset, or the computational resources spent training the model.

After all, the training mode relevant to the generation represents only a fraction of the training time of autoregressive models.

Fundamentals Of Generative Large Language Models And Perspectives In Cyber-Defense

Abstract: Generative Language Models gained significant attention in late 2022 / early 2023, notably with the introduction of models refined to act consistently with users' expectations of interactions with AI (conversational models). Arguably the focal point of public attention has been such a refinement of the GPT3 model-the ChatGPT and its subsequent integration with auxiliary capabilities, including search as part of Microsoft Bing. Despite extensive prior research invested in their development, their performance and applicability to a range of daily tasks remained unclear and niche. However, their wider utilization without a requirement for technical expertise, made in large part possible through conversational fine-tuning, revealed the extent of their true capabilities in a real-world environment. This has garnered both public excitement for their potential applications and concerns about their capabilities and potential malicious uses. This review aims to provide a brief overview of the history, state of the art, and implications of Generative Language Models in terms of their principles, abilities, limitations, and future prospects-especially in the context of cyber-defense, with a focus on the Swiss operational environment.

Generating Sentences From A Continuous Space

Abstract: The standard recurrent neural network language model (RNNLM) generates sentences one word at a time and does not work from an explicit global sentence representation. In this work, we introduce and study an RNN-based variational autoencoder generative model that incorporates distributed latent representations of entire sentences. This factorization allows it to explicitly model holistic properties of sentences such as style, topic, and high-level syntactic features. Samples from the prior over these sentence representations remarkably produce diverse and well-formed sentences through simple deterministic decoding. By examining paths through this latent space, we are able to generate coherent novel sentences that interpolate between known sentences. We present techniques for solving the difficult learning problem presented by this model, demonstrate its effectiveness in imputing missing words, explore many interesting properties of the model's latent sentence space, and present negative results on the use of the model in language modeling.


Because the entropy of unknown samples is usually larger than known ones', we could set a pre-defined threshold ρ and recognize those samples whose entropy is larger than ρ as unknown. Following #REFR , ρ is set to be ln(K)/2.

The key challenge of open set domain adaptation is how to separate common samples from private samples in the target domain.

Assuming the model has more confidence (lower entropy) in shared-class samples than unknown samples, one mainstream method in OSDA is to set a confidence threshold on the entropy of the output of the classifier.

However, the threshold leads the model to be sensitive to hyperparameter tuning that may influence the robustness of the model in real world scenarios.

We tackle this problem by proposing the movingthreshold estimation method.

One important drawback of the previous methods is that they ignore the semantic knowledge within unknown samples.

In fact, there are various kinds of unknown images in an open world that could become unknown target samples.

As a result, the relationships between unknown classes and known classes can vary greatly.

Imbalanced Open Set Domain Adaptation Via Moving-Threshold Estimation And Gradual Alignment

Abstract: Multimedia applications are often associated with cross-domain knowledge transfer, where Unsupervised Domain Adaptation (UDA) can be used to reduce the domain shifts. Open Set Domain Adaptation (OSDA) aims to transfer knowledge from a well-labeled source domain to an unlabeled target domain under the assumption that the target domain contains unknown classes. Existing OSDA methods consistently lay stress on the covariate shift, ignoring the potential label shift problem. The performance of OSDA methods degrades drastically under intradomain class imbalance and inter-domain label shift. However, little attention has been paid to this issue in the community. In this paper, the Imbalanced Open Set Domain Adaptation (IOSDA) is explored where the covariate shift, label shift and category mismatch exist simultaneously. To alleviate the negative effects raised by label shift in OSDA, we propose Open-set Movingthreshold Estimation and Gradual Alignment (OMEGA)-a novel architecture that improves existing OSDA methods on classimbalanced data. Specifically, a novel unknown-aware target clustering scheme is proposed to form tight clusters in the target domain to reduce the negative effects of label shift and intra-domain class imbalance. Furthermore, moving-threshold estimation is designed to generate specific thresholds for each target sample rather than using one for all. Extensive experiments on IOSDA, OSDA and OPDA benchmarks demonstrate that our method could significantly outperform existing state-of-the-arts. Code and data are available at https://github.com/mendicant04/OMEGA.

Universal Domain Adaptation Through Self Supervision

Abstract: Unsupervised domain adaptation methods traditionally assume that all source categories are present in the target domain. In practice, little may be known about the category overlap between the two domains. While some methods address target settings with either partial or open-set categories, they assume that the particular setting is known a priori. We propose a more universally applicable domain adaptation approach that can handle arbitrary category shift, called Domain Adaptative Neighborhood Clustering via Entropy optimization (DANCE). DANCE combines two novel ideas: First, as we cannot fully rely on source categories to learn features discriminative for the target, we propose a novel neighborhood clustering technique to learn the structure of the target domain in a self-supervised way. Second, we use entropy-based feature alignment and rejection to align target features with the source, or reject them as unknown categories based on their entropy. We show through extensive experiments that DANCE outperforms baselines across open-set, open-partial and partial domain adaptation settings.


Recent research has shown how to leverage modern networks to achieve high throughput and low latency with an RPC-based system #REFR .

Ordered key-value stores expand the set of supported applications by providing an efficient SCAN operation to retrieve the key-value pairs whose keys are within a specified range.

For example, distributed file systems can use SCAN to map ranges of logical file offsets to the nodes storing the data #OTHEREFR .

It is also used to query graph stores #OTHEREFR 50] and the popular Redis [14] offers sorted sets.

We describe Honeycomb, a system that provides hardware acceleration for an in-memory ordered key-value store.

There is a large body of research on improving the performance of ordered key-value stores, e.g., #OTHEREFR .

Other research has explored using one-sided RDMA reads to bypass the server CPU for GET and SCAN operations #OTHEREFR .

Since RDMA NICs only provide simple one-sided reads of contiguous memory regions, these systems require at least two RDMA reads per operation when supporting variable-sized keys or values, and they require client-side caching to avoid additional RDMAs when traversing the tree data structures stored by the servers.

Honeycomb accelerates an ordered key-value store using an FPGA-based SmartNIC #OTHEREFR attached to a host CPU.

These SmartNICs are widely deployed in data centers #OTHEREFR .

They enable effective CPU offload by avoiding the functionality limitations of RDMA and the performance problems of SmartNICs based on general-purpose, low-power cores #OTHEREFR .

Honeycomb: Ordered Key-Value Store Acceleration On An Fpga-Based Smartnic

Abstract: In-memory ordered key-value stores are an important building block in modern distributed applications. We present Honeycomb, a hybrid software-hardware system for accelerating read-dominated workloads on ordered key-value stores that provides linearizability for all operations including scans. Honeycomb stores a B-Tree in host memory, and executes SCAN and GET on an FPGA-based SmartNIC, and PUT, UPDATE and DELETE on the CPU. This approach enables large stores and simplifies the FPGA implementation but raises the challenge of data access and synchronization across the slow PCIe bus. We describe how Honeycomb overcomes this challenge with careful data structure design, caching, request parallelism with out-of-order request execution, wait-free read operations, and batching synchronization between the CPU and the FPGA. For read-heavy YCSB workloads, Honeycomb improves the throughput of a state-of-the-art ordered key-value store by at least 1.8×. For scan-heavy workloads inspired by cloud storage, Honeycomb improves throughput by more than 2×. The cost-performance, which is more important for large-scale deployments, is improved by at least 1.5× on these workloads. * This work was done when affiliated with Microsoft † This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

Datacenter Rpcs Can Be General And Fast

Abstract: It is commonly believed that datacenter networking software must sacrifice generality to attain high performance. The popularity of specialized distributed systems designed specifically for niche technologies such as RDMA, lossless networks, FPGAs, and programmable switches testifies to this belief. In this paper, we show that such specialization is not necessary. eRPC is a new general-purpose remote procedure call (RPC) library that offers performance comparable to specialized systems, while running on commodity CPUs in traditional datacenter networks based on either lossy Ethernet or lossless fabrics. eRPC performs well in three key metrics: message rate for small messages; bandwidth for large messages; and scalability to a large number of nodes and CPU cores. It handles packet loss, congestion, and background request execution. In microbenchmarks, one CPU core can handle up to 10 million small RPCs per second, or send large messages at 75 Gbps. We port a production-grade implementation of Raft state machine replication to eRPC without modifying the core Raft source code. We achieve 5.5 µs of replication latency on lossy Ethernet, which is faster than or comparable to specialized replication systems that use programmable switches, FPGAs, or RDMA.


We take the original human face FFHQ dataset #REFR and create a set of image masks based on our matting network.

We utilize Fast Multi-Level Foreground Estimation #OTHEREFR to extract foregrounds.

These foregrounds are blended (1) with new backgrounds, and the resulting compositions are used as inputs for StyleMatte.

Additional background images were obtained from the BG-20k dataset #OTHEREFR .

First, we filter the dataset to contain images with only one person using an instance segmentation neural network #OTHEREFR .

The filtration step is obligatory because some visual artifacts arise when skipping it.

We obtain the results of StyleMatte for the remaining 90% images as additional αchannels.

We perform an additional filtration step to get Method FID FFHQ AFHQ v2 PSeg #OTHEREFR 62 rid of some masks that are visually inconsistent with the contours of the portrait.

To do so, we extract segmentation masks s with a basic semantic segmentation network #OTHEREFR pretrained on ResNet50 #OTHEREFR and choose only those that are aligned with the matte.

Adversarially-Guided Portrait Matting

Abstract: We present a method for generating alpha mattes using a limited data source. We pretrain a novel transformerbased model (StyleMatte) on portrait datasets. We utilize this model to provide image-mask pairs for the StyleGAN3based network (StyleMatteGAN). This network is trained unsupervisedly and generates previously unseen imagemask training pairs that are fed back to StyleMatte. We demonstrate that the performance of the matte pulling network improves during this cycle and obtains top results on the used datasets. Furthermore, StyleMatte-GAN provides high-resolution, privacy-preserving portraits with alpha mattes, making it suitable for various image composition tasks. Our code is available at https://github.com/chroneus/stylematte.

A Style-Based Generator Architecture For Generative Adversarial Networks

Abstract: We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.


Because Of The Challenges Of Limitation In Network Bandwidth

Inspired by prototypical networks #REFR , which adopts a single prototype to represent each class by calculating the mean of the class's embedding space.

FedPer #OTHEREFR proposes a strategy of adding a personalized layer to the base layer and suggests updating only the base layer during the federated training process.

Afterwards, clients can update their personalized layer based on their own local data.

Additionally, #OTHEREFR explores a benchmark for non-IID settings, they divide non-IID settings into five cases, such as label distribution skew, feature distribution skew, quantity skew, etc.

Further, as #OTHEREFR mentioned, some existing studies #OTHEREFR - #OTHEREFR , #OTHEREFR cover only one non-IID case, which do not give sufficient evaluations to this challenge.

Therefore, to avoid the influence of biased global models and to evaluate non-IID cases as comprehensively as possible, we focus on personalized FL by optimizing the local objective of each local client under the label and feature distribution skewness.

This prototype can serve as an important information carrier to boost the performance of various learning domains, and has been successfully applied in meta-learning #OTHEREFR , multi-task learning #OTHEREFR , and transfer learning #OTHEREFR .

There have been some existing works #OTHEREFR , #OTHEREFR - #OTHEREFR introducing the concept of prototypes into FL.

FedProto #OTHEREFR proposes to reduce communication overhead by only exchanging prototypes between clients and the server, instead of exchanging gradients or model parameters.

FedPCL #OTHEREFR proposes to use multiple pre-trained models to extract the features separately, and then they use a projection network to fuse these extracted features in a personalized way while keeping the shared representation compact for efficient communication.

These works adopt a single prototype to represent each class and argue that directly averaging the representations from heterogeneous data across clients can effectively capture the embedding representations of each class.

Mp-Fedcl: Multi-Prototype Federated Contrastive Learning For Edge Intelligence

Abstract: Federated learning-assisted edge intelligence enables privacy protection in modern intelligent services. However, not Independent and Identically Distributed (non-IID) distribution among edge clients can impair the local model performance. The existing single prototype-based strategy represents a sample by using the mean of the feature space. However, feature spaces are usually not clustered, and a single prototype may not represent a sample well. Motivated by this, this paper proposes a multiprototype federated contrastive learning approach (MP-FedCL) which demonstrates the effectiveness of using a multi-prototype strategy over a single-prototype under non-IID settings, including both label and feature skewness. Specifically, a multi-prototype computation strategy based on k-means is first proposed to capture different embedding representations for each class space, using multiple prototypes (k centroids) to represent a class in the embedding space. In each global round, the computed multiple prototypes and their respective model parameters are sent to the edge server for aggregation into a global prototype pool, which is then sent back to all clients to guide their local training. Finally, local training for each client minimizes their own supervised learning tasks and learns from shared prototypes in the global prototype pool through supervised contrastive learning, which encourages them to learn knowledge related to their own class from others and reduces the absorption of unrelated knowledge in each global iteration. Experimental results on MNIST, Digit-5, Office-10, and DomainNet show that our method outperforms multiple baselines, with an average test accuracy improvement of about 4.6% and 10.4% under feature and label non-IID distributions, respectively.

Prototypical Networks For Few-Shot Learning

Abstract: A recent approach to few-shot classification called matching networks has demonstrated the benefits of coupling metric learning with a training procedure that mimics test. This approach relies on a complicated fine-tuning procedure and an attention scheme that forms a distribution over all points in the support set, scaling poorly with its size. We propose a more streamlined approach, prototypical networks, that learns a metric space in which few-shot classification can be performed by computing Euclidean distances to prototype representations of each class, rather than individual points. Our method is competitive with state-of-the-art one-shot classification approaches while being much simpler and more scalable with the size of the support set. We empirically demonstrate the performance of our approach on the Omniglot and mini-ImageNet datasets. We further demonstrate that a similar idea can be used for zero-shot learning, where each class is described by a set of attributes, and achieve state-of-the-art results on the Caltech UCSD bird dataset.


Personally identifiable data can be anonymized using on-device resources before it is further processed by a third party #REFR .

Fog computing is an abstraction around the combined compute and storage resources at the edge, in the core network, and in the cloud #OTHEREFR .

Fog computing is a promising paradigm for emerging domains such as the IoT: By extending cloud resources close to the edge of the network, where data is generated, fog applications can benefit from low-latency, high-bandwidth, privacy-preserving data processing infrastructure #OTHEREFR .

For example, IoT sensor data can be processed by compute resources at a local radio gateway and sent to actuators at the edge without incurring a significant network delay #OTHEREFR .

Similarly, data from multiple sensors can be aggregated and filtered at the edge in order to limit the network strain of sending all data to the cloud.

The downside of geo-distributed, heterogeneous fog infrastructure is the complexity of its management for applications, i.e., building approaches for distributing data, deploying services, and managing fault-tolerance #OTHEREFR .

Researchers have proposed abstractions in the form of fog data processing platforms that manage this complexity for applications.

For example, NebulaStream #OTHEREFR is an end-to-end IoT data management system.

A key novelty of NebulaStream is its ability to autonomously deploy different data processing approaches such as stream and complex event processing in a geo-distributed environment.

SoFA #OTHEREFR uses Apache Spark to combine all available fog resources for stream processing operator deployment.

Towards A Benchmark For Fog Data Processing

Abstract: Fog data processing systems provide key abstractions to manage data and event processing in the geo-distributed and heterogeneous fog environment. The lack of standardized benchmarks for such systems, however, hinders their development and deployment, as different approaches cannot be compared quantitatively. Existing cloud data benchmarks are inadequate for fog computing, as their focus on workload specification ignores the tight integration of application and infrastructure inherent in fog computing. In this paper, we outline an approach to a fog-native data processing benchmark that combines workload specifications with infrastructure specifications. This holistic approach allows researchers and engineers to quantify how a software approach performs for a given workload on given infrastructure. Further, by basing our benchmark in a realistic IoT sensor network scenario, we can combine paradigms such as low-latency event processing, machine learning inference, and offline data analytics, and analyze the performance impact of their interplay in a fog data processing system.

Fog Computing As Privacy Enabler

Abstract: Despite broad discussions on privacy challenges arising from fog computing, the authors argue that privacy and security requirements might actually drive the adoption of fog computing. They present four patterns of fog computing fostering data privacy and the security of business secrets. Their practical application is illuminated on the basis of three case studies.


Cross-Modal Adaptive Message Passing (CAMP) #REFR explores a region-word affinity matrix via inner product and transfers cross-modality contents to improve the region and word representations, which are then aggregated as the holistic image and text features to compute the final similarity.

To demonstrate their great applicability, we apply these two regulators to many existing methods based on cross-modal interaction:

Stacked Cross Attention (SCAN) #OTHEREFR first computes all region-word similarities and aligns each region/word with its corresponding words/regions.

The final similarity is obtained by averaging all region/word-based cosine distances.

Bidirectional Focal Attention (BFAN) #OTHEREFR extends the generic attention by reassigning more fine-grained attention weight for each region-word pair and calculates the matching result by summing up region-based and word-based scores.

Position Focused Attention (PFAN) #OTHEREFR enhances region features by introducing extra position information to promote region-word correspondences and integrates all region/wordattended cosine similarities as the prediction.

Similarity Graph Reasoning and Attention Filtration (SGRAF) #OTHEREFR adopts cosine similarities multiplied with a fixed temperature as region-word attention weights, followed by the complex graph and attention modules to map hierarchical similarity features into a matching score. Fig.

4 illustrates how we plug the RCR or RAR into the above matching approaches.

Specifically, cross-modal attention utilizes the cosine metric or inner product as region-word affinity weights, and outputs each region/word along with its related words/regions.

With these paired features, the RCR first constructs the alignment vectors and then learns the corresponding weight vectors and temperature factors via Eq.

(7)- #OTHEREFR , which in turn refine the region-word feature distances and optimize the cross-modal interaction via Eq. (9)- #OTHEREFR .

Plug-And-Play Regulators For Image-Text Matching

Abstract: Exploiting fine-grained correspondence and visualsemantic alignments has shown great potential in image-text matching. Generally, recent approaches first employ a crossmodal attention unit to capture latent region-word interactions, and then integrate all the alignments to obtain the final similarity. However, most of them adopt one-time forward association or aggregation strategies with complex architectures or additional information, while ignoring the regulation ability of network feedback. In this paper, we develop two simple but quite effective regulators which efficiently encode the message output to automatically contextualize and aggregate cross-modal representations. Specifically, we propose (i) a Recurrent Correspondence Regulator (RCR) which facilitates the cross-modal attention unit progressively with adaptive attention factors to capture more flexible correspondence, and (ii) a Recurrent Aggregation Regulator (RAR) which adjusts the aggregation weights repeatedly to increasingly emphasize important alignments and dilute unimportant ones. Besides, it is interesting that RCR and RAR are "plug-and-play": both of them can be incorporated into many frameworks based on cross-modal interaction to obtain significant benefits, and their cooperation achieves further improvements. Extensive experiments on MSCOCO and Flickr30K datasets validate that they can bring an impressive and consistent R@1 gain on multiple models, confirming the general effectiveness and generalization ability of the proposed methods.

Camp: Cross-Modal Adaptive Message Passing For Text-Image Retrieval

Abstract: Text-image cross-modal retrieval is a challenging task in the field of language and vision. Most previous approaches independently embed images and sentences into a joint embedding space and compare their similarities. However, previous approaches rarely explore the interactions between images and sentences before calculating similarities in the joint space. Intuitively, when matching between images and sentences, human beings would alternatively attend to regions in images and words in sentences, and select the most salient information considering the interaction between both modalities. In this paper, we propose Cross-modal Adaptive Message Passing (CAMP), which adaptively controls the information flow for message passing across modalities. Our approach not only takes comprehensive and fine-grained cross-modal interactions into account, but also properly handles negative pairs and irrelevant information with an adaptive gating scheme. Moreover, instead of conventional joint embedding approaches for text-image matching, we infer the matching score based on the fused features, and propose a hardest negative binary cross-entropy loss for training. Results on COCO and Flickr30k significantly surpass state-of-the-art methods, demonstrating the effectiveness of our approach.

cross-modality contents
Cross-Modal Adaptive Message


They design a regression model using BERT-based features extracted from review texts, star rating, and product type information from Amazon product review dataset #REFR .

These features are fed into conventional classifiers such as SVM, Random Forest, and gradient boosting to identify helpful reviews.

• TextCNN #OTHEREFR employs a textbased CNN model #OTHEREFR to automatically capture the character-level, word-level, and topic-level features for helpfulness prediction.

• MTNL (Fan et al., 2018) utilizes end-to-end multi-task neural learning (MTNL) architecture for classifying helpful reviews.

They take the help of an auxiliary task, such as rating regression, to boost the performance of the original task, which is review helpfulness identification.

• BERTHelp #OTHEREFR develop their helpfulness prediction model using pre-trained BERT #OTHEREFR .

On The Role Of Reviewer Expertise In Temporal Review Helpfulness Prediction

Abstract: Helpful reviews have been essential for the success of e-commerce services, as they help customers make quick purchase decisions and benefit the merchants in their sales. While many reviews are informative, others provide little value and may contain spam, excessive appraisal, or unexpected biases. With the large volume of reviews and their uneven quality, the problem of detecting helpful reviews has drawn much attention lately. Existing methods for identifying helpful reviews primarily focus on review text and ignore the two key factors of (1) who post the reviews and (2) when the reviews are posted. Moreover, the helpfulness votes suffer from scarcity for less popular products and recently submitted (a.k.a., coldstart) reviews. To address these challenges, we introduce a dataset and develop a model that integrates the reviewer's expertise, derived from the past review history of the reviewers, and the temporal dynamics of the reviews to automatically assess review helpfulness. We conduct experiments on our dataset to demonstrate the effectiveness of incorporating these factors and report improved results compared to several well-established baselines.

Ups And Downs: Modeling The Visual Evolution Of Fashion Trends With One-Class Collaborative Filtering

Abstract: Building a successful recommender system depends on understanding both the dimensions of people's preferences as well as their dynamics. In certain domains, such as fashion, modeling such preferences can be incredibly difficult, due to the need to simultaneously model the visual appearance of products as well as their evolution over time. The subtle semantics and non-linear dynamics of fashion evolution raise unique challenges especially considering the sparsity and large scale of the underlying datasets. In this paper we build novel models for the One-Class Collaborative Filtering setting, where our goal is to estimate users' fashion-aware personalized ranking functions based on their past feedback. To uncover the complex and evolving visual factors that people consider when evaluating products, our method combines high-level visual features extracted from a deep convolutional neural network, users' past feedback, as well as evolving trends within the community. Experimentally we evaluate our method on two large real-world datasets from Amazon.com, where we show it to outperform state-of-the-art personalized ranking measures, and also use it to visualize the high-level fashion trends across the 11-year span of our dataset.


Tellez et al., #REFR showed in a benchmark study that VAE compression of medical tissue images to a latent space of 128 (>5000:1 compression ratio) retained the most details of the original whole slide image compared to 4 other encoders.

Compression and scaling has also been found to adversely effect tissue segmentation up to ratios of 50:1 #OTHEREFR .

In contrast to discrete cosine transformation models, neural networks have been proven to retain high efficiency and fidelity in the lossy compression of image data #OTHEREFR .

While neural networks seek to store image data in latent space representations, not every network does this at equivalent efficiency or accuracy #OTHEREFR .

For better visualization, early stopping is not used for these experiments.

coders (VAEs) retain higher image quality and lower noise ratios at extreme compression ratios #OTHEREFR .

In the current study, we develop a VAE to compress and index images in latent space for fast complex search of whole slide H&E cancer images.

Clinically Relevant Latent Space Embedding Of Cancer Histopathology Slides Through Variational Autoencoder Based Image Compression

Abstract: In this paper, we introduce a Variational Autoencoder (VAE) based training approach that can compress and decompress cancer pathology slides at a compression ratio of 1:512, which is better than the previously reported state of the art (SOTA) in the literature, while still maintaining accuracy in clinical validation tasks. The compression approach was tested on more common computer vision datasets such as CIFAR10, and we explore which image characteristics enable this compression ratio on cancer imaging data but not generic images. We generate and visualize embeddings from the compressed latent space and demonstrate how they are useful for clinical interpretation of data, and how in the future such latent embeddings can be used to accelerate search of clinical imaging data.

Neural Image Compression For Gigapixel Histopathology Image Analysis

Abstract: We propose Neural Image Compression (NIC), a two-step method to build convolutional neural networks for gigapixel image analysis solely using weak image-level labels. First, gigapixel images are compressed using a neural network trained in an unsupervised fashion, retaining high-level information while suppressing pixel-level noise. Second, a convolutional neural network (CNN) is trained on these compressed image representations to predict image-level labels, avoiding the need for fine-grained manual annotations. We compared several encoding strategies, namely reconstruction error minimization, contrastive training and adversarial feature learning, and evaluated NIC on a synthetic task and two public histopathology datasets. We found that NIC can exploit visual cues associated with image-level labels successfully, integrating both global and local visual information. Furthermore, we visualized the regions of the input gigapixel images where the CNN attended to, and confirmed that they overlapped with annotations from human experts.


For experiments on PACS and VisDA-C, we also apply WeightNorm #REFR on the classifier.

Following #OTHEREFR , we use a subset of it that contains 126 classes from 4 domains (Real, Sketch, Clipart, Painting) and we refer to it as DomainNet-126.

We evaluate 7 domain shifts built from the 4 domains and we report the top-1 accuracy under each domain shift as well as their average (Avg.). Implementation details.

We use standard classification architectures comprising a feature extractor followed by a Single-Source UDA Method SF P → A P → C P → S A → P A → C A → S Avg.

Following SHOT #OTHEREFR , we add an extra 256-dimensional fully-connected+BatchNorm bottleneck after the encoder output.

For source training, we initialise the ResNet backbone with ImageNet-1K #OTHEREFR pre-trained weights available in the Pytorch model zoo.

We train the source model with the standard cross-entropy loss and with label-smoothing like in #OTHEREFR .

For the adaptation phase, the target model is initialised with the source model's parameters.

For more details, the code that is available at https://github.com/MattiaLitrico/Guiding-Pseudo-labels-with-Uncertainty-Estimation-for-Sourcefree-Unsupervised-Domain-Adaptation.

Guiding Pseudo-Labels With Uncertainty Estimation For Source-Free Unsupervised Domain Adaptation

Abstract: Standard Unsupervised Domain Adaptation (UDA) methods assume the availability of both source and target data during the adaptation. In this work, we investigate the Source-free Unsupervised Domain Adaptation (SF-UDA), a specific case of UDA where a model is adapted to a target domain without access to source data. We propose a novel approach for the SF-UDA setting based on a loss reweighting strategy that brings robustness against the noise that inevitably affects the pseudo-labels. The classification loss is reweighted based on the reliability of the pseudo-labels that is measured by estimating their uncertainty. Guided by such reweighting strategy, the pseudo-labels are progressively refined by aggregating knowledge from neighbouring samples. Furthermore, a self-supervised contrastive framework is leveraged as a target space regulariser to enhance such knowledge aggregation. A novel negative pairs exclusion strategy is proposed to identify and exclude negative pairs made of samples sharing the same class, even in presence of some noise in the pseudo-labels. Our method outperforms previous methods on three major benchmarks by a large margin. We set the new SF-UDA state-of-the-art on VisDA-C and DomainNet with a performance gain of +1.8% on both benchmarks and on PACS with +12.3% in the single-source setting and +6.6% in multi-target adaptation. Additional analyses demonstrate that the proposed approach is robust to the noise, which results in significantly more accurate pseudo-labels compared to state-of-the-art approaches.

Weight Normalization: A Simple Reparameterization To Accelerate Training Of Deep Neural Networks

Abstract: We present weight normalization: a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction. By reparameterizing the weights in this way we improve the conditioning of the optimization problem and we speed up convergence of stochastic gradient descent. Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also be applied successfully to recurrent models such as LSTMs and to noise-sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited. Although our method is much simpler, it still provides much of the speed-up of full batch normalization. In addition, the computational overhead of our method is lower, permitting more optimization steps to be taken in the same amount of time. We demonstrate the usefulness of our method on applications in supervised image recognition, generative modelling, and deep reinforcement learning.


Gradientbased methods may generate the same results using an untrained model as those from a trained model, indicating that the method demonstrates only general model characteristics #REFR .

However, as inferred from the algorithm strategy, the computation of integrated gradients requires repeated calculation over multiple iterations. To reduce computational cost, #OTHEREFR

(2021) proposed a special class of CNNs to compute the result of integrated gradients with only one forward-backward pass.

As a result, the saliency map itself can be used as an effective tool; for example, it can be integrated into regular training as priors #OTHEREFR .

Each of the discussed methods has its own strengths and weaknesses.

For example, the perturbation-based method may underestimate the importance of features because other features already saturate the output #OTHEREFR .

Although CAM methods can generate a saliency map by passing through fewer non-linear layers and potentially suffer from fewer issues, they may predict inaccurate object locations due to the coarse resolution of the saliency map.

In this paper, we explore the characteristics of different "feature attribution-based" model explanation approaches and use two GeoAI-ready natural feature datasets in an image classification task to compare model-learned features with human-understandable features.

In addition to interpreting a model's reasoning process, we compare the strengths and weaknesses of these popular approaches and discuss ways to improve them within the growing field of explainable GeoAI.

Explainable Geoai: Can Saliency Maps Help Interpret Artificial Intelligence'S Learning Process? An Empirical Study On Natural Feature Detection

Abstract: Improving the interpretability of geospatial artificial intelligence (GeoAI) models has become critically important to open the "black box" of complex AI models, such as deep learning. This paper compares popular saliency map generation techniques and their strengths and weaknesses in interpreting GeoAI and deep learning models' reasoning behaviors, particularly when applied to geospatial analysis and image processing tasks. We surveyed two broad classes of model explanation methods: perturbation-based and gradient-based methods. The former identifies important image areas, which help machines make predictions by modifying a localized area of the input image. The latter evaluates the contribution of every single pixel of the input image to the model's prediction results through gradient backpropagation. In this study, three algorithms-the occlusion method, the integrated gradients method, and the class activation map method-are examined for a natural feature detection task using deep learning. The algorithms' strengths and weaknesses are discussed, and the consistency between model-learned and human-understandable concepts for object recognition is also compared. The experiments used two GeoAIready datasets to demonstrate the generalizability of the research findings.

Sanity Checks For Saliency Maps

Abstract: Saliency methods have emerged as a popular tool to highlight features in an input deemed relevant for the prediction of a learned model. Several saliency methods have been proposed, often guided by visual appeal on image data. In this work, we propose an actionable methodology to evaluate what kinds of explanations a given method can and cannot provide. We find that reliance, solely, on visual assessment can be misleading. Through extensive experiments we show that some existing saliency methods are independent both of the model and of the data generating process. Consequently, methods that fail the proposed tests are inadequate for tasks that are sensitive to either data or model, such as, finding outliers in the data, explaining the relationship between inputs and outputs that the model learned, and debugging the model. We interpret our findings through an analogy with edge detection in images, a technique that requires neither training data nor model. Theory in the case of a linear model and a single-layer convolutional neural network supports our experimental findings.


We use AdamW optimizer #REFR with a learning rate of 1e − 4, batch size of 256, without weight decay.

During training, we use λ vlb = 1.0, and define t to vary between [1, T ], where T = 1000 corresponds to a pure Gaussian distribution.

At inference, we start from We report the results after retraining AvatarPoser, and report the same results as in #OTHEREFR for methods with a star (*).

pure Gaussian noise, and we use DDIM sampling [47] with 50 steps.

We set the variance Σ θ of the reverse noise to zero.

This configuration turns the model into a deterministic mapping from Gaussian noise to motions, allowing it to do much fewer denoising steps without degrading the quality of synthesized motions.

Our model has 22M parameters and is trained for 1.5 days on four NVIDIA Quadro RTX 8000. More implementation details are in the Supplementary Material (Sec. A.1).

Our approach has no limitations concerning the length of the generated sequences.

We can synthesize motions of arbitrary length by applying BoDiffusion in an autoregressive manner using a sliding window over the input data.

We refer the reader to the Supplementary Material for more explanation of our inference-time protocol (see Sec. A.2).

Bodiffusion: Diffusing Sparse Observations For Full-Body Human Motion Synthesis

Abstract: Mixed reality applications require tracking the user's fullbody motion to enable an immersive experience. However, typical head-mounted devices can only track head and hand movements, leading to a limited reconstruction of full-body motion due to variability in lower body configurations. We propose BoDiffusion-a generative diffusion model for motion synthesis to tackle this under-constrained reconstruction problem. We present a time and space conditioning scheme that allows BoDiffusion to leverage sparse tracking inputs while generating smooth and realistic full-body motion sequences. To the best of our knowledge, this is the first approach that uses the reverse diffusion process to model full-body tracking as a conditional sequence generation task. We conduct experiments on the large-scale motion-capture dataset AMASS and show that our approach outperforms the state-of-the-art approaches by a significant margin in terms of full-body motion realism and joint reconstruction error.

Fixing Weight Decay Regularization In Adam

Abstract: We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor. We propose a simple way to resolve this issue by decoupling weight decay and the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence. Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-of-the-art results on CIFAR-10 and ImageNet32x32. Our source code is available at https://github.com/loshchil/AdamW-and-SGDW


However, entropy maximization occasionally also reaches the state with all points in the same cluster (this is consistent with previous literature, e.g., #REFR ).

Our method significantly outperforms others on all datasets and metrics.

Observe that the unregularized model exhibits total collapse in all experiments, placing all points in the same cluster and consequently achieving a cluster performance no better than random guessing.

In many of our experiments, the performance of SS is not much better.

By making a slight change to the author's original method, we could actually significantly improve its results (see appendix), but it was still unreliable and less accurate than our method.

The other two existing partition support methods do a reasonable job of avoiding partition collapse.

The Sinkhorn-Knopp method is more reliable, but by far the most uniform cluster sizes are produced by our method.

Note that we do not employ our assignment algorithm at inference time, instead we just assign each point to the cluster with the nearest centroid.

This shows that our cluster centroids are well-distributed around the data manifold, each capturing a sizeable subset of the data even when the explicit support is removed.

Together, these figures show that (a) some form of partition support is necessary to learn anything meaningful, (b) our method of combination assignment is better at avoiding partition Table 1 : Effect, on cluster size and clustering performance, of our method compared to two existing methods of preventing partition collapse.

"CA" refers to our method of combination assignment, "SK" refers to Sinkhorn-Knopp regularization, as proposed by #OTHEREFR and #OTHEREFR , "Ent" refers to entropy maximization, as used by #OTHEREFR , "SS" is the sum of squares minimization proposed in #OTHEREFR and others, and "No-Reg" is the model without any partition support component.

Hard Regularization To Prevent Deep Clustering Collapse Without Data Augmentation

Abstract: Online deep clustering refers to the joint use of a feature extraction network and a clustering model to assign cluster labels to each new data point or batch as it is processed. While faster and more versatile than offline methods, online clustering can easily reach the collapsed solution where the encoder maps all inputs to the same point and all are put into a single cluster. Successful existing models have employed various techniques to avoid this problem, most of which require data augmentation or which aim to make the average soft assignment across the dataset the same for each cluster. We propose a method that does not require data augmentation, and that, differently from existing methods, regularizes the hard assignments. Using a Bayesian framework, we derive an intuitive optimization objective that can be straightforwardly included in the training of the encoder network. Tested on four image datasets, it consistently avoids collapse more robustly than other methods and leads to more accurate clustering. We also conduct further experiments and analyses justifying our choice to regularize the hard cluster assignments.

Learning Discrete Representations Via Information Maximizing Self Augmented Training

Abstract: Learning discrete representations of data is a central machine learning task because of the compactness of the representations and ease of interpretation. The task includes clustering and hash learning as special cases. Deep neural networks are promising to be used because they can model the non-linearity of data and scale to large datasets. However, their model complexity is huge, and therefore, we need to carefully regularize the networks in order to learn useful representations that exhibit intended invariance for applications of interest. To this end, we propose a method called Information Maximizing Self-Augmented Training (IMSAT). In IMSAT, we use data augmentation to impose the invari-ance on discrete representations. More specifically, we encourage the predicted representations of augmented data points to be close to those of the original data points in an end-to-end fashion. At the same time, we maximize the information-theoretic dependency between data and their predicted discrete representations. Extensive experiments on benchmark datasets show that IMSAT produces state-of-the-art results for both clustering and unsupervised hash learning.


We use an attentionbased Recurrent Neural Network #REFR as the language generator Φ(•), which utilizes these features to reconstruct nouns and verbs of one query.

Researchers #OTHEREFR have observed that neural networks will fit correct labels (informative labels) much faster than wrong labels (uninformative labels).

Thus, from a perspective of algorithm optimization, networks will predict unobserved positive moments more readily than negative ones when the number of positives is constrained #OTHEREFR .

Semantic reconstruction To better uncover potential positive moments, we build a lightweight semanticreconstruction model that reconstructs semantic information of queries from proposal features.

Based on a single positive label, a sequential video embedding within its interval is first extracted from the video embedding V.

Then, a random weighted averaging operation is applied to augment N s fixed-dimensional features {f v i } Ns i=1 .

In the optimization phase, a standard captioning loss is used to maximize the normalized log-likelihood of the correct words,

where x l denotes a noun/verb of one query and T s is total time step of the recurrent model.

Intuitively, positive moments can better reconstruct the semantic information of a query.

In the estimation phase, we leverage this recurrent model to generate words for both multi-scale proposals and the labeled moment, and then compare semantic similarities between them.

Towards Diverse Temporal Grounding Under Single Positive Labels

Abstract: Temporal grounding aims to retrieve moments of the described event within an untrimmed video by a language query. Typically, existing methods assume annotations are precise and unique, yet one query may describe multiple moments in many cases. Hence, simply taking it as a one-vs-one mapping task and striving to match single-label annotations will inevitably introduce false negatives during optimization. In this study, we reformulate this task as a one-vs-many optimization problem under the condition of single positive labels. The unlabeled moments are considered unobserved rather than negative, and we explore mining potential positive moments to assist in multiple moment retrieval. In this setting, we propose a novel Diverse Temporal Grounding framework, termed DTG-SPL, which mainly consists of a positive moment estimation (PME) module and a diverse moment regression (DMR) module. PME leverages semantic reconstruction information and an expected positive regularization to uncover potential positive moments in an online fashion. Under the supervision of these pseudo positives, DMR is able to localize diverse moments in parallel that meet different users. The entire framework allows for end-toend optimization as well as fast inference. Extensive experiments on Charades-STA and ActivityNet Captions show that our method achieves superior performance in terms of both single-label and multi-label metrics.

Neural Machine Translation By Jointly Learning To Align And Translate

Abstract: Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.


We have implemented the LL simulator for IEEE 802.11p and LTE C-V2X in Python in our previous work where more simulator details can be found in #REFR .

For the link simulation, it's necessary to build the entire transmission and receiving operations.

The simulation pipeline is the same as the Section.III.B in #OTHEREFR . In Fig. 3 , the simulation processing procedure is shown. Here we give short introductions about each processing step.

1) Control channel processing: In this step, a new SCI message is created which includes the MCS value, resource indication value, group destination identity, etc.

The created SCI is a binary message which is encoded by using a convolutional encoder followed by rate matching, interleaving, and a 16-bit Cyclic Redundancy Check (CRC) attached to the encoded message.

Once we have the binary codes, the next step is to process PSCCH scrambling.

There are 240 PSCCHgenerated symbols and are cyclically shifted with a random value chosen from the set [0, 3, 6, 9] to reduce the effect of interference.

Effect Of Variable Physical Numerologies On Link-Level Performance Of 5G Nr V2X

Abstract: With technology and societal development, the 5th generation wireless communication (5G) contributes significantly to different societies like industries or academies. Vehicle-to-Everything (V2X) communication technology has been one of the leading services for 5G which has been applied in vehicles. It's used to exchange their status information with other traffic and traffic participants to increase traffic safety and efficiency. Cellular-V2X (C-V2X) is one of the emerging technologies to enable V2X communications. The first Long-Term Evolution (LTE) based C-V2X was released on the 3rd Generation Partnership Project (3GPP) standard. 3GPP is working towards the development of New Radio (NR) systems that it's called 5G NR V2X. One single numerology in LTE cannot satisfy most performance requirements because of the variety of deployment options and scenarios. For this reason, in order to meet the diverse requirements, the 5G NR Physical Layer (PHY) is designed to provide a highly flexible framework. Scalable Orthogonal Frequency-Division Multiplexing (OFDM) numerologies make flexibility possible. The term numerology refers to the PHY waveform parametrization and allows different Subcarrier Spacings (SCSs), symbols, and slot duration. This paper implements the Link-Level (LL) simulations of LTE C-V2X communication and 5G NR V2X communication where simulation results are used to compare similarities and differences between LTE and 5G NR. We detect the effect of variable PHY Numerologies of 5G NR on the LL performance of V2X. The simulation results show that the performance of 5G NR improved by using variable numerologies.

Link Level Performance Comparison Of C-V2X And Its-G5 For Vehicular Channel Models

Abstract: V2X communications plays a significant role in increasing traffic safety and efficiency by enabling vehicles to exchange their status information with other vehicles and traffic entities in their proximity. In this regard, two technologies emerged as the main contenders for enabling V2X communications which have stringent requirements in terms of latency and reliability due to their apparent safety criticality. The first one is the DSRC standard (referred to as ITS-G5 in Europe) that is well researched since 20 years and has attained enough technical maturity for current deployment. The second one is the relatively new CV2X standard that is nevertheless, based on the 3GPP standard family that have successful deployments in almost every corner of the globe. In this work, we compare the link level performance of the PHY protocols for both the technologies for different vehicular fading channel models. To this end, we construct and simulate the PHY pipelines and show the performance results by means of BLER} versus SNR graphs. Our investigations show that CV2X performs better than ITS-G5 for almost all the considered channel models due to better channel coding and estimation schemes.


In the original U-net architecture, the encoder features are directly concatenated with the features obtained by the decoder #REFR .

This fusion facilitates the network to integrate the higher and lower-layer features effectively but lacks the ability to extract high-frequency features.

To tackle this issue, we introduce a High-Frequency feature guided Cross-Attention mechanism (HF-guided CA) to recover fine-grained high-frequency details.

The flow of the HF-guided CA is illustrated in Fig.5 .

We utilize the pre-trained CNN prediction by extracting theĤ i ,V i , andD i coefficients at the i-th level of the DWT.

By adding these extracted coefficients with a linear projection, we obtain the feature map Q with aggregated highfrequency information:

Resdiff: Combining Cnn And Diffusion Model For Image Super-Resolution

Abstract: Adapting the Diffusion Probabilistic Model (DPM) for direct image super-resolution is wasteful, given that a simple Convolutional Neural Network (CNN) can recover the main low-frequency content. Therefore, we present ResDiff, a novel Diffusion Probabilistic Model based on Residual structure for Single Image Super-Resolution (SISR). ResDiff utilizes a combination of a CNN, which restores primary low-frequency components, and a DPM, which predicts the residual between the ground-truth image and the CNNpredicted image. In contrast to the common diffusion-based methods that directly use LR images to guide the noise towards HR space, ResDiff utilizes the CNN's initial prediction to direct the noise towards the residual space between HR space and CNN-predicted space, which not only accelerates the generation process but also acquires superior sample quality. Additionally, a frequency-domain-based loss function for CNN is introduced to facilitate its restoration, and a frequency-domain guided diffusion is designed for DPM on behalf of predicting high-frequency details. The extensive experiments on multiple benchmark datasets demonstrate that ResDiff outperforms previous diffusionbased methods in terms of shorter model convergence time, superior generation quality, and more diverse samples.

U-Net: Convolutional Networks For Biomedical Image Segmentation

Abstract: There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .

encoder features
original U-net architecture


It is difficult to compare the results, since these works do not use the same evaluation dataset and English works can benefit from specialized models such as ClinicalBERT #REFR .

The Table IV shows the model with the highest F 1 -score of this paper with the results of previous work on ICD-10 code association.

For French baseline, we implemented and trained the model proposed in #OTHEREFR on ICD-10-HNFC dataset. The result is shown in parallel with our proposal.

Our model clearly outperforms the classification method used in #OTHEREFR .

On the same validation dataset, with class reduction (1564 labels) the F 1 -score goes from 0.35 obtained with the model proposed in #OTHEREFR to 0.55 with our proposal, i.e. an improvement of 57%.

With the raw codes (6161 labels), the F 1 -score goes from 0.27 to 0.45, i.e. an improvement of 66.6%.

The difference in scores with the results of PLM-ICD can be explained by the use of a context specific (medical) Transformers which has a vocabulary more adapted to the content of the documents.

Automatic Icd-10 Code Association: A Challenging Task On French Clinical Texts

Abstract: Automatically associating ICD codes with electronic health data is a well-known NLP task in medical research. NLP has evolved significantly in recent years with the emergence of pre-trained language models based on Transformers architecture, mainly in the English language. This paper adapts these models to automatically associate the ICD codes. Several neural network architectures have been experimented with to address the challenges of dealing with a large set of both input tokens and labels to be guessed. In this paper, we propose a model that combines the latest advances in NLP and multi-label classification for ICD-10 code association. Fair experiments on a Clinical dataset in the French language show that our approach increases the F1-score metric by more than 55% compared to state-of-the-art results.

Publicly Available Clinical Bert Embeddings

Abstract: Contextual word embedding models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) have dramatically improved performance for many natural language processing (NLP) tasks in recent months. However, these models have been minimally explored on specialty corpora, such as clinical text; moreover, in the clinical domain, no publicly-available pre-trained BERT models yet exist. In this work, we address this need by exploring and releasing BERT models for clinical text: one for generic clinical text and another for discharge summaries specifically. We demonstrate that using a domain-specific model yields performance improvements on three common clinical NLP tasks as compared to nonspecific embeddings. These domain-specific models are not as performant on two clinical de-identification tasks, and argue that this is a natural consequence of the differences between de-identified source text and synthetically non de-identified task text.


Panoptic segmentation #REFR is a task that aims to accurately segment both object instances and stuff regions in an image, providing a comprehensive understanding of the scene.

General approaches to this task include the use of a two-stage pipeline #OTHEREFR - #OTHEREFR , where instance segmentation and semantic segmentation are performed independently and then fused together.

An alternative approach is the use of a single network #OTHEREFR - #OTHEREFR that is able to predict both instancelevel and semantic-level segmentation maps simultaneously.

The Panoptic FCN methodology #OTHEREFR is a single-stage approach that has been shown to achieve state-of-the-art performance using an elegant dynamic-convolutions based approach, which has the property that each detected object has a learned compressed representation, i.e. kernel, that could be used for tracking.

Unified Perception: Efficient Depth-Aware Video Panoptic Segmentation With Minimal Annotation Costs

Abstract: Depth-aware video panoptic segmentation is a promising approach to camera based scene understanding. However, the current state-of-the-art methods require costly video annotations and use a complex training pipeline compared to their image-based equivalents. In this paper, we present a new approach titled Unified Perception that achieves state-of-theart performance without requiring video-based training. Our method employs a simple two-stage cascaded tracking algorithm that (re)uses object embeddings computed in an imagebased network. Experimental results on the Cityscapes-DVPS dataset demonstrate that our method achieves an overall DVPQ of 57.1, surpassing state-of-the-art methods. Furthermore, we show that our tracking strategies are effective for long-term object association on KITTI-STEP, achieving an STQ of 59.1 which exceeded the performance of state-of-the-art methods that employ the same backbone network.

Panoptic Segmentation

Abstract: We propose and study a task we name panoptic segmentation (PS). Panoptic segmentation unifies the typically distinct tasks of semantic segmentation (assign a class label to each pixel) and instance segmentation (detect and segment each object instance). The proposed task requires generating a coherent scene segmentation that is rich and complete, an important step toward real-world vision systems. While early work in computer vision addressed related image/scene parsing tasks, these are not currently popular, possibly due to lack of appropriate metrics or associated recognition challenges. To address this, we propose a novel panoptic quality (PQ) metric that captures performance for all classes (stuff and things) in an interpretable and unified manner. Using the proposed metric, we perform a rigorous study of both human and machine performance for PS on three existing datasets, revealing interesting insights about the task. The aim of our work is to revive the interest of the community in a more unified view of image segmentation. For more analysis and up-to-date results, please check the arXiv version of the paper: {\small\url{https://arxiv.org/abs/1801.00868}}.


Inspired by #REFR , we clip and make w ∈ [0, 1], ∀w ∈ W i for regularization via σ(w) = max(0, min(1, w)).

Following existing pFL paradigms, we globally share the representation part of the segmentation network (encoder F e and the SCR module) and personalize the task head (decoder F d ).

To element-wisely aggregate two models without introducing multiple aggregation weight matrices, we adopt an adaptive learning based approach similar to residual learning for updating the weight matrices. The adaptive head aggregation is formulated aŝ

At the beginning, every element in W i is initialized to be 1, and then iteratively gets updated via

3and 4are respectively utilized to alternatively updateθ t i and W i .

Upon convergence, the initialization parameterθ t i for F d in each federation round is obtained, after which local models are trained using Eq. (1).

Unifying And Personalizing Weakly-Supervised Federated Medical Image Segmentation Via Adaptive Representation And Aggregation

Abstract: Federated learning (FL) enables multiple sites to collaboratively train powerful deep models without compromising data privacy and security. The statistical heterogeneity (e.g., non-IID data and domain shifts) is a primary obstacle in FL, impairing the generalization performance of the global model. Weakly supervised segmentation, which uses sparsely-grained (i.e., point-, bounding box-, scribble-, block-wise) supervision, is increasingly being paid attention to due to its great potential of reducing annotation costs. However, there may exist label heterogeneity, i.e., different annotation forms across sites. In this paper, we propose a novel personalized FL framework for medical image segmentation, named FedICRA, which uniformly leverages heterogeneous weak supervision via adaptIve Contrastive Representation and Aggregation. Concretely, to facilitate personalized modeling and to avoid confusion, a channel selection based site contrastive representation module is employed to adaptively cluster intra-site embeddings and separate inter-site ones. To effectively integrate the common knowledge from the global model with the unique knowledge from each local model, an adaptive aggregation module is applied for updating and initializing local models at the element level. Additionally, a weakly supervised objective function that leverages a multiscale tree energy loss and a gated CRF loss is employed to generate more precise pseudo-labels and further boost the segmentation performance. Through extensive experiments on two distinct medical image segmentation tasks of different modalities, the proposed FedICRA demonstrates overwhelming performance over other state-ofthe-art personalized FL methods. Its performance even approaches that of fully supervised training on centralized data. Our code and data are available at https://github.com/llmir/FedICRA.

Adaptive Gradient Methods With Dynamic Bound Of Learning Rate

Abstract: Adaptive optimization methods such as AdaGrad, RMSprop and Adam have been proposed to achieve a rapid training process with an element-wise scaling term on learning rates. Though prevailing, they are observed to generalize poorly compared with SGD or even fail to converge due to unstable and extreme learning rates. Recent work has put forward some algorithms such as AMSGrad to tackle this issue but they failed to achieve considerable improvement over existing methods. In our paper, we demonstrate that extreme learning rates can lead to poor performance. We provide new variants of Adam and AMSGrad, called AdaBound and AMSBound respectively, which employ dynamic bounds on learning rates to achieve a gradual and smooth transition from adaptive methods to SGD and give a theoretical proof of convergence. We further conduct experiments on various popular tasks and models, which is often insufficient in previous work. Experimental results show that new variants can eliminate the generalization gap between adaptive methods and SGD and maintain higher learning speed early in training at the same time. Moreover, they can bring significant improvement over their prototypes, especially on complex deep networks. The implementation of the algorithm can be found at this https URL .