Comparison to existing results for MOD: Finally, we can compare our result with that in #REFR for MOD with an ℓ 1 -minimisation based sparse approximation routine.

The main reason is that (13) is a very light condition which does not exclude the existence and rare selection of sparse supports, leading to ill-conditioned or even rank-deficient matrices Ψ I .

Such supports cannot be recovered by thresholding or any other sparse approximation algorithm.

This failure probability stops the algorithm from attaining arbitrarily small precision.

If we exchange (13) by the more restrictive assumption 2Sµ(Φ) < 1 then all supports of size S are well enough conditioned to be identified by a sparse algorithm like OMP using Φ (or some very small perturbation of it).

This means that Φ is a fixed point of the algorithm and that for a close enough initialisation we have convergence to Φ in expectation.

Assuming a signal model with uniformly selected random supports and a non-zero coefficient distribution, which is bounded from above, it is shown that for an approximately tight dictionary with coherence µ(Φ) 1/ √ d and a sparsity level S d 1/6 MOD will converge from any initialisation satisfying ε(Ψ, Φ) 1/S 2 , given enough training signals.

In this special case our assumptions on the generating dictionary -µ(Φ) 1/ log K and S d/ log(K) -are more relaxed.

As already seen, this comes at the price of a theoretical limit on the achievable accuracy, which in practice, however, is determined by the number of available signals.

Further, we also have lighter conditions on the distance between the initialisation and the generating dictionary.

To see this, note that for a tight dictionary our conditions can be written as

Convergence Of Alternating Minimisation Algorithms For Dictionary Learning

Abstract: In this paper we derive sufficient conditions for the convergence of two popular alternating minimisation algorithms for dictionary learning-the Method of Optimal Directions (MOD) and Online Dictionary Learning (ODL), which can also be thought of as approximative K-SVD. We show that given a well-behaved initialisation that is either within distance at most 1/ log(K) to the generating dictionary or has a special structure ensuring that each element of the initialisation only points to one generating element, both algorithms will converge with geometric convergence rate to the generating dictionary. This is done even for data models with non-uniform distributions on the supports of the sparse coefficients. These allow the appearance frequency of the dictionary elements to vary heavily and thus model real data more closely.

Learning Sparsely Used Overcomplete Dictionaries Via Alternating Minimization

Abstract: We consider the problem of sparse coding, where each sample consists of a sparse linear combination of a set of dictionary atoms, and the task is to learn both the dictionary elements and the mixing coefficients. Alternating minimization is a popular heuristic for sparse coding, where the dictionary and the coefficients are estimated in alternate steps, keeping the other fixed. Typically, the coefficients are estimated via $\ell_1$ minimization, keeping the dictionary fixed, and the dictionary is estimated through least squares, keeping the coefficients fixed. In this paper, we establish local linear convergence for this variant of alternating minimization and establish that the basin of attraction for the global optimum (corresponding to the true dictionary and the coefficients) is $\mathcal{O}(1/s^{2})$, where $s$ is the sparsity level in each sample and the dictionary satisfies restricted isometry property. Combined with the recent results of approximate dictionary estimation, this yields provable guar...


For example, differential privacy (DP) can be implemented, in a similar concept to DP in training ML models #REFR .

Sharing the deep features across clients can potentially increase the risk for membership inference attacks, meaning attacks that determine if a data point (i.e., a medical image) is part of the training process.

However, membership inference attacks may not be a significant concern in such a healthcare setting, as two institutions would need the same or a very similar data point of a patient, and the maximum information gained would be that a patient visited both institutions.

More concerning would be reconstruction attacks, in which SL clients could reconstruct training data points (e.g., medical images) from the data.

Such reconstruction attacks are difficult here, since only the last output layer of the neural network in an inference with training data points is shared. Contemporary attacks require access to more layers #OTHEREFR .

Still, there could be ways for additional protection and guarantees against such hypothetically possible attack scenarios.

Thereby, before clients submit the vectors of the last output layer of each training data point to the servers, they can add Gaussian noise to the vectors.

The standard deviation of the noise could depend on the standard deviation of each vector dimension, and a scaling factor which controls the level of privacy protection. The mean of the noise is 0. Additionally, a clipping can be implemented. We leave this implementation to future research.

An interesting question for future research would be the impact of DP on the SV estimation accuracy, and how it affects SVs (e.g., SVs become more uniform with more noise).

Scalable Data Point Valuation In Decentralized Learning

Abstract: Existing research on data valuation in federated and swarm learning focuses on valuing client contributions and works best when data across clients is independent and identically distributed (IID). In practice, data is rarely distributed IID. We develop an approach called DDVal for decentralized data valuation, capable of valuing individual data points in federated and swarm learning. DDVal is based on sharing deep features and approximating Shapley values through a k-nearest neighbor approximation method. This allows for novel applications, for example, to simultaneously reward institutions and individuals for providing data to a decentralized machine learning task. The valuation of data points through DDVal allows to also draw hierarchical conclusions on the contribution of institutions, and we empirically show that the accuracy of DDVal in estimating institutional contributions is higher than existing Shapley value approximation methods for federated learning. Specifically, it reaches a cosine similarity in approximating Shapley values of 99.969 % in both, IID and non-IID data distributions across institutions, compared with 99.301 % and 97.250 % for the best state of the art methods. DDVal scales with the number of data points instead of the number of clients, and has a loglinear complexity. This scales more favorably than existing approaches with an exponential complexity. We show that DDVal is especially efficient in data distribution scenarios with many clients that have few data points-for example, more than 16 clients with 8,000 data points each. By integrating DDVal into a decentralized system, we show that it is not only suitable for centralized federated learning, but also decentralized swarm learning, which aligns well with the research on emerging internet technologies such as web3 to reward users for providing data to algorithms.

Deep Learning With Differential Privacy

Abstract: Machine learning techniques based on neural networks are achieving remarkable results in a wide variety of domains. Often, the training of models requires large, representative datasets, which may be crowdsourced and contain sensitive information. The models should not expose private information in these datasets. Addressing this goal, we develop new algorithmic techniques for learning and a refined analysis of privacy costs within the framework of differential privacy. Our implementation and experiments demonstrate that we can train deep neural networks with non-convex objectives, under a modest privacy budget, and at a manageable cost in software complexity, training efficiency, and model quality.


Meanwhile, #REFR thought that bounding box level tracking is saturating and introduced pixel-level tracking as a way to improve performance.

Another work #OTHEREFR focused on improving tracking performance with simple motion prediction using a Kalman filter to predict object location in the next frame.

While this approach is fast and effective in simple environments, it has limitations in complex scenarios with a high number of objects to track.

Other studies, such as #OTHEREFR , focused on using appearance information to match objects, by extracting feature vectors from detected objects and comparing them across frames.

Many studies #OTHEREFR have proposed various approaches to enhance Multi-Object Tracking (MOT) performance with low computational cost.

#OTHEREFR addressed the issue of the lack of a tracking dataset compared to the detection dataset.

Other studies, such as #OTHEREFR , focused on learning instance embeddings using 2D point cloud representations to avoid background features.

#OTHEREFR proposed a real-time Joint Detection and Embedding (JDE) system for MOT, which outperformed the Separate Detection and Embedding (SDE) system used in previous studies.

#OTHEREFR treated MOT as a multi-task learning problem of object detection and asso- 1Single-Camera Tracking is first conducted with a standard tracking-by-detection scheme, where the extracted frames are fed into a detector and a feature extractor to get the detections and re-id features for the preliminary tracking.

(2) Then Anchor-Guided Clustering will assign a global ID as well as fixing the ID switches for each trajectory.

(3) Lastly, Spatio-Temporal Consistency ID Reassignment will utilize the 2D human pose with camera self-calibration to reproject on the map for final post-processing.

Enhancing Multi-Camera People Tracking With Anchor-Guided Clustering And Spatio-Temporal Consistency Id Re-Assignment

Abstract: Multi-camera multiple people tracking has become an increasingly important area of research due to the growing demand for accurate and efficient indoor people tracking systems, particularly in settings such as retail, healthcare centers, and transit hubs. We proposed a novel multi-camera multiple people tracking method that uses anchor-guided clustering for cross-camera re-identification and spatio-temporal consistency for geometry-based crosscamera ID reassigning. Our approach aims to improve the accuracy of tracking by identifying key features that are unique to every individual and utilizing the overlap of views between cameras to predict accurate trajectories without needing the actual camera parameters. The method has demonstrated robustness and effectiveness in handling both synthetic and real-world data. The proposed method is evaluated on CVPR AI City Challenge 2023 dataset, achieving IDF1 of 95.36% with the first-place ranking in the challenge. The code is available at: https://github. com/ipl-uw/AIC23_Track1_UWIPL_ETRI. * indicates equal contributions.

Mots: Multi-Object Tracking And Segmentation

Abstract: This paper extends the popular task of multi-object tracking to multi-object tracking and segmentation (MOTS). Towards this goal, we create dense pixel-level annotations for two existing tracking datasets using a semi-automatic annotation procedure. Our new annotations comprise 65,213 pixel masks for 977 distinct objects (cars and pedestrians) in 10,870 video frames. For evaluation, we extend existing multi-object tracking metrics to this new task. Moreover, we propose a new baseline method which jointly addresses detection, tracking, and segmentation with a single convolutional network. We demonstrate the value of our datasets by achieving improvements in performance when training on MOTS annotations. We believe that our datasets, metrics and baseline will become a valuable resource towards developing multi-object tracking approaches that go beyond 2D bounding boxes. We make our annotations, code, and models available at this https URL.


Knowing the previous switching time t τ −1 and the parameters for the next stage, i.e., k τ +1 and β τ +1 , we follow the steps of #REFR to determine the optimal switching times in theory, and by this to analyze possible gains of the proposed scheme.

To optimize the speed of convergence, we group iterations into multiple stages τ ∈ N, where each stage is characterized by a certain number k τ of workers to wait for and a certain scale parameter β τ for the batch size.

We declare a new stage upon changing k and/or β.

The time when switching from stage τ to τ +1 is denoted by t τ , where t 0 := 0.

Consequently, the duration of a stage is t τ − t τ −1 .

In contrast to #OTHEREFR , the order statistics and the error floors now also depend on β. The result is given in Theorem 2.

For ease of notation, we omit the dependencies on k τ and β τ and refer to the order statistics µ kτ :n (β τ ) as µ τ .

We define φ τ := k τ β τ to reflect the batch size in stage τ . Theorem 2.

Given t τ −1 , the switching time t τ is given by

t τ = t τ −1 + µ τ α − log(µ τ ηLσ 2 grad (φ τ +1 − φ τ )) + log(µ τ +1 −µ τ ) + log(φ τ +1 (2cφ τ se(t τ −1 )−ηLσ 2 grad )) ,

Fast And Straggler-Tolerant Distributed Sgd With Reduced Computation Load

Abstract: In distributed machine learning, a central node outsources computationally expensive calculations to external worker nodes. The properties of optimization procedures like stochastic gradient descent (SGD) can be leveraged to mitigate the effect of unresponsive or slow workers called stragglers, that otherwise degrade the benefit of outsourcing the computation. This can be done by only waiting for a subset of the workers to finish their computation at each iteration of the algorithm. Previous works proposed to adapt the number of workers to wait for as the algorithm evolves to optimize the speed of convergence. In contrast, we model the communication and computation times using independent random variables. Considering this model, we construct a novel scheme that adapts both the number of workers and the computation load throughout the run-time of the algorithm. Consequently, we improve the convergence speed of distributed SGD while significantly reducing the computation load, at the expense of a slight increase in communication load. This project has received funding from the German Research Foundation (DFG) under Grant Agreement Nos. BI 2492/1-1 and WA 3907/7-1. 1 This holds true if the workers' data is generated from the same distribution. 2 Readers who are not familiar with GD algorithms are referred to Section II.

Adaptive Distributed Stochastic Gradient Descent For Minimizing Delay In The Presence Of Stragglers

Abstract: We consider the setting where a master wants to run a distributed stochastic gradient descent (SGD) algorithm on $n$ workers each having a subset of the data. Distributed SGD may suffer from the effect of stragglers, i.e., slow or unresponsive workers who cause delays. One solution studied in the literature is to wait at each iteration for the responses of the fastest $k<n$ workers before updating the model, where $k$ is a fixed parameter. The choice of the value of $k$ presents a trade-off between the runtime (i.e., convergence rate) of SGD and the error of the model. Towards optimizing the error-runtime trade-off, we investigate distributed SGD with adaptive $k$. We first design an adaptive policy for varying $k$ that optimizes this trade-off based on an upper bound on the error as a function of the wall-clock time which we derive. Then, we propose an algorithm for adaptive distributed SGD that is based on a statistical heuristic. We implement our algorithm and provide numerical simulations which confirm our intuition and theoretical analysis.


Using Pre-Activation of ResNet-20 (PreResNet-20) #REFR as an example, we show the relation of the memory cost of each block between width-wise and depth-wise training in Table 1 .

To simulate the non-IID setting with class imbalance, we follow #OTHEREFR to distribute each class to clients using the Dirichlet distribution with α(λ), where λ = {0.3, 1.0}.

Besides, we adopt pathological non-IID data partition β(Λ) that is used by the selected baselines -Het-eroFL and SplitMix #OTHEREFR , where each device has unique Λ labels with Λ = {2, 5} for CIFAR-10 while Λ = {10, 30} for CIFAR-100.

We note that the balanced data partition is applied by default, which makes each client holds the same number of examples.

The unbalanced αu(λ) non-IID, where clients may have a different amount of samples with different feature distribution skew, is also used to evaluate the stability of FEDEPTH.

We consider 100 clients in all experiments, and in Figure 7 2. Memory budgets.

We can see that if clients afford to train #OTHEREFR 6width PreResNet-20, they can train the full-size neural network via depth-wise training. B4 → B5,6 → B7,8,9 }.

Inspired by this example, we simulate three memory budget scenarios.

The memory budgets depend on the hidden channel shrinkage ratio, r = { 1 6 , 1 3 , 1 2 , 1}, and are uniformly distributed into clients.

It means 1/4 of clients can train a PreResNet-20 model within a maximal width of #OTHEREFR 6 -, 1 3 -, 1 2and full width, respectively.

Memory-Adaptive Depth-Wise Heterogenous Federated Learning

Abstract: Federated learning is a promising paradigm that allows multiple clients to collaboratively train a model without sharing the local data. However, the presence of heterogeneous devices in federated learning, such as mobile phones and IoT devices with varying memory capabilities, would limit the scale and hence the performance of the model could be trained. The mainstream approaches to address memory limitations focus on width-slimming techniques, where different clients train subnetworks with reduced widths locally and then the server aggregates the subnetworks. The global model produced from these methods suffers from performance degradation due to the negative impact of the actions taken to handle the varying subnetwork widths in the aggregation phase. In this paper, we introduce a memory-adaptive depth-wise learning solution in FL called FEDEPTH, which adaptively decomposes the full model into blocks according to the memory budgets of each client and trains blocks sequentially to obtain a full inference model. Our method outperforms state-of-the-art approaches, achieving 5% and more than 10% improvements in top-1 accuracy on CIFAR-10 and CIFAR-100, respectively. We also demonstrate the effectiveness of depth-wise fine-tuning on ViT.

Identity Mappings In Deep Residual Networks

Abstract: Deep residual networks have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62 % error) and CIFAR-100, and a 200-layer ResNet on ImageNet. Code is available at: https://github.com/KaimingHe/resnet-1k-layers.


Regarding ProbLog2, its performance in LUBM010 turned out to be too low to be further considered, as also observed by #REFR .

For LUBM, we present a comparison between LTGs and all the other engines.

For DBpedia, Claros, Smokers, YAGO, and WN18RR, the comparison does not consider ProbLog2 and Scallop.

Regarding Scallop, either it did not support some rules in the benchmarks or the data was too large to be loaded (Scallop's authors' highlight scalability as a direction for future work #OTHEREFR ).

For VQAR, we show a comparison only between LTGs w/ and Scallop: neither LTGs w/o nor vProbLog were able to compute the least parameterized model due to the combinatorial explosion of the derivations.

LTGs is the only technique that can compute the full least parameterized model despite this explosion.

Notice that computing the full lineage does not necessarily mean that we can always compute its exact probability, as the problem is #P-hard #OTHEREFR .

To deal with such cases, approximations can be employed either upfront, by reducing the size of the lineage (Scallop) or after the full lineage has been collected like #OTHEREFR .

Probabilistic Reasoning At Scale: Trigger Graphs To The Rescue

Abstract: The role of uncertainty in data management has become more prominent than ever before, especially because of the growing importance of machine learning-driven applications that produce large uncertain databases. A well-known approach to querying such databases is to blend rule-based reasoning with uncertainty. However, techniques proposed so far struggle with large databases. In this paper, we address this problem by presenting a new technique for probabilistic reasoning that exploits Trigger Graphs (TGs)-a notion recently introduced for the non-probabilistic setting. The intuition is that TGs can effectively store a probabilistic model by avoiding an explicit materialization of the lineage and by grouping together similar derivations of the same fact. Firstly, we show how TGs can be adapted to support the possible world semantics. Then, we describe techniques for efficiently computing a probabilistic model, and formally establish the correctness of our approach. We also present an extensive empirical evaluation using a prototype called LTGs. Our comparison against other leading engines shows that LTGs is not only faster, even against approximate reasoning techniques, but can also reason over probabilistic databases that existing engines cannot scale to.

Beyond The Grounding Bottleneck: Datalog Techniques For Inference In Probabilistic Logic Programs (Technical Report)

Abstract: State-of-the-art inference approaches in probabilistic logic programming typically start by computing the relevant ground program with respect to the queries of interest, and then use this program for probabilistic inference using knowledge compilation and weighted model counting. We propose an alternative approach that uses efficient Datalog techniques to integrate knowledge compilation with forward reasoning with a non-ground program. This effectively eliminates the grounding bottleneck that so far has prohibited the application of probabilistic logic programming in query answering scenarios over knowledge graphs, while also providing fast approximations on classical benchmarks in the field.


Second, there are many other forms of the convection term as shown in #REFR , whose theoretical and computational properties can be studied.

The numerical simulations show that the fully implicit scheme is more stable and accurate and efficient than the semi-implicit form.

Note it is the limit scheme, the one achieved when the iterative projections are convergent in each time step, that endows this method the properties such as stability, accuracy, and robustness to Reynolds numbers.

Thus, there are many ways this work can be improved and extended.

First, the scheme proposed in this work does not use any stabilization techniques mentioned in #OTHEREFR .

A simple and appealing stabilization is the grad-div method as mentioned in #OTHEREFR , also used in the augmented Lagrangian method #OTHEREFR , which has been shown to achieve robust error estimates for non-div-free FEMs.

Third, the convergence rate of the projection iterations for (4) and (5) is first order, and its comparison in efficiency with Newton's method with GMRES approach of solving the nonlinear system as in #OTHEREFR will be an interesting future work.

Finally, this iterative projection method is a completely new and efficient numerical method for the saddle point problem from the discretization of the time dependent Navier-Stokes equations.

It is an extension of the Uzawa method by adding only one Poisson equation in the pressure space, which is from the Hodge decomposition of the intermediate velocity.

The same as the Uzawa method, the method has a linear convergence rate.

However, normal mode analysis and numerical experiments reveal that with proper parameters, the iterative projection method is much faster than the Uzawa method when the viscosity is small or the Reynolds number is large.

Iterative Projection Method For Unsteady Navier-Stokes Equations With High Reynolds Numbers

Abstract: A new approach, iteration projection method, is proposed to solve the saddle point problem obtained after the full discretization of the unsteady Navier-Stokes equations. It is motivated by the Uzawa method, an iterative method for saddle point problems, and the convectional projection method for the Navier-Stokes equations that projects the intermediate velocity to the divergence free space only once per time step. The proposed method iterates projections in each time step with a proper convection form. We prove that the projection iterations converge with a certain parameter regime. Optimal iteration convergence can be achieved with the modulation of parameter values. This new method has several significant improvements over the Uzawa method and the projection method both theoretically and practically. First, when the iterative projections are fully convergent in each time step, the numerical velocity is weakly divergence free (pointwise divergence free in divergence free finite element methods), and the stability and error estimate are rigorously proven. With proper parameters, this method converges much faster than the Uzawa method. Second, numerical simulations show that with rather relaxed stopping criteria which require only a few iterations each time step, the numerical solution preserves stability and accuracy for high Reynolds numbers, where the convectional projection method would fail. Furthermore, this method retains the efficiency of the traditional projection method by decoupling the velocity and pressure fields, which splits the saddle point system into small elliptic problems. Three dimensional simulations with Taylor-Hood P2/P1 finite elements are presented to demonstrate the performance and efficiency of this method. More importantly, this method is a generic approach and thus there are many potential improvements and extensions of the iterative projection method, including utilization of various convection forms, association with stabilization techniques for high Reynolds numbers, applications to other saddle point problems.

On Conservation Laws Of Navier-Stokes Galerkin Discretizations

Abstract: We study conservation properties of Galerkin methods for the incompressible Navier–Stokes equations, without the divergence constraint strongly enforced. In typical discretizations such as the mixed finite element method, the conservation of mass is enforced only weakly, and this leads to discrete solutions which may not conserve energy, momentum, angular momentum, helicity, or vorticity, even though the physics of the Navier–Stokes equations dictate that they should. We aim in this work to construct discrete formulations that conserve as many physical laws as possible without utilizing a strong enforcement of the divergence constraint, and doing so leads us to a new formulation that conserves each of energy, momentum, angular momentum, enstrophy in 2D, helicity and vorticity (for reference, the usual convective formulation does not conserve most of these quantities). Several numerical experiments are performed, which verify the theory and test the new formulation.


This is particularly important where end users are accountable for the decisions made using an AI/ML system #REFR .

Through increasing transparency of an AI/ML system it may become apparent to the designers or the end users that the system is unfair #OTHEREFR , thereby possibly leading to internal or external pressure to increase fairness.

Furthermore, if the designers of an AI/ML system are required to demonstrate that the system is fair, an effective means for doing so is to increase transparency (and possibly explainability).

The appropriate definition of fairness for an AI/ML system will depend on the context of its use #OTHEREFR .

Disclosing how designers have understood and implemented fairness will assist the end users of an AI/ML system in evaluating whether it is likely to produce fair outcomes.

A possible tension between fairness and transparency is the potential for malicious or mischievous actors to exploit an AI/ML system to their own advantage based on the information disclosed about the system [34] , #OTHEREFR .

In such cases, transparency may need to be limited to protect the fairness of the system.

The designers should consider to whom the AI/ML system should be transparent, and what information should be provided #OTHEREFR .

A similar tension is created if aspects of an AI/ML system are protected as trade secrets #OTHEREFR .

To maintain accountability where public transparency is impossible, the AI/ML system may be made transparent to a trusted auditor for review #OTHEREFR (cf. Section III-J).

Implementing Responsible Ai: Tensions And Trade-Offs Between Ethics Aspects

Abstract: Many sets of ethics principles for responsible AI have been proposed to allay concerns about misuse and abuse of AI/ML systems. The underlying aspects of such sets of principles include privacy, accuracy, fairness, robustness, explainability, and transparency. However, there are potential tensions between these aspects that pose difficulties for AI/ML developers seeking to follow these principles. For example, increasing the accuracy of an AI/ML system may reduce its explainability. As part of the ongoing effort to operationalise the principles into practice, in this work we compile and discuss a catalogue of 10 notable tensions, trade-offs and other interactions between the underlying aspects. We primarily focus on two-sided interactions, drawing on support spread across a diverse literature. This catalogue can be helpful in raising awareness of the possible interactions between aspects of ethics principles, as well as facilitating well-supported judgements by the designers and developers of AI/ML systems.

Fairness And Accountability Design Needs For Algorithmic Support In High-Stakes Public Sector Decision-Making

Abstract: Calls for heightened consideration of fairness and accountability in algorithmically-informed public decisions-like taxation, justice, and child protection-are now commonplace. How might designers support such human values? We interviewed 27 public sector machine learning practitioners across 5 OECD countries regarding challenges understanding and imbuing public values into their work. The results suggest a disconnect between organisational and institutional realities, constraints and needs, and those addressed by current research into usable, transparent and 'discrimination-aware' machine learning-absences likely to undermine practical initiatives unless addressed. We see design opportunities in this disconnect, such as in supporting the tracking of concept drift in secondary data sources, and in building usable transparency tools to identify risks and incorporate domain knowledge, aimed both at managers and at the 'street-level bureaucrats' on the frontlines of public service. We conclude by outlining ethical challenges and future directions for collaboration in these high-stakes applications.


The Final Loss Is Calculated Using A Linear Combination Of Lp And Ls As Follows:

Looking at the results, we were able to replicate the performance metrics as mentioned by the author #REFR with an error threshold of 0.5% . A sample prediction is displayed in Fig. 6 .

Losses are then back propagated to the trainable parameters which are a combination of ResNet-50 encoder, Depth Decoder and Pose Decoder models. The optimizer used is Adam with a StepLR scheduler.

We don't use losses from depth maps to train the network.

The depth metrics -rms, squared relative, log_rms, a1, a2, a3 are as proposed #OTHEREFR .

where, Next, coming to the hyperparameters, we used a batch size of 12, lr=1e-4, a lr scheduler with steps every 15 batches for a total of 20 epochs. The training completed in approximately 8 hrs.

Self-Supervised Learning Based Depth Estimation From Monocular Images

Abstract: Depth Estimation has wide reaching applications in the field of Computer vision such as target tracking, augmented reality, and self-driving cars. The goal of Monocular Depth Estimation is to predict the depth map, given a 2D monocular RGB image as input. The traditional depth estimation methods are based on depth cues and used concepts like epipolar geometry. With the evolution of Convolutional Neural Networks, depth estimation has undergone tremendous strides. In this project, our aim is to explore possible extensions to existing SoTA Deep Learning based Depth Estimation Models and to see whether performance metrics could be further improved. In a broader sense, we are looking at the possibility of implementing Pose Estimation, Efficient Sub-Pixel Convolution Interpolation, Semantic Segmentation Estimation techniques to further enhance our proposed architecture and to provide fine-grained and more globally coherent depth map predictions. We also plan to do away with camera intrinsic parameters during training and apply weather augmentations to further generalize our model.

Digging Into Self-Supervised Monocular Depth Estimation

Abstract: Per-pixel ground-truth depth data is challenging to acquire at scale. To overcome this limitation, self-supervised learning has emerged as a promising alternative for training models to perform monocular depth estimation. In this paper, we propose a set of improvements, which together result in both quantitatively and qualitatively improved depth maps compared to competing self-supervised methods. Research on self-supervised monocular training usually explores increasingly complex architectures, loss functions, and image formation models, all of which have recently helped to close the gap with fully-supervised methods. We show that a surprisingly simple model, and associated design choices, lead to superior predictions. In particular, we propose (i) a minimum reprojection loss, designed to robustly handle occlusions, (ii) a full-resolution multi-scale sampling method that reduces visual artifacts, and (iii) an auto-masking loss to ignore training pixels that violate camera motion assumptions. We demonstrate the effectiveness of each component in isolation, and show high quality, state-of-the-art results on the KITTI benchmark.


To this end, we use the real-world LUNA16 dataset #REFR , which provides chest CTscans with corresponding lung segmentation mask.

To demonstrate the potential for real-world applications, we conduct an experiment for multimodal matching in the context of medical image analysis.

Based on the provided segmentation masks, we reconstruct 3D lung shapes represented as triangle meshes and simulate partial point clouds using a subset of slices of the volumetric images.

Since there are no ground-truth correspondences available, we restrict ourselves to qualitative results, which are shown in Fig.

7 , where it can be seen that reliable correspondences between different lung shapes can be obtained from our method.

Self-Supervised Learning For Multimodal Non-Rigid 3D Shape Matching

Abstract: The matching of 3D shapes has been extensively studied for shapes represented as surface meshes, as well as for shapes represented as point clouds. While point clouds are a common representation of raw real-world 3D data (e.g. from laser scanners), meshes encode rich and expressive topological information, but their creation typically requires some form of (often manual) curation. In turn, methods that purely rely on point clouds are unable to meet the matching quality of mesh-based methods that utilise the additional topological structure. In this work we close this gap by introducing a self-supervised multimodal learning strategy that combines mesh-based functional map regularisation with a contrastive loss that couples mesh and point cloud data. Our shape matching approach allows to obtain intramodal correspondences for triangle meshes, complete point clouds, and partially observed point clouds, as well as correspondences across these data modalities. We demonstrate that our method achieves state-of-the-art results on several challenging benchmark datasets even in comparison to recent supervised methods, and that our method reaches previously unseen cross-dataset generalisation ability. Our code is available at https://github.com/ dongliangcao/Self-Supervised-Multimodal-Shape-Matching.

Validation, Comparison, And Combination Of Algorithms For Automatic Detection Of Pulmonary Nodules In Computed Tomography Images: The Luna16 Challenge

Abstract: Automatic detection of pulmonary nodules in thoracic computed tomography (CT) scans has been an active area of research for the last two decades. However, there have only been few studies that provide a comparative performance evaluation of different systems on a common database. We have therefore set up the LUNA16 challenge, an objective evaluation framework for automatic nodule detection algorithms using the largest publicly available reference database of chest CT scans, the LIDC-IDRI data set. In LUNA16, participants develop their algorithm and upload their predictions on 888 CT scans in one of the two tracks: 1) the complete nodule detection track where a complete CAD system should be developed, or 2) the false positive reduction track where a provided set of nodule candidates should be classified. This paper describes the setup of LUNA16 and presents the results of the challenge so far. Moreover, the impact of combining individual systems on the detection performance was also investigated. It was observed that the leading solutions employed convolutional networks and used the provided set of nodule candidates. The combination of these solutions achieved an excellent sensitivity of over 95% at fewer than 1.0 false positives per scan. This highlights the potential of combining algorithms to improve the detection performance. Our observer study with four expert readers has shown that the best system detects nodules that were missed by expert readers who originally annotated the LIDC-IDRI data. We released this set of additional nodules for further development of CAD systems.

corresponding lung segmentation
chest CTscans


KG-BERT #REFR utilizes the power of pre-trained language models to capture more contextualized information during KGC.

There are emerging works bridging highly structured molecule graph with language descriptions, which facilitates human understanding of profound molecular knowledge and novel molecule exploration.

In the following, we briefly overview some representative works in these four topics.

Text To Knowledge Graph Generation. Li et al.

#OTHEREFR treat text-to-KG construction as a process of knowledge graph completion (KGC), where missing terms are progressively covered by inference.

A bilinear model and another DNN-based model are adopted to embed terms and compute score of arbitrary tuples for additive operation.

The idea is to represent triplets as textual sequences and models graph completion as a sequence classification problem by finetuned BERT model. Malaviya et al.

#OTHEREFR propose an approach incorporating graph convolutional network (GCN) for to extract more structural and semantic context.

It also tackles graph sparsity and scalability issues by introducing graph augmentation and progressive masking strategies.

Alternatively, another line of works #OTHEREFR directly query pre-trained language models to obtain a semantic knowledge network.

Specifically, language models are repeatedly prompted to predict the masked terms in cloze sentence to acquire relational knowledge.

A Comprehensive Survey Of Ai-Generated Content (Aigc): A History Of Generative Ai From Gan To Chatgpt

Abstract: Recently, ChatGPT, along with DALL-E-2 [1] and Codex [2],has been gaining significant attention from society. As a result, many individuals have become interested in related resources and are seeking to uncover the background and secrets behind its impressive performance. In fact, ChatGPT and other Generative AI (GAI) techniques belong to the category of Artificial Intelligence Generated Content (AIGC), which involves the creation of digital content, such as images, music, and natural language, through AI models. The goal of AIGC is to make the content creation process more efficient and accessible, allowing for the production of high-quality content at a faster pace. AIGC is achieved by extracting and understanding intent information from instructions provided by human, and generating the content according to its knowledge and the intent information. In recent years, large-scale models have become increasingly important in AIGC as they provide better intent extraction and thus, improved generation results. With the growth of data and the size of the models, the distribution that the model can learn becomes more comprehensive and closer to reality, leading to more realistic and high-quality content generation. This survey provides a comprehensive review on the history of generative models, and basic components, recent advances in AIGC from unimodal interaction and multimodal interaction. From the perspective of unimodality, we introduce the generation tasks and relative models of text and image. From the perspective of multimodality, we introduce the cross-application between the modalities mentioned above. Finally, we discuss the existing open problems and future challenges in AIGC.

Kg-Bert: Bert For Knowledge Graph Completion

Abstract: Knowledge graphs are important resources for many artificial intelligence tasks but often suffer from incompleteness. In this work, we propose to use pre-trained language models for knowledge graph completion. We treat triples in knowledge graphs as textual sequences and propose a novel framework named Knowledge Graph Bidirectional Encoder Representations from Transformer (KG-BERT) to model these triples. Our method takes entity and relation descriptions of a triple as input and computes scoring function of the triple with the KG-BERT language model. Experimental results on multiple benchmark knowledge graphs show that our method can achieve state-of-the-art performance in triple classification, link prediction and relation prediction tasks.

contextualized information
pre-trained language models


In our study, we utilize the SentencePiece #REFR tokenizer and experiment with dictionary sizes of 1000, 4000, and 7000 sub-words to evaluate their respective impacts on NMT performance.

(1) no 0.62 #OTHEREFR yes 0.98 Table 7 . Validation scores during the exploration of the model architecture.

As previously discussed in Section 3.2, the selection of appropriate vocabulary size is a trade-off between enhancing the representation of rare words or producing shorter sequences.

Results in Table 6 show that larger dictionary improves results, in our experiments it increases 0.14 points of reduced BLEU score.

Thus, we always use a Sentencepiece tokenizer with a vocabulary of 7000 sub-words.

A current observation in Transformers is that increasing the number of parameters will improve the performance.

However, in low-resource languages, increasing the number of model parameters can hinder performance #OTHEREFR .

We study the effect of using a deeper and shallower Transformer by changing the number of layers in the encoder and decoder, the number of attention heads, the feed-forward layer dimension, and embedding dimensions.

Sign Language Translation From Instructional Videos

Abstract: The advances in automatic sign language translation (SLT) to spoken languages have been mostly benchmarked with datasets of limited size and restricted domains. Our work advances the state of the art by providing the first baseline results on How2Sign, a large and broad dataset. We train a Transformer over I3D video features, using the reduced BLEU as a reference metric for validation, instead of the widely used BLEU score. We report a result of 8.03 on the BLEU score, and publish the first open-source implementation of its kind to promote further advances.

Sentencepiece: A Simple And Language Independent Subword Tokenizer And Detokenizer For Neural Text Processing

Abstract: This paper describes SentencePiece, a language-independent subword tokenizer and detokenizer designed for Neural-based text processing, including Neural Machine Translation. It provides open-source C++ and Python implementations for subword units. While existing subword segmentation tools assume that the input is pre-tokenized into word sequences, SentencePiece can train subword models directly from raw sentences, which allows us to make a purely end-to-end and language independent system. We perform a validation experiment of NMT on English-Japanese machine translation, and find that it is possible to achieve comparable accuracy to direct subword training from raw sentences. We also compare the performance of subword training and segmentation with various configurations. SentencePiece is available under the Apache 2 license at https://github.com/google/sentencepiece.


Figure 2 also indicates that indices similar to those in #REFR , using a PCA as a first compression and IVFPQ as a quantizer, have some drop off in performance, as with the CLIP only indices.

Figure 2 presents the retrieval benchmarks for the L400M and L2B datasets.

We assessed each method across a range of benchmarks and created Pareto fronts by varying the number of chunks for product quantization (i.e., M = 2, 4, 8) and the n probe parameter (τ = 1, 2, 4).

Notably, MSE and SNIP performed similarly on image similarity search, while CLIP-only loss networks consistently performed worse.

However, as mentioned, the CLIP networks (both CLIP and SNIP) do perform better on multi-modal tasks.

Thus, SNIP can be seen as a best of both worlds descriptor for these types of indices.

On The De-Duplication Of Laion-2B

Abstract: Generative models, such as DALL-E, Midjourney, and Stable Diffusion, have societal implications that extend beyond the field of computer science. These models require large image databases like LAION-2B, which contain two billion images. At this scale, manual inspection is difficult and automated analysis is challenging. In addition, recent studies show that duplicated images pose copyright problems for models trained on LAION2B, which hinders its usability. This paper proposes an algorithmic chain that runs with modest compute, that compresses CLIP features to enable efficient duplicate detection, even for vast image volumes. Our approach demonstrates that roughly 700 million, or about a third of LAION-2B's images, are duplicates. Our method also provides the histograms of duplication on this dataset, which we use to reveal more examples of verbatim copies by Stable Diffusion and further justify the approach. The current version of the de-duplicated set can be found at github.com/ryanwebster90/snip-dedup.

Billion-Scale Similarity Search With Gpus

Abstract: Similarity search finds application in specialized database systems handling complex data such as images or videos, which are typically represented by high-dimensional features and require specific indexing structures. This paper tackles the problem of better utilizing GPUs for this task. While GPUs excel at data-parallel tasks, prior approaches are bottlenecked by algorithms that expose less parallelism, such as k-min selection, or make poor use of the memory hierarchy. ::: We propose a design for k-selection that operates at up to 55% of theoretical peak performance, enabling a nearest neighbor implementation that is 8.5x faster than prior GPU state of the art. We apply it in different similarity search scenarios, by proposing optimized design for brute-force, approximate and compressed-domain search based on product quantization. In all these setups, we outperform the state of the art by large margins. Our implementation enables the construction of a high accuracy k-NN graph on 95 million images from the Yfcc100M dataset in 35 minutes, and of a graph connecting 1 billion vectors in less than 12 hours on 4 Maxwell Titan X GPUs. We have open-sourced our approach for the sake of comparison and reproducibility.


We use Ray #REFR to implement parallel sampling and use 16 workers with different seeds to sample experiences in parallel across four scenarios. Each scenario is assigned 4 workers.

The maximum steps for one episode are T and when an agent survives T steps in the environment, we call it "timeout".

An agent is marked as a success only when it passes through the interaction zone 1 before the timeout occurs.

Training Framework: We use PyTorch 1.9.1 2 to implement policy architectures and use Adam #OTHEREFR to optimize policies. All models are trained with one Nvidia RTX 3090.

Detailed parameters of Soft-Actor-Critic (SAC) are shown in Table I .

We use precisely the same parameters for both singleand multi-agent policy learning.

Zero-Shot Transfer Learning Of Driving Policy Via Socially Adversarial Traffic Flow

Abstract: Acquiring driving policies that can transfer to unseen environments is challenging when driving in dense traffic flows. The design of traffic flow is essential and previous studies are unable to balance interaction and safety-criticism. To tackle this problem, we propose a socially adversarial traffic flow. We propose a Contextual Partially-Observable Stochastic Game to model traffic flow and assign Social Value Orientation (SVO) as context. We then adopt a two-stage framework. In Stage 1, each agent in our socially-aware traffic flow is driven by a hierarchical policy where upper-level policy communicates genuine SVOs of all agents, which the lower-level policy takes as input. In Stage 2, each agent in the socially adversarial traffic flow is driven by the hierarchical policy where upper-level communicates mistaken SVOs, taken by the lower-level policy trained in Stage 1. Driving policy is adversarially trained through a zero-sum game formulation with upper-level policies, resulting in a policy with enhanced zero-shot transfer capability to unseen traffic flows. Comprehensive experiments on cross-validation verify the superior zero-shot transfer performance of our method.

Ray: A Distributed Framework For Emerging Ai Applications

Abstract: The next generation of AI applications will continuously interact with the environment and learn from these interactions. These applications impose new and demanding systems requirements, both in terms of performance and flexibility. In this paper, we consider these requirements and present Ray--a distributed system to address them. Ray implements a unified interface that can express both task-parallel and actor-based computations, supported by a single dynamic execution engine. To meet the performance requirements, Ray employs a distributed scheduler and a distributed and fault-tolerant store to manage the system's control state. In our experiments, we demonstrate scaling beyond 1.8 million tasks per second and better performance than existing specialized systems for several challenging reinforcement learning applications.


For baseline, we use pretrained and finetuned until convergence BART model #REFR with cross entropy loss.

First, we evaluate the performance of SentenceBERT-based post-processing method. Then, we compared different aspects of unlikelihood training objectives.

Lastly, we show the efficacy of our novel block loss term in the setting of sequence-level unlikelihood objective.

Token-level unlikelihood objective is applied during fine-tuning of pretrained BART model.

Sequence-level unlikelihood objective is applied 50% of the time during fine-tuning of pretrained BART model, with the other 50% of the time using basic likelihood loss.

Teacher forcing #OTHEREFR is applied during training to ensure a faster and more stable convergence.

We use beam search as the decoding strategy with beam size 5 across all experiments.

All training is done on an Azure ML Compute instance with 4 Nvidia V100 GPUs.

Multi-Aspect Repetition Suppression And Content Moderation Of Large Language Models Authors

Abstract: Natural language generation is one of the most impactful fields in NLP, and recent years have witnessed its evolution brought about by large language models (LLMs). As the key instrument for writing assistance applications, they are generally prone to replicating or extending offensive content provided in the input. In low-resource data regime, they can also lead to repetitive outputs (Holtzman et al., 2019) [1]. Usually, offensive content and repetitions are mitigated with post-hoc methods, including n-gram level blocklists, top-k and nucleus sampling. In this paper, we introduce a combination of exact and nonexact repetition suppression using token and sequence level unlikelihood loss, repetition penalty during training, inference, and post-processing respectively. We further explore multi-level unlikelihood loss to the extent that it endows the model with abilities to avoid generating offensive words and phrases from the beginning. Finally, with comprehensive experiments, we demonstrate that our proposed methods work exceptionally in controlling the repetition and content quality of LLM outputs.

Bart: Denoising Sequence-To-Sequence Pre-Training For Natural Language Generation, Translation, And Comprehension

Abstract: We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and many other more recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of the original sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa with comparable training resources on GLUE and SQuAD, achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 6 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also report ablation experiments that replicate other pretraining schemes within the BART framework, to better measure which factors most influence end-task performance.


For our learned hypernymization approach, we use the T5-small #REFR language model, which we fine-tune for 5 epochs on one Google Cloud TPU with learning rate of 0.0001, batch size of 12, and gradient accumulation step of 8.

To perform named entity recognition, we use the off-the-shelf Flair tagger #OTHEREFR , available from Hugging Face #OTHEREFR .

Given the affinity of caption reconstruction with summarization, we add the "summarize:" prompt to the beginning of each input caption.

To create the artificial, NE-enriched captions, we set = 0.7, − = = 0.3 (Section 3.2).

For pre-training, we adapt the code from #OTHEREFR with a ResNet50 #OTHEREFR as the visual encoder and a frozen BERT-base model #OTHEREFR as the text encoder.

We set the learning rate to 0.001 (decreased by a factor of 10 at 50% and 80% of training).

We set the batch size to 9 and we train using 3 Nvidia Titan X GPUs.

Hypernymization Of Named Entity-Rich Captions For Grounding-Based Multi-Modal Pretraining

Abstract: Figure 1: Our key ideas: an object mentioned using a named entity cannot be well grounded with an image. We thus introduce two methods to carry out hypernymization on the caption and show better grounding performance between the image and the hypernymized captions.

Exploring The Limits Of Transfer Learning With A Unified Text-To-Text Transformer

Abstract: Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.

learned hypernymization approach
T5-small language model


Inference Of Yarn Characteristics From Input Images

We considered the saliency maps #REFR of networks trained to predict the set of raw yarn's parameters and the set of the flyaway parameters with two independent models.

We model the prediction of the parameters for our procedural yarn model from images as a regression problem.

Our training and validation dataset consists of annotated synthetic yarn images that we use to train a model that allows inferring yarn parameters from novel images of yarns not seen during training or validation.

Before providing details of our respective approach (see Section 4.1), we motivate our choice of a suitable network architecture that is capable of handling the challenging nature of the underlying problem.

An entry m i, j in a saliency map for a model f that has been trained on a subset of P parameters is the maximum derivative of the average value of the predicted parameters with respect to a pixel x i, j,c in the input image over the color channels c, i.e.

In contrast to saliency maps that have been proposed within the context of classification networks, we consider the derivative of the mean of the predicted parameters because we need to investigate the effect of a pixel on the entire subset on which the network was trained.

The saliency maps (Figure 7 ) for the network trained to predict the raw yarn parameters illustrate a higher susceptibility to changes in the yarn center region of the image.

On the other hand, the saliency maps for the network that predicts the flyaway parameters indicate that such a network exhibits a higher sensitivity towards the border regions within the input images.

Neural Inverse Procedural Modeling Of Knitting Yarns From Images

Abstract: We investigate the capabilities of neural inverse procedural modeling to infer high-quality procedural yarn models with fiber-level details from single images of depicted yarn samples. While directly inferring all parameters of the underlying yarn model based on a single neural network may seem an intuitive choice, we show that the complexity of yarn structures in terms of twisting and migration characteristics of the involved fibers can be better encountered in terms of ensembles of networks that focus on individual characteristics. We analyze the effect of different loss functions including a parameter loss to penalize the deviation of inferred parameters to ground truth annotations, a reconstruction loss to enforce similar statistics of the image generated for the estimated parameters in comparison to training images as well as an additional regularization term to explicitly penalize deviations between latent codes of synthetic images and the average latent code of real images in the encoder's latent space. We demonstrate that the combination of a carefully designed parametric, procedural yarn model with respective network ensembles as well as loss functions even allows robust parameter inference when solely trained on synthetic data. Since our approach relies on the availability of a yarn database with parameter annotations and we are not aware of such a respectively available dataset, we additionally provide, to the best of our knowledge, the first dataset of yarn images with annotations regarding the respective yarn parameters. For this purpose, we use a novel yarn generator that improves the realism of the produced results over previous approaches.

Deep Inside Convolutional Networks: Visualising Image Classification Models And Saliency Maps

Abstract: This paper addresses the visualisation of image classification models, learnt using deep Convolutional Networks (ConvNets). We consider two visualisation techniques, based on computing the gradient of the class score with respect to the input image. The first one generates an image, which maximises the class score [5], thus visualising the notion of the class, captured by a ConvNet. The second technique computes a class saliency map, specific to a given image and class. We show that such maps can be employed for weakly supervised object segmentation using classification ConvNets. Finally, we establish the connection between the gradient-based ConvNet visualisation methods and deconvolutional networks [13].


It was followed by a series of improvements, such as Cascade R-CNN #REFR , etc.

• We show the impact of MSIL on regression and classification by using relevant information through visualization.

And experiments on two datasets verify that MSIL can leverage the relevant information between semantics to improves the performance of different methods.

#OTHEREFR further improves the efficiency by simplifying the pooling operation with RoI pool #OTHEREFR , and integrating the regressor into the neural network.

Faster R-CNN #OTHEREFR is the paradigm of the R-CNN series, which proposed the Region Proposal Network (RPN) based on subsequent improvements.

Discarding the previous selective search #OTHEREFR algorithm, RPN uses neural networks to generate candidate boxes.

It is worth mentioning that the R-CNN series has always used one branch to predict position and class.

Despite various adjustments made by subsequent improvements, regression and classification have always been coupled together. Later, Double-Head #OTHEREFR discussed in this regard.

YoloV1 #OTHEREFR is the first single-stage detector, which divides the image into individual lattices, and then predicts all lattices at once.

It is worth mentioning that YoloV1 only uses points near the center of the object to predict edges.

Because these points are thought to produce higher quality detections. However, this strategy leads to low recall of YoloV1.

Multi-Semantic Interactive Learning For Object Detection

Abstract: Single-branch object detection methods use shared features for localization and classification, yet the shared features are not fit for the two different tasks simultaneously. Multi-branch object detection methods usually use different features for localization and classification separately, ignoring the relevance between different tasks. Therefore, we propose multi-semantic interactive learning (MSIL) to mine the semantic relevance between different branches and extract multi-semantic enhanced features of objects. MSIL first performs semantic alignment of regression and classification branches, then merges the features of different branches by semantic fusion, finally extracts relevant information by semantic separation and passes it back to the regression and classification branches respectively. More importantly, MSIL can be integrated into existing object detection nets as a plug-and-play component. Experiments on the MS COCO, and Pascal VOC datasets show that the integration of MSIL with existing algorithms can utilize the relevant information between semantics of different tasks and achieve better performance.

Cascade R-Cnn: Delving Into High Quality Object Detection

Abstract: In object detection, an intersection over union (IoU) threshold is required to define positives and negatives. An object detector, trained with low IoU threshold, e.g. 0.5, usually produces noisy detections. However, detection performance tends to degrade with increasing the IoU thresholds. Two main factors are responsible for this: 1) overfitting during training, due to exponentially vanishing positive samples, and 2) inference-time mismatch between the IoUs for which the detector is optimal and those of the input hypotheses. A multi-stage object detection architecture, the Cascade R-CNN, is proposed to address these problems. It consists of a sequence of detectors trained with increasing IoU thresholds, to be sequentially more selective against close false positives. The detectors are trained stage by stage, leveraging the observation that the output of a detector is a good distribution for training the next higher quality detector. The resampling of progressively improved hypotheses guarantees that all detectors have a positive set of examples of equivalent size, reducing the overfitting problem. The same cascade procedure is applied at inference, enabling a closer match between the hypotheses and the detector quality of each stage. A simple implementation of the Cascade R-CNN is shown to surpass all single-model object detectors on the challenging COCO dataset. Experiments also show that the Cascade R-CNN is widely applicable across detector architectures, achieving consistent gains independently of the baseline detector strength. The code will be made available at this https URL.


The PPO agent and base controllers, i.e., H ∞ and PIDs, are implemented based on Pytorch #REFR to facilitate vectorized computing.

We perform all experiments on a single computer (AMD Ryzen Threadripper 3960X, 24x 3.8GHz, NVIDIA GeForce RTX2080 Ti, 11GB).

Both TurtleSim and the blimp simulator are implemented based on the ROS and the latter is integrated with the Gazebo SITL simulation.

We compare our robust H ∞ -RL controller to the previous PID-RL baseline in the TurtleSim (Sec. III-B.1) and blimp simulator (Sec. III-B.2).

1) H ∞ -PPO agent: our proposed approach 2) PID-PPO agent: previous approach #OTHEREFR .

We reimplemented with a randomized mixing factor q and a servo control so that it has the chance to challenge our more difficult waypoint following task.

Note when the mixing factor is q = 1 we recover the base control, and q = 0 corresponds to the pure PPO agent.

Autonomous Blimp Control Via H ∞ Robust Deep Residual Reinforcement Learning

Abstract: Due to their superior energy efficiency, blimps may replace quadcopters for long-duration aerial tasks. However, designing a controller for blimps to handle complex dynamics, modeling errors, and disturbances remains an unsolved challenge. One recent work combines reinforcement learning (RL) and a PID controller to address this challenge and demonstrates its effectiveness in real-world experiments. In the current work, we build on that using an H∞ robust controller to expand the stability margin and improve the RL agent's performance. Empirical analysis of different mixing methods reveals that the resulting H∞-RL controller outperforms the prior PID-RL combination and can handle more complex tasks involving intensive thrust vectoring. We provide our code as open-source at https://github.com/robot-perception-group/ robust_deep_residual_blimp.

Pytorch: An Imperative Style, High-Performance Deep Learning Library

Abstract: Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it provides an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. ::: In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. ::: We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several common benchmarks.


When we further adjust the validation resize size and train crop size, it can not only get a slightly higher performance gain (3.34% vs 3.13%) but also speed up the training process, similar to the conclusions in #REFR .

To verify the effectiveness of the proposed three strategies (i.e., KL-+Snet6, SITrans, and InterSST), we progressively add these components to the original ST and compare their main network performance. All experiments are performed with ResNet50 on ImageNet.

As shown in Table 3 , original ST can already improve the Top-1 classification results of CT by 0.86%.

When adding KL-+Snet6 and SITrans to ST respectively, the Top-1 performance improvement increases to 1.83% and 1.86% respectively.

When adding both KL-+Snet6 and SITrans to ST, the Top-1 performance improvement is up to 3.13%.

Then, when we add the above strategies and InterSSR to ST, the Top-1 performance gain is boosted to 3.80%.

In addition, the proposed stimulative training and three simple-but-effective strategies are complementary to other training ingredients in essence.

To verify this, on the basis of the above strategies, when we further add a data augmentation strategy called uniform augment #OTHEREFR and extend the training epoch to 400, the Top-1 performance gain continues to increase to 4.27%.

In summary, the proposed stimulative training and its associated three strategies, although simple, can remarkably improve the performance of residual networks and serve as a strong booster for the current training scheme.

Stimulative Training++: Go Beyond The Performance Limits Of Residual Networks

Abstract: Residual networks have shown great success and become indispensable in recent deep neural network models. In this work, we aim to re-investigate the training process of residual networks from a novel social psychology perspective of loafing, and further propose a new training scheme as well as three improved strategies for boosting residual networks beyond their performance limits. Previous research has suggested that residual networks can be considered as ensembles of shallow networks, which implies that the final performance of a residual network is influenced by a group of subnetworks. We identify a previously overlooked problem that is analogous to social loafing, where subnetworks within a residual network are prone to exert less effort when working as part of a group compared to working alone. We define this problem as network loafing. Similar to the decreased individual productivity and overall performance as demonstrated in society, network loafing inevitably causes sub-par performance. Inspired by solutions from social psychology, we first propose a novel training scheme called stimulative training, which randomly samples a residual subnetwork and calculates the KL divergence loss between the sampled subnetwork and the given residual network for extra supervision. In order to unleash the potential of stimulative training, we further propose three simple-yet-effective strategies, including a novel KL-loss that only aligns the network logits direction, random smaller inputs for subnetworks, and inter-stage sampling rules. Comprehensive experiments and analysis verify the effectiveness of stimulative training as well as its three improved strategies. For example, the proposed method can boost the performance of ResNet50 on ImageNet to 80.5% Top1 accuracy without using any extra data, model, trick, or changing the structure. With only uniform augment, the performance can be further improved to 81.0% Top1 accuracy, better than the best training recipes provided by Timm library and PyTorch official version. We also verify its superiority on various typical models, datasets, and tasks and give some theoretical analysis. As such, we advocate utilizing the proposed method as a general and next-generation technology to train residual networks. The code is available at https://github.com/Sunshine-Ye/NIPS22-ST.

Fixing The Train-Test Resolution Discrepancy: Fixefficientnet

Abstract: This note complements the paper"Fixing the train-test resolution discrepancy"that introduced the FixRes method. First, we show that this strategy is advantageously combined with recent training recipes from the literature. Most importantly, we provide new results for the EfficientNet architecture. The resulting network, called FixEfficientNet, significantly outperforms the initial architecture with the same number of parameters. For instance, our FixEfficientNet-B0 trained without additional training data achieves 79.3% top-1 accuracy on ImageNet with 5.3M parameters. This is a +0.5% absolute improvement over the Noisy student EfficientNet-B0 trained with 300M unlabeled images and +1.7% compared to the EfficientNet-B0 trained with adversarial examples. An EfficientNet-L2 pre-trained with weak supervision on 300M unlabeled images and further optimized with FixRes achieves 88.5% top-1 accuracy (top-5: 98.7%), which establishes the new state of the art for ImageNet with a single crop.


Panoptic segmentation #REFR of images is a core computer vision task that jointly solves two related problems -semantic segmentation and instance segmentation.

With the rise of robotics and emerging autonomous driving markets, efficient visual perception stacks are in high demand.

However, large-scale supervised learning of panoptic segmentation #OTHEREFR is prohibitively expensive as it requires dense annotations for both semantics and instances, which require time-consuming manual labeling.

ground truth CVRN PQ=32.1 EDAPS PQ=41.2 Figure 1 : EDAPS is an architecture and a collection of recipes, designed specifically for Domain-Adaptive Panoptic Segmentation.

It demonstrates a significant improvement over the prior art on the challenging synthetic-to-real benchmarks.

As shown above, it is better than CVRN #OTHEREFR on SYNTHIA → Cityscapes by 9 mPQ.

Edaps: Enhanced Domain-Adaptive Panoptic Segmentation

Abstract: With autonomous industries on the rise, domain adaptation of the visual perception stack is an important research direction due to the cost savings promise. Much prior art was dedicated to domain-adaptive semantic segmentation in the synthetic-to-real context. Despite being a crucial output of the perception stack, panoptic segmentation has been largely overlooked by the domain adaptation community. Therefore, we revisit well-performing domain adaptation strategies from other fields, adapt them to panoptic segmentation, and show that they can effectively enhance panoptic domain adaptation. Further, we study the panoptic network design and propose a novel architecture (EDAPS) designed explicitly for domain-adaptive panoptic segmentation. It uses a shared, domain-robust transformer encoder to facilitate the joint adaptation of semantic and instance features, but task-specific decoders tailored for the specific requirements of both domain-adaptive semantic and instance segmentation. As a result, the performance gap seen in challenging panoptic benchmarks is substantially narrowed. EDAPS significantly improves the state-of-the-art performance for panoptic segmentation UDA by a large margin of 25% on SYNTHIA-to-Cityscapes and even 72% on the more challenging SYNTHIA-to-Mapillary Vistas. The implementation is available at https://github.com/susaha/edaps.

Panoptic Segmentation

Abstract: We propose and study a task we name panoptic segmentation (PS). Panoptic segmentation unifies the typically distinct tasks of semantic segmentation (assign a class label to each pixel) and instance segmentation (detect and segment each object instance). The proposed task requires generating a coherent scene segmentation that is rich and complete, an important step toward real-world vision systems. While early work in computer vision addressed related image/scene parsing tasks, these are not currently popular, possibly due to lack of appropriate metrics or associated recognition challenges. To address this, we propose a novel panoptic quality (PQ) metric that captures performance for all classes (stuff and things) in an interpretable and unified manner. Using the proposed metric, we perform a rigorous study of both human and machine performance for PS on three existing datasets, revealing interesting insights about the task. The aim of our work is to revive the interest of the community in a more unified view of image segmentation. For more analysis and up-to-date results, please check the arXiv version of the paper: {\small\url{https://arxiv.org/abs/1801.00868}}.


On the one hand they appear in the component-by-component (CBC) construction for determining the lattice generating vector z gen , which in turn determines the interpolation points through #REFR .

In implementations of the lattice-based kernel interpolant of #OTHEREFR the weights γ u play a dual role.

On the other hand they appear in the formula (10) for the kernel.

In both roles only special forms of weights are feasible, given that there are 2 s subsets u ⊆ {1 : s}.

The SPOD weights described above are feasible (and were used in the computations in #OTHEREFR ), but encounter two computational bottlenecks:

The CBC construction used to obtain the generating vector z gen using SPOD weights has cost O(s n log(n) + s 3 α 2 n), see Table 1 and #OTHEREFR . 2.

Lattice-Based Kernel Approximation And Serendipitous Weights For Parametric Pdes In Very High Dimensions

Abstract: We describe a fast method for solving elliptic partial differential equations (PDEs) with uncertain coefficients using kernel interpolation at a lattice point set. By representing the input random field of the system using the model proposed by Kaarnioja, Kuo, and Sloan (SIAM J. Numer. Anal. 2020), in which a countable number of independent random variables enter the random field as periodic functions, it was shown by Kaarnioja, Kazashi, Kuo, Nobile, and Sloan (Numer. Math. 2022) that the lattice-based kernel interpolant can be constructed for the PDE solution as a function of the stochastic variables in a highly efficient manner using fast Fourier transform (FFT). In this work, we discuss the connection between our model and the popular "affine and uniform model" studied widely in the literature of uncertainty quantification for PDEs with uncertain coefficients. We also propose a new class of weights entering the construction of the kernel interpolant-serendipitous weights-which dramatically improve the computational performance of the kernel interpolant for PDE problems with uncertain coefficients, and allow us to tackle function approximation problems up to very high dimensionalities. Numerical experiments are presented to showcase the performance of the serendipitous weights.

Analysis Of Quasi-Monte Carlo Methods For Elliptic Eigenvalue Problems With Stochastic Coefficients

Abstract: We consider the forward problem of uncertainty quantification for the generalised Dirichlet eigenvalue problem for a coercive second order partial differential operator with random coefficients, motivated by problems in structural mechanics, photonic crystals and neutron diffusion. The PDE coefficients are assumed to be uniformly bounded random fields, represented as infinite series parametrised by uniformly distributed i.i.d. random variables. The expectation of the fundamental eigenvalue of this problem is computed by (a) truncating the infinite series which define the coefficients; (b) approximating the resulting truncated problem using lowest order conforming finite elements and a sparse matrix eigenvalue solver; and (c) approximating the resulting finite (but high dimensional) integral by a randomly shifted quasi-Monte Carlo lattice rule, with specially chosen generating vector. We prove error estimates for the combined error, which depend on the truncation dimension s, the finite element mesh diameter h, and the number of quasi-Monte Carlo samples N. Under suitable regularity assumptions, our bounds are of the particular form \({\mathcal {O}}(h^2 + N^{-1 + \delta })\), where \(\delta > 0\) is arbitrary and the hidden constant is independent of the truncation dimension, which needs to grow as \(h\rightarrow 0\) and \(N \rightarrow \infty \). As for the analogous PDE source problem, the conditions under which our error bounds hold depend on a parameter \(p \in (0, 1)\) representing the summability of the terms in the series expansions of the coefficients. Although the eigenvalue problem is nonlinear, which means it is generally considered harder than the source problem, in almost all cases (\(p \ne 1\)) we obtain error bounds that converge at the same rate as the corresponding rate for the source problem. The proof involves a detailed study of the regularity of the fundamental eigenvalue as a function of the random parameters. As a key intermediate result in the analysis, we prove that the spectral gap (between the fundamental and the second eigenvalues) is uniformly positive over all realisations of the random problem.


For category-level object detection, we train on category-level sketch/photo pairs in QuickDraw-Extended #REFR having 330k sketches and 204k photos from 110 categories.

Dataset We train our object detector using existing crosscategory FG-SBIR dataset -Sketchy #OTHEREFR that contains 125 categories, each with 100 photos.

Every photo in #OTHEREFR has at least 5 instance-level paired sketches.

To evaluate finegrained object detection, we use SketchyCOCO #OTHEREFR comprising of natural images in MS-COCO #OTHEREFR with instancelevel paired sketches. Following Liu et al.

#OTHEREFR , we select 1, 225 sketch/photo pairs from SketchyCOCO #OTHEREFR with at least one foreground sketched object.

We filter the overlapping categories of in SketchyCOCO #OTHEREFR from Sketchy #OTHEREFR to measure true zero-shot performance.

Following #OTHEREFR , we evaluate on a subset of standard object detection datasets like PASCAL-VOC #OTHEREFR and MS-COCO #OTHEREFR that have 20 and 56 overlapping categories in QuickDraw #OTHEREFR .

Implementation Details Our model is implemented in PyTorch on a 11GB Nvidia RTX 2080-Ti GPU.

First, we train a generalised cross-category FG-SBIR with image size (224×224) by adapting CLIP with ViT #OTHEREFR backbone (ViT-B/32 weights) using prompt learning #OTHEREFR .

The prompts (P = 3) are trained with triplet loss #OTHEREFR , margin µ = 0.3, Adam optimiser with learning rate 1e − 4 for 60 epochs, and batch size 64.

Our object detection pipeline is build using Detectron2 [93] .

What Can Human Sketches Do For Object Detection?

Abstract: Sketches are highly expressive, inherently capturing subjective and fine-grained visual cues. The exploration of such innate properties of human sketches has, however, been limited to that of image retrieval. In this paper, for the first time, we cultivate the expressiveness of sketches but for the fundamental vision task of object detection. The end result is a sketch-enabled object detection framework that detects based on what you sketch-that "zebra" (e.g., one that is eating the grass) in a herd of zebras (instance-aware detection), and only the part (e.g., "head" of a "zebra") that you desire (part-aware detection). We further dictate that our model works without (i) knowing which category to expect at testing (zero-shot) and (ii) not requiring additional bounding boxes (as per fully supervised) and class labels (as per weakly supervised). Instead of devising a model from the ground up, we show an intuitive synergy between foundation models (e.g., CLIP) and existing sketch models build for sketch-based image retrieval (SBIR), which can already elegantly solve the task-CLIP to provide model generalisation, and SBIR to bridge the (sketch→photo) gap. In particular, we first perform independent prompting on both sketch and photo branches of an SBIR model to build highly generalisable sketch and photo encoders on the back of the generalisation ability of CLIP. We then devise a training paradigm to adapt the learned encoders for object detection, such that the region embeddings of detected boxes are aligned with the sketch and photo embeddings from SBIR. Evaluating our framework on standard object detection datasets like PASCAL-VOC and MS-COCO outperforms both supervised (SOD) and weakly-supervised object detectors (WSOD) on zero-shot setups. Project

Doodle To Search: Practical Zero-Shot Sketch-Based Image Retrieval

Abstract: In this paper, we investigate the problem of zero-shot sketch-based image retrieval (ZS-SBIR), where human sketches are used as queries to conduct retrieval of photos from unseen categories. We importantly advance prior arts by proposing a novel ZS-SBIR scenario that represents a firm step forward in its practical application. The new setting uniquely recognizes two important yet often neglected challenges of practical ZS-SBIR, (i) the large domain gap between amateur sketch and photo, and (ii) the necessity for moving towards large-scale retrieval. We first contribute to the community a novel ZS-SBIR dataset, QuickDraw-Extended, that consists of 330,000 sketches and 204,000 photos spanning across 110 categories. Highly abstract amateur human sketches are purposefully sourced to maximize the domain gap, instead of ones included in existing datasets that can often be semi-photorealistic. We then formulate a ZS-SBIR framework to jointly model sketches and photos into a common embedding space. A novel strategy to mine the mutual information among domains is specifically engineered to alleviate the domain gap. External semantic knowledge is further embedded to aid semantic transfer. We show that, rather surprisingly, retrieval performance significantly outperforms that of state-of-the-art on existing datasets that can already be achieved using a reduced version of our model. We further demonstrate the superior performance of our full model by comparing with a number of alternatives on the newly proposed dataset. The new dataset, plus all training and testing code of our model, will be publicly released to facilitate future research


We showed that this problem is coNP-complete, which is the same complexity we obtained in the relational case #REFR .

We have presented two algorithms that decide the definability problem by quantifier-free first-order formulas over a purely functional language.

These algorithms rely on the semantic characterization of quantifier-free definability given in #OTHEREFR .

Prior to this we developed an algorithm that gives a characterization of the isomorphism type of a tupleā in the given structure (in particular, the subuniverse it generates is obtained).

Our first definability-decision procedure (Merging Algorithm) partitions the set of all ktuples from the universe with k ∈ spec(R), and has to exhaust the search space to give a positive answer.

Therefore, the execution time of our algorithm depends exponentially on the parameter k.

The empirical tests confirm the exponential dependence on k, and give us an idea of the impact of the cardinality of the input structure and the family of algebras on which we performed the tests.

They also confirm the conjecture that this strategy, which is based on the existence of symmetries, would perform better on models with a large amount of subisomorphisms, as we The Splitting algorithm starts with a single block of tuples, and partitions blocks as the determination of the isomorphism types of their tuples progresses.

Two Algorithms To Decide Quantifier-Free Definability In Finite Algebraic Structures

Abstract: This work deals with the definability problem by quantifier-free first-order formulas over a finite algebraic structure. We show the problem to be coNP-complete and present two decision algorithms based on a semantical characterization of definable relations as those preserved by isomorphisms of substructures, the second one also providing a formula in the positive case. Our approach also includes the design of an algorithm that computes the isomorphism type of a tuple in a finite algebraic structure. Proofs of soundness and completeness of the algorithms are presented, as well as empirical tests assessing their performances.

The Complexity Of Definability By Open First-Order Formulas

Abstract: In this article we formally define and investigate the computational complexity of the Definability Problem for open first-order formulas (i.e., quantifier free first-order formulas) with equality. Given a logic $\mathbf{\mathcal{L}}$, the $\mathbf{\mathcal{L}}$-Definability Problem for finite structures takes as input a finite structure $\mathbf{A}$ and a target relation $T$ over the domain of $\mathbf{A}$, and determines whether there is a formula of $\mathbf{\mathcal{L}}$ whose interpretation in $\mathbf{A}$ coincides with $T$. We show that the complexity of this problem for open first-order formulas (open definability, for short) is coNP-complete. We also investigate the parametric complexity of the problem, and prove that if the size and the arity of the target relation $T$ are taken as parameters then open definability is $\mathrm{coW}[1]$-complete for every vocabulary $\tau$ with at least one, at least binary, relation.


Generative models often implement a Transformer architecture that includes an encoder and a decoder, utilizing an attention mechanism that allows back-references to prior content #REFR .

In recent years, various generative models have emerged for tasks such as text-to-text #OTHEREFR and text-to-image #OTHEREFR generation, music composition based on prompts #OTHEREFR , and text simplification #OTHEREFR .

Other models generate images #OTHEREFR , poetry #OTHEREFR , rap lyrics #OTHEREFR , or music #OTHEREFR in the style of previous works.

These models are typically trained to predict probable next tokens from previous text. Related tasks include infilling masked tokens.

Thus, they learn connections between subsequent words and are well-suited for text generation.

In this work, we focus on text generation with GPT-3.

Released in 2020, GPT-3 currently is a particularly large and powerful language model #OTHEREFR .

It uses deep learning methods such as attention mechanisms and the Transformer architecture with autoregressive pretraining #OTHEREFR .

The Ai Ghostwriter Effect: Users Do Not Perceive Ownership Of Ai-Generated Text But Self-Declare As Authors

Abstract: Human-AI interaction in text production increases complexity in authorship. In two empirical studies (n1 = 30 & n2 = 96), we investigate authorship and ownership in human-AI collaboration for personalized language generation models. We show an AI Ghostwriter Effect: Users do not consider themselves the owners and authors of AI-generated text but refrain from publicly declaring AI authorship. The degree of personalization did not impact the AI Ghostwriter Effect, and control over the model increased participants' sense of ownership. We also found that the discrepancy between the sense of ownership and the authorship declaration is stronger in interactions with a human ghostwriter and that people use similar rationalizations for authorship in AI ghostwriters and human ghostwriters. We discuss how our findings relate to psychological ownership and human-AI interaction to lay the foundations for adapting authorship frameworks and user interfaces in AI in text-generation tasks. CCS Concepts: • Computing methodologies → Natural language generation; • Applied computing → Publishing; Text editing; • Social and professional topics → Intellectual property.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.


We note that the approach in #REFR also used a similar dimension for the compressed space before quantization.

We then compared the index retrieval results with this ground truth as follows:

R ∩ k := 1 N q∈Q |k-NN(q i ) ∩ k-ANN(q i )| k (R ∩ k)

The approximate k-nearest neighbors (k-ANN) were computed for the index under review, while the exhaustive search on raw features (k-NN) served as the ground truth.

It's worth noting that the approach used to construct the ground truth for the (non-multimodal) Deep1B dataset #OTHEREFR is similar to the one described in #OTHEREFR .

We did ablations over the latent space dimensions and found that encoding to a relatively high dimension of 128 first (from 1024 for ViT-H-14) was more effective, than for example encoding to a low dimension directly and giving more memory to the quantizer.

Also like in #OTHEREFR , we used the IVFPQ index for every index we built atop of our descriptors and finally did several comparisons to a vanilla IVFPQ on raw features.

For the IVF, we chose 2 #OTHEREFR centroids, and for the PQ we constructed collections of indices with different values of M (the number of dimension chunks) with 2 8 centroids each.

Finally, we highlight the popular open source tool AutoFaiss, which will automatically construct and efficient PQ index given a memory budget.

All of our AutoFaiss indices use multiple PQ steps, with an initial optimized product quantization (OPQ) step for dimensionality reduction (see #OTHEREFR ), as well as a hierarchical navigable world search (HSNW) #OTHEREFR .

For instance, when providing the tool with "6G" total memory for the index, it produced a faiss construction pipeline of "OPQ16 112,IVF65536 HNSW32,PQ16x8" (Please see the faiss library for more details #OTHEREFR ).

On The De-Duplication Of Laion-2B

Abstract: Generative models, such as DALL-E, Midjourney, and Stable Diffusion, have societal implications that extend beyond the field of computer science. These models require large image databases like LAION-2B, which contain two billion images. At this scale, manual inspection is difficult and automated analysis is challenging. In addition, recent studies show that duplicated images pose copyright problems for models trained on LAION2B, which hinders its usability. This paper proposes an algorithmic chain that runs with modest compute, that compresses CLIP features to enable efficient duplicate detection, even for vast image volumes. Our approach demonstrates that roughly 700 million, or about a third of LAION-2B's images, are duplicates. Our method also provides the histograms of duplication on this dataset, which we use to reveal more examples of verbatim copies by Stable Diffusion and further justify the approach. The current version of the de-duplicated set can be found at github.com/ryanwebster90/snip-dedup.

Billion-Scale Similarity Search With Gpus

Abstract: Similarity search finds application in specialized database systems handling complex data such as images or videos, which are typically represented by high-dimensional features and require specific indexing structures. This paper tackles the problem of better utilizing GPUs for this task. While GPUs excel at data-parallel tasks, prior approaches are bottlenecked by algorithms that expose less parallelism, such as k-min selection, or make poor use of the memory hierarchy. ::: We propose a design for k-selection that operates at up to 55% of theoretical peak performance, enabling a nearest neighbor implementation that is 8.5x faster than prior GPU state of the art. We apply it in different similarity search scenarios, by proposing optimized design for brute-force, approximate and compressed-domain search based on product quantization. In all these setups, we outperform the state of the art by large margins. Our implementation enables the construction of a high accuracy k-NN graph on 95 million images from the Yfcc100M dataset in 35 minutes, and of a graph connecting 1 billion vectors in less than 12 hours on 4 Maxwell Titan X GPUs. We have open-sourced our approach for the sake of comparison and reproducibility.