formalisms: - id: pca name: Principal Component Analysis year: 1901 origin: Pearson; Hotelling (1933) signature: operation: project domain: vector codomain: vector objective_family: none meso_type: linear_projection macro_type: eigenvalue_problem canonical_reference: Pearson, K. (1901). On lines and planes of closest fit. status: seed - id: cca name: Canonical Correlation Analysis year: 1936 origin: Hotelling signature: operation: project domain: vector codomain: vector objective_family: correlation meso_type: joint_embedding macro_type: eigenvalue_problem canonical_reference: Hotelling, H. (1936). Relations between two sets of variates. status: seed - id: kernel_cca name: Kernel Canonical Correlation Analysis year: 2002 origin: Bach & Jordan signature: operation: project domain: vector codomain: vector objective_family: correlation meso_type: kernel_method macro_type: eigenvalue_problem canonical_reference: Bach, F.R. & Jordan, M.I. (2002). Kernel independent component analysis. JMLR. status: seed - id: ica name: Independent Component Analysis year: 1994 origin: Comon signature: operation: decompose domain: vector codomain: vector objective_family: information meso_type: linear_projection macro_type: optimization canonical_reference: Comon, P. (1994). Independent component analysis, a new concept? status: seed - id: fisher_lda name: Fisher Linear Discriminant Analysis year: 1936 origin: Fisher signature: operation: project domain: vector codomain: vector objective_family: none meso_type: linear_projection macro_type: eigenvalue_problem canonical_reference: Fisher, R.A. (1936). The use of multiple measurements in taxonomic problems. status: seed - id: mds name: Multi-Dimensional Scaling year: 1952 origin: Torgerson signature: operation: project domain: matrix codomain: vector objective_family: none meso_type: spectral_method macro_type: eigenvalue_problem canonical_reference: 'Torgerson, W.S. (1952). Multidimensional scaling: I. Theory and method.' status: seed - id: isomap name: Isometric Feature Mapping year: 2000 origin: Tenenbaum, de Silva & Langford signature: operation: project domain: manifold codomain: vector objective_family: none meso_type: spectral_method macro_type: eigenvalue_problem canonical_reference: Tenenbaum, J.B., de Silva, V., & Langford, J.C. (2000). A global geometric framework for nonlinear dimensionality reduction. Science. status: seed - id: lle name: Locally Linear Embedding year: 2000 origin: Roweis & Saul signature: operation: project domain: manifold codomain: vector objective_family: none meso_type: spectral_method macro_type: eigenvalue_problem canonical_reference: Roweis, S.T. & Saul, L.K. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science. status: seed - id: laplacian_eigenmaps name: Laplacian Eigenmaps year: 2003 origin: Belkin & Niyogi signature: operation: project domain: manifold codomain: vector objective_family: energy meso_type: spectral_method macro_type: eigenvalue_problem canonical_reference: Belkin, M. & Niyogi, P. (2003). Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation. researchor_artifact_id: laplacian_matrix status: seed - id: svd name: Singular Value Decomposition year: 1873 origin: Beltrami; Jordan; Golub & Reinsch signature: operation: decompose domain: matrix codomain: matrix objective_family: none meso_type: linear_projection macro_type: eigenvalue_problem canonical_reference: Golub, G.H. & Reinsch, C. (1970). Singular value decomposition and least squares solutions. researchor_artifact_id: singular_value_decomposition status: seed - id: nmf name: Non-negative Matrix Factorization year: 1999 origin: Lee & Seung signature: operation: decompose domain: matrix codomain: matrix objective_family: divergence meso_type: linear_projection macro_type: optimization canonical_reference: Lee, D.D. & Seung, H.S. (1999). Learning the parts of objects by non-negative matrix factorization. Nature. status: seed canonical_arxiv_id: 0408058 - id: kernel_pca name: Kernel Principal Component Analysis year: 1998 origin: Schölkopf, Smola & Müller signature: operation: project domain: vector codomain: vector objective_family: none meso_type: kernel_method macro_type: eigenvalue_problem canonical_reference: Schölkopf, B., Smola, A., & Müller, K.-R. (1998). Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation. status: seed - id: kernel_ridge_regression name: Kernel Ridge Regression year: 1998 origin: Saunders, Gammerman & Vovk; extended by many signature: operation: transform domain: vector codomain: scalar objective_family: none meso_type: kernel_method macro_type: optimization canonical_reference: Saunders, C., Gammerman, A., & Vovk, V. (1998). Ridge regression learning algorithm in dual variables. ICML. status: seed - id: gaussian_process name: Gaussian Process Regression year: 1996 origin: Williams & Rasmussen; roots in kriging (Matheron 1963) signature: operation: transform domain: vector codomain: distribution objective_family: likelihood meso_type: kernel_method macro_type: statistical_inference canonical_reference: Rasmussen, C.E. & Williams, C.K.I. (2006). Gaussian Processes for Machine Learning. MIT Press. status: seed - id: svm name: Support Vector Machine year: 1995 origin: Cortes & Vapnik signature: operation: transform domain: vector codomain: assignment objective_family: none meso_type: kernel_method macro_type: optimization canonical_reference: Cortes, C. & Vapnik, V. (1995). Support-vector networks. Machine Learning. status: seed - id: mmd name: Maximum Mean Discrepancy year: 2006 origin: Gretton, Borgwardt, Rasch, Schölkopf & Smola signature: operation: match domain: distribution codomain: scalar objective_family: none meso_type: kernel_method macro_type: statistical_inference canonical_reference: Gretton, A. et al. (2012). A kernel two-sample test. JMLR. status: seed - id: kl_divergence_min name: KL Divergence Minimization year: 1951 origin: Kullback & Leibler signature: operation: minimize domain: distribution codomain: distribution objective_family: divergence meso_type: information_geometry macro_type: optimization canonical_reference: Kullback, S. & Leibler, R.A. (1951). On information and sufficiency. researchor_artifact_id: relative_entropy status: seed - id: mutual_info_max name: Mutual Information Maximization year: 1948 origin: Shannon; McGill (1954) signature: operation: maximize domain: distribution codomain: scalar objective_family: information meso_type: information_geometry macro_type: optimization canonical_reference: Shannon, C.E. (1948). A mathematical theory of communication. researchor_artifact_id: mutual_information status: seed - id: cross_entropy_min name: Cross-Entropy Minimization year: 1948 origin: Shannon; Good (1956) signature: operation: minimize domain: distribution codomain: distribution objective_family: divergence meso_type: information_geometry macro_type: optimization canonical_reference: Good, I.J. (1956). The population frequencies of species and the estimation of population parameters. researchor_artifact_id: entropy status: seed - id: infonce name: InfoNCE / Contrastive Estimation year: 2018 origin: van den Oord, Li & Vinyals signature: operation: maximize domain: vector codomain: scalar objective_family: information meso_type: joint_embedding macro_type: optimization canonical_reference: van den Oord, A., Li, Y., & Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv:1807.03748. status: seed canonical_arxiv_id: '1807.03748' - id: js_divergence name: Jensen-Shannon Divergence year: 1991 origin: Lin; extended from KL signature: operation: match domain: distribution codomain: scalar objective_family: divergence meso_type: information_geometry macro_type: statistical_inference canonical_reference: Lin, J. (1991). Divergence measures based on the Shannon entropy. IEEE Trans. Info. Theory. status: seed - id: elbo name: Evidence Lower BOund Maximization year: 1999 origin: Jordan, Ghahramani, Jaakkola & Saul; variational Bayes signature: operation: maximize domain: distribution codomain: distribution objective_family: divergence meso_type: variational macro_type: optimization canonical_reference: Jordan, M.I. et al. (1999). An introduction to variational methods for graphical models. Machine Learning. status: seed canonical_arxiv_id: '1312.6114' - id: variational_inference name: Variational Inference year: 1990 origin: Hinton & van Camp; Jordan et al.; Wainwright & Jordan signature: operation: minimize domain: distribution codomain: distribution objective_family: divergence meso_type: variational macro_type: statistical_inference canonical_reference: Wainwright, M.J. & Jordan, M.I. (2008). Graphical models, exponential families, and variational inference. Foundations & Trends in ML. status: seed - id: mean_field name: Mean-Field Approximation year: 1937 origin: Landau; Weiss; applied to stat mech and later VI signature: operation: minimize domain: distribution codomain: distribution objective_family: divergence meso_type: mean_field macro_type: statistical_inference canonical_reference: Parisi, G. (1988). Statistical Field Theory. Addison-Wesley. status: seed - id: mle name: Maximum Likelihood Estimation year: 1922 origin: Fisher signature: operation: maximize domain: distribution codomain: scalar objective_family: likelihood meso_type: probabilistic_inference macro_type: statistical_inference canonical_reference: Fisher, R.A. (1922). On the mathematical foundations of theoretical statistics. status: seed - id: map_estimation name: Maximum A Posteriori Estimation year: 1763 origin: Bayes; Laplace signature: operation: maximize domain: distribution codomain: scalar objective_family: likelihood meso_type: probabilistic_inference macro_type: statistical_inference canonical_reference: Berger, J.O. (1985). Statistical Decision Theory and Bayesian Analysis. Springer. status: seed - id: em_algorithm name: Expectation-Maximization year: 1977 origin: Dempster, Laird & Rubin signature: operation: maximize domain: distribution codomain: distribution objective_family: likelihood meso_type: variational macro_type: optimization canonical_reference: Dempster, A.P., Laird, N.M., & Rubin, D.B. (1977). Maximum likelihood from incomplete data via the EM algorithm. JRSS-B. status: seed - id: mcmc name: Markov Chain Monte Carlo year: 1953 origin: Metropolis, Rosenbluth, Rosenbluth, Teller & Teller; Hastings (1970) signature: operation: sample domain: distribution codomain: sequence objective_family: none meso_type: probabilistic_inference macro_type: stochastic_process canonical_reference: Metropolis, N. et al. (1953). Equation of state calculations by fast computing machines. researchor_artifact_id: markov_chain status: seed - id: empirical_bayes name: Empirical Bayes / Type-II Maximum Likelihood year: 1955 origin: Robbins; Efron & Morris signature: operation: maximize domain: distribution codomain: distribution objective_family: likelihood meso_type: probabilistic_inference macro_type: statistical_inference canonical_reference: Robbins, H. (1955). An empirical Bayes approach to statistics. Proc. Third Berkeley Symp. status: seed - id: normalizing_flow name: Normalizing Flow year: 2015 origin: Rezende & Mohamed; Dinh, Krueger & Bengio (NICE 2014) signature: operation: transform domain: distribution codomain: distribution objective_family: likelihood meso_type: variational macro_type: statistical_inference canonical_reference: Rezende, D.J. & Mohamed, S. (2015). Variational inference with normalizing flows. ICML. status: seed canonical_arxiv_id: '1505.05770' - id: boltzmann_distribution name: Boltzmann / Gibbs Distribution year: 1868 origin: Boltzmann; Gibbs (1902) signature: operation: sample domain: scalar_field codomain: distribution objective_family: energy meso_type: energy_model macro_type: hamiltonian_system canonical_reference: Gibbs, J.W. (1902). Elementary Principles in Statistical Mechanics. researchor_mental_model_id: ergodicity status: seed - id: free_energy_min name: Free Energy Minimization year: 1873 origin: Helmholtz; Gibbs signature: operation: minimize domain: distribution codomain: distribution objective_family: energy meso_type: energy_model macro_type: hamiltonian_system canonical_reference: Helmholtz, H. (1882). Die Thermodynamik chemischer Vorgänge. researchor_mental_model_id: phase_transitions status: seed - id: hopfield_network name: Hopfield Network / Spin Glass year: 1982 origin: Hopfield; Sherrington & Kirkpatrick (1975 spin glass) signature: operation: minimize domain: vector codomain: assignment objective_family: energy meso_type: spin_system macro_type: hamiltonian_system canonical_reference: Hopfield, J.J. (1982). Neural networks and physical systems with emergent collective computational abilities. PNAS. researchor_mental_model_id: attractors_and_basins status: seed - id: renormalization_group name: Renormalization Group year: 1971 origin: Wilson; Kadanoff (block-spin 1966) signature: operation: transform domain: scalar_field codomain: scalar_field objective_family: none meso_type: energy_model macro_type: hamiltonian_system canonical_reference: Wilson, K.G. (1971). Renormalization group and critical phenomena. Phys. Rev. B. researchor_mental_model_id: scale_invariance status: seed - id: langevin_dynamics name: Langevin Dynamics year: 1908 origin: Langevin; adapted to sampling by Parisi (1981) signature: operation: sample domain: scalar_field codomain: distribution objective_family: energy meso_type: dynamical_system macro_type: stochastic_process canonical_reference: Langevin, P. (1908). Sur la théorie du mouvement brownien. researchor_artifact_id: brownian_motion status: seed - id: hamiltonian_monte_carlo name: Hamiltonian Monte Carlo year: 1987 origin: Duane, Kennedy, Pendleton & Roweth; Neal (2011 MCMC handbook) signature: operation: sample domain: distribution codomain: sequence objective_family: energy meso_type: dynamical_system macro_type: hamiltonian_system canonical_reference: Neal, R.M. (2011). MCMC using Hamiltonian dynamics. Handbook of Markov Chain Monte Carlo. status: seed - id: diffusion_sde name: Diffusion Process / Reverse-Time SDE year: 2015 origin: Sohl-Dickstein et al.; Ho, Jain & Abbeel (DDPM 2020); Song et al. (SDE 2021) signature: operation: sample domain: distribution codomain: distribution objective_family: none meso_type: diffusion_process macro_type: stochastic_process canonical_reference: Song, Y. et al. (2021). Score-based generative modeling through stochastic differential equations. ICLR. researchor_artifact_id: brownian_motion researchor_mental_model_id: irreversibility status: seed canonical_arxiv_id: '2011.13456' - id: wasserstein_distance name: Wasserstein Distance / Earth Mover's Distance year: 1781 origin: Monge; Kantorovich (1942) signature: operation: match domain: distribution codomain: scalar objective_family: none meso_type: optimal_transport macro_type: optimization canonical_reference: Kantorovich, L.V. (1942). On the translocation of masses. status: seed - id: sinkhorn_algorithm name: Sinkhorn-Knopp Algorithm / Entropic OT year: 1967 origin: Sinkhorn & Knopp; Cuturi (2013) for ML signature: operation: match domain: distribution codomain: matrix objective_family: divergence meso_type: optimal_transport macro_type: optimization canonical_reference: 'Cuturi, M. (2013). Sinkhorn distances: lightspeed computation of optimal transport. NeurIPS.' status: seed canonical_arxiv_id: '1306.0895' - id: kantorovich_dual name: Kantorovich Duality year: 1942 origin: Kantorovich signature: operation: maximize domain: distribution codomain: scalar objective_family: none meso_type: optimal_transport macro_type: optimization canonical_reference: 'Villani, C. (2008). Optimal Transport: Old and New. Springer.' status: seed - id: gradient_descent name: Gradient Descent year: 1847 origin: Cauchy signature: operation: minimize domain: scalar_field codomain: vector objective_family: none meso_type: none macro_type: optimization canonical_reference: Cauchy, A. (1847). Méthode générale pour la résolution des systèmes d'équations simultanées. researchor_artifact_id: gradient_descent status: seed - id: sgd name: Stochastic Gradient Descent year: 1951 origin: Robbins & Monro signature: operation: minimize domain: scalar_field codomain: vector objective_family: none meso_type: none macro_type: optimization canonical_reference: Robbins, H. & Monro, S. (1951). A stochastic approximation method. researchor_artifact_id: stochastic_gradient_descent status: seed - id: lagrange_multiplier name: Constrained Optimization (Lagrange Multiplier) year: 1788 origin: Lagrange signature: operation: minimize domain: scalar_field codomain: vector objective_family: none meso_type: none macro_type: optimization canonical_reference: Lagrange, J.-L. (1788). Mécanique Analytique. status: seed - id: proximal_gradient name: Proximal Gradient Method year: 2005 origin: Combettes & Wajs; Beck & Teboulle (FISTA 2009) signature: operation: minimize domain: scalar_field codomain: vector objective_family: none meso_type: none macro_type: optimization canonical_reference: Beck, A. & Teboulle, M. (2009). A fast iterative shrinkage-thresholding algorithm. SIAM J. Imaging Sci. status: seed - id: adam name: Adam Optimizer year: 2014 origin: Kingma & Ba signature: operation: minimize domain: scalar_field codomain: vector objective_family: none meso_type: none macro_type: optimization canonical_reference: 'Kingma, D.P. & Ba, J. (2015). Adam: a method for stochastic optimization. ICLR.' status: seed canonical_arxiv_id: '1412.6980' - id: softmax_attention name: Softmax Attention / Weighted Aggregation year: 2014 origin: Bahdanau, Cho & Bengio; Vaswani et al. (Transformer 2017) signature: operation: aggregate domain: sequence codomain: vector objective_family: none meso_type: none macro_type: none canonical_reference: Vaswani, A. et al. (2017). Attention is all you need. NeurIPS. status: seed canonical_arxiv_id: '1706.03762' - id: residual_connection name: Residual Connection / Skip Connection year: 2016 origin: He, Zhang, Ren & Sun; earlier in Hochreiter & Schmidhuber (LSTM 1997) signature: operation: transform domain: vector codomain: vector objective_family: none meso_type: none macro_type: none canonical_reference: He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. CVPR. status: seed canonical_arxiv_id: '1512.03385' - id: batch_normalization name: Batch Normalization year: 2015 origin: Ioffe & Szegedy signature: operation: transform domain: vector codomain: vector objective_family: none meso_type: none macro_type: none canonical_reference: 'Ioffe, S. & Szegedy, C. (2015). Batch normalization: accelerating deep network training. ICML.' status: seed canonical_arxiv_id: '1502.03167' - id: vae name: Variational Autoencoder year: 2013 origin: Kingma & Welling; Rezende, Mohamed & Wierstra signature: operation: transform domain: vector codomain: distribution objective_family: divergence meso_type: variational macro_type: statistical_inference canonical_reference: Kingma, D.P. & Welling, M. (2014). Auto-encoding variational Bayes. ICLR. status: seed canonical_arxiv_id: '1312.6114' - id: gan name: Generative Adversarial Network year: 2014 origin: Goodfellow et al. signature: operation: minimize domain: distribution codomain: distribution objective_family: adversarial meso_type: game_theoretic macro_type: optimization canonical_reference: Goodfellow, I.J. et al. (2014). Generative adversarial nets. NeurIPS. researchor_mental_model_id: nash_equilibrium status: seed canonical_arxiv_id: '1406.2661' - id: contrastive_learning name: Contrastive Learning year: 2005 origin: Chopra, Hadsell & LeCun (Siamese, 2005); Hadsell et al. (dimensionality reduction 2006) signature: operation: minimize domain: vector codomain: scalar objective_family: energy meso_type: joint_embedding macro_type: optimization canonical_reference: Hadsell, R., Chopra, S., & LeCun, Y. (2006). Dimensionality reduction by learning an invariant mapping. CVPR. researchor_mental_model_id: signal_vs_noise status: seed - id: knowledge_distillation name: Knowledge Distillation year: 2015 origin: Hinton, Vinyals & Dean signature: operation: minimize domain: distribution codomain: distribution objective_family: divergence meso_type: none macro_type: optimization canonical_reference: Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. NeurIPS workshop. status: seed canonical_arxiv_id: '1503.02531' - id: layer_normalization name: Layer Normalization year: 2016 origin: Ba, Kiros & Hinton signature: operation: transform domain: vector codomain: vector objective_family: none meso_type: none macro_type: none canonical_reference: Ba, J.L., Kiros, J.R., & Hinton, G.E. (2016). Layer normalization. arXiv:1607.06450. status: seed canonical_arxiv_id: '1607.06450' - id: ridge_regression name: Ridge Regression (L2 Regularization) year: 1970 origin: Hoerl & Kennard; Tikhonov (1943) signature: operation: minimize domain: vector codomain: scalar objective_family: none meso_type: none macro_type: optimization canonical_reference: 'Hoerl, A.E. & Kennard, R.W. (1970). Ridge regression: biased estimation for nonorthogonal problems.' status: seed - id: lasso name: LASSO (L1 Regularization) year: 1996 origin: Tibshirani signature: operation: minimize domain: vector codomain: scalar objective_family: none meso_type: none macro_type: optimization canonical_reference: Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. JRSS-B. status: seed - id: elastic_net name: Elastic Net year: 2005 origin: Zou & Hastie signature: operation: minimize domain: vector codomain: scalar objective_family: none meso_type: none macro_type: optimization canonical_reference: Zou, H. & Hastie, T. (2005). Regularization and variable selection via the elastic net. JRSS-B. status: seed - id: spectral_clustering name: Spectral Clustering year: 2000 origin: Shi & Malik (normalized cuts); Ng, Jordan & Weiss (2002) signature: operation: decompose domain: graph codomain: assignment objective_family: none meso_type: spectral_method macro_type: eigenvalue_problem canonical_reference: 'Ng, A.Y., Jordan, M.I., & Weiss, Y. (2002). On spectral clustering: analysis and an algorithm. NeurIPS.' researchor_artifact_id: laplacian_matrix status: seed - id: pagerank name: PageRank year: 1998 origin: Page, Brin, Motwani & Winograd signature: operation: propagate domain: graph codomain: vector objective_family: none meso_type: spectral_method macro_type: eigenvalue_problem canonical_reference: 'Page, L. et al. (1999). The PageRank citation ranking: bringing order to the web.' researchor_artifact_id: adjacency_matrix researchor_mental_model_id: network_centrality status: seed - id: fourier_transform name: Fourier Transform / Spectral Decomposition year: 1822 origin: Fourier; Cooley & Tukey (FFT 1965) signature: operation: decompose domain: sequence codomain: sequence objective_family: none meso_type: spectral_method macro_type: none canonical_reference: Fourier, J.B.J. (1822). Théorie analytique de la chaleur. researchor_mental_model_id: signal_vs_noise status: seed